How is Vaccine Efficacy Measured?
With several COVID-19 vaccines becoming widely available (at least in the USA), a common question is: How much protection do they offer? That is, how effective are they? When you hear that a vaccine had “an efficacy of 80%”, what does that mean?
Let’s first go over what that does not mean.
- It does not mean that you have a 80% chance of not getting COVID-19 after getting the vaccine.
- It does not mean that if you are exposed to the virus that causes COVID-19, you only have a 20% chance of getting infected.
The efficacy of a vaccine usually refers to the relative risk reduction the vaccine offers compared to not getting the vaccine. For example, if an unvaccinated person would have a 10% chance of contracting COVID-19 over a three-month period, then a vaccine that was 80% effective would reduce that risk to 2% over the same period, all else being equal.
$$ \displaylines{\text{No vaccine: 10% risk of infection} \\ \Downarrow \\ \text{80% effective vaccine} \\ \Downarrow \\ \text{With vaccine: 2% risk of infection}} $$
This constitutes an 80% reduction in the risk of contracting the disease relative to someone who did not receive the vaccine, which is where the “80% effective” comes from.
The reason the effectiveness is measured this way mostly has to do with how the research trials are conducted.
Testing a vaccine
Testing whether a vaccine (or indeed any drug) actually works is trickier than it might seem. Of course, you should probably do the research into how the vaccine works and understand the fundamental science behind its mechanism of action to start.
But the human body is complex and no two humans are alike, so until you actually use it on a person there is still a high level of uncertainty as to how well it will work. Even then, the answer to whether a given drug will “work” is rarely a hard yes/no. Instead, you have to think of the effect of the drug on a population or group of individuals and how many people on average it helped, and by how much.
A straightforward way might be to give the vaccine to a bunch of people (after at least determining it’s safe and does not cause harm) and see whether they develop the disease that the vaccine is intended to prevent. But how do you interpret the outcome? Let’s say you gave 10,000 people the vaccine and 10 eventually got sick from the disease. Did the vaccine work? You might say that, “no, it didn’t work, because some people still got sick”, but how do you know that more people would not have gotten sick if the vaccine was not administered?
In fact, this experiment tells us very little. What we really want to know, in addition to the performance of the group who got the vaccine, is what would their outcome have been had they not received the vaccine? We can then compare the difference in outcomes between these two scenarios to understand if the vaccine reduced the number of people getting sick or not. This is known as a counterfactual outcome because it is a hypothetical outcome that did not actually happen. (“counter” to the facts) So, how can we estimate this counterfactual outcome?
This is where some basic statistics can help. If we could assemble a group of 10,000 identical people, they could serve as our counterfactual outcome by withholding the vaccine from them and comparing their outcome with the 10,000 who did receive the vaccine. Now, we can’t obviously assemble a group of 10,000 identical people, but what we can do is:
- Randomly sample 10,000 people from the population for the vaccine group.
- Randomly sample another 10,000 different people from the population for the counterfactual group that does not receive the vaccine.
If we do this carefully to ensure there are no systemic differences between the groups (across different demographics like age, socioeconomic status, race, etc.) then we can reasonably assume that in aggregate, the risk for the disease between the two groups should be roughly the same.
Therefore, if the first group receives the vaccine and the second does not, any difference we observe in the rates of contracting the disease can be reasonably associated with the vaccine.
If we do this experiment, and after some period of time (letting those people go about their normal lives), we observe that:
- 10 people from the vaccine group got sick from the disease.
(In other words, there was a 10/10,000 = 0.1% risk) - 100 people from the non-vaccine group got sick from the disease.
(In other words, there was a 100/10,000 = 1% risk)
In other words, the vaccinated group had a 0.1% chance of getting sick, while the non-vaccinated group had a 1% chance. These two numbers are combined to calculate a Risk Ratio or Relative Risk (RR) as a means of measuring the vaccine efficacy. In this case, the RR is risk in the group who got the vaccine divided by the risk in the group who did not receive the vaccine:
$$ RR = {{\text{risk for treatment}}\over{\text{risk for control}}} = {{0.1\%}\over{1\%}} = 0.1 $$
This is a risk ratio or relative risk (RR) of 0.1. Equivalently, we can say that the vaccine offered a risk reduction of 90%, and this vaccine would be deemed to be “90% effective”. Note that this is just an estimate, which may have some uncertainty around it.
The RR measures the change in outcome (in this case, the outcome is “getting sick”) between those that got the vaccine and those that did not. It is always non-negative. Here are some interpretations of possible values of RR for a vaccine:
- RR = 0 means no one who got the vaccine developed the disease; 100% effective
- RR < 1 means that the vaccine helped reduce the risk; the closer to 0, the better
- RR = 1 means that vaccine did not help
- RR > 1 means that the vaccine actually made things worse
As a final note, this sort of clinical trial is only one step of the complicated process any drug goes through before it is approved by the FDA. I’ve greatly simplified my description of this phase to help convey the main points.
Using understandable language
Using the terms “Risk Ratio of 0.1”, or perhaps even more ambiguous, “90% effective” can be confusing or convey the wrong message to the lay person. Take, for example, this CNN Travel article where they stated:
In addition, real world studies of the Pfizer-BioNTech and Moderna vaccines show they are only 90% protective against the coronavirus, not 95% as reported in clinical trials. Translated into reality, that means for every million fully vaccinated people who fly, some 100,000 could still become infected.
(The error has since been corrected on the current version of the article)
If you’ve followed my explanation above, you’ll realize that this interpretation is simply incorrect. The only way it would be correct was if we assumed that every non-vaccinated person that flew would be infected! It also strains credulity and doesn’t pass the smell test - even at the height of the pandemic, I don’t recall planes producing thousands of infected individuals.
Based on this, I think there’s room for improvement in how the messaging of vaccine efficacy or effectiveness is conveyed. The CDC actually does a decent job of this in their reports, where they have a graphic about the mRNA COVID-19 vaccines that says:
Those who were fully vaccinated were 90% less likely to get infected
I would argue this language is easier to understand than just the term “90% effective”.
Why do we test this way?
Note the limitations of the above methodology. We don’t know how often the people from each group were actually exposed to the virus that causes the disease, but it doesn’t matter since the behaviour across both groups should be similar, and we’re measuring the difference between the groups. However, if the rate of exposure were so low that even unvaccinated people did not get sick very often (the base rate), then this sort of study might not produce useful results.
You might think that testing a vaccine by intentionally exposing a participant to the virus would be the most effective way to test, but this has some obvious major ethical problems. It violates the “first, do no harm” principle and would surely result in damage being done to some trial participants. Note that even with a proper trial procedure like the above, if a vaccine is shown to be highly effective, it may be considered unethical to withhold the vaccine from the placebo group that did not receive the vaccine.
Testing on human subjects is also restricted and regulated not only for ethical reasons but also for historical reasons where such “testing” was abused in horrible ways. Nazi “medical experiments” (more akin to torture) are perhaps the most infamous, but the Tuskegee Syphilis Study is one that is closer to home.
You might think that offering participants money in exchange for being exposed to the virus might “solve” the problem, but that would just piggyback upon the existing inequities in society: The well-off would not need to volunteer as money would not be seen as adequate compensation for risk of death or injury; whereas the socioeconomically deprived, who already have a higher risk of poor health outcomes, would be more drawn to participate. This is neither a fair nor desirable outcome.