Friday, January 29, 2021

How Vaccine Effectiveness Is Calculated: A Case Study

The CDC says this: "Vaccine effectiveness is the percent reduction in the frequency of... illness among vaccinated people compared to people not vaccinated...". ("How Flu Vaccine Effectiveness and Efficacy are Measured," CDC, Jan. 29, 2016, <https://www.cdc.gov/flu/vaccines-work/effectivenessqa.htm>.)

The Washington Post recently illustrated how this works -- in conjunction with a covid vaccine. (Flu vaccines are handled the same way.)

In a trial of 44,000, Pfizer gave the covid vaccine to half.

Out of all 44,000 people, 170 people got covid: 162 were counted as unvaccinated and 8 were counted as vaccinated. 

(I say "counted as" because a person wasn't considered "vaccinated" until "7 days after the second dose." [See "COVID-19 Frequently Asked Questions," <https://dhhr.wv.gov/COVID-19/Documents/COVID-19%20Vaccine%20FAQ.pdf>] So, if a person got covid in between the first shot and the 7-day mark after the second shot, they were counted as "unvaccinated.")

Here is how the "95% effectiveness" is calculated: by taking the "reduction in the frequency of ...illness among vaccinated people compared to people not vaccinated...". 

So... it's basically the number of people who were vaccinated and ill (8) represents a "reduction" compared to the 162 who were unvaccinated and ill.

In other words, I understand the "effectiveness" claim to be saying: since out of 170 ill people, 162 were unvaccinated, we can say that the vaccine took away 95% of the illness (162/170 = 0.95294117647).

But, the fact is, out of 22,000 unvaccinated people, only 162 got covid. So, the base rate of infection is 162/22,000 = 0.00736363636, or ~ 0.74%.

Among the vaccinated, 8 of 22,000 got covid. So, that's 8/22,000 = 0.00036363636, or ~ 0.04%.

Indeed, the percent decrease is = 95.0617%. But that's only impressive-sounding if you forget about, or neglect to weigh, the initial, infinitesimal infection risk.

Arguably, most outlets repeating the "95% effective" claim commit the so-called "base-rate fallacy." But that's a topic for another time.