Pfizer-BioNTech just announced the results of a COVID-19 vaccine clinical trial of 12- to 15-year olds. Because the vaccinated group had 0 infections, the news media jumped to the conclusion that the vaccine has “100% efficacy” on 12- to 15-year olds. That is what the chyron said on NBC news. A look at the published trial results and quick analysis with current methods confirms that the vaccine works for that population but not that it eliminates 100% of the infections.

## The COVID-19 Data From Pfizer

A look at the data shows that, yes, their COVID-19 vaccine works in that population but in no way does it show that it is “100% effective,” as can be seen from the study’s contingency table:

begin{matrix}
textbf{Group} & textbf{Infected} & textbf{Not: Infected} & textbf{Total}\
Vaccinated& hphantom{1}0 & 1,131 & 1,131\
hphantom{100}Placebo& 18& 1,111& 1,129\
hphantom{1000}Total& 18 & 2,242 & 2,260
end{matrix}
## Testing The Proportions Of COVID-19 Infections

The relative frequency of infections in the Vaccinated group is hat{p}_{vac} = frac{0}{1,131} = 0 and, in the Placebo group is hat{p}_{pla} = frac{18}{1,129} = 1.59%

If we apply the prop.test function from R to test whether these results contradict the hypothesis that the probability of COVID-19 infection is no better in the Vaccinated group than in the Placebo group, and it gives a p-value of 2.8times 10^{-5}, meaning that these numbers prove the vaccine works on the teenagers. How well it works, however, is a separate question.

## COVID-19 Vaccine Efficacy Confidence Interval

That hat{p}_{vac} = 0 on a sample of 1,131 in no way proves that it would still be 0 on a sample of 1,000,000. As in Acceptance Sampling In The Age of Low PPM Defectives, the logic of extrapolating from the frequency observed on the small sample breaks down when the number of occurrences is 0. As explained by Carlo Graziani, the parameter of interest is the Vaccine Efficacy (VE), which is the reduction in the probability of contracting COVID-19 due to the vaccine.

Based on a few assumptions on the data, he arrives at a confidence interval for VE. Perhaps it is too complicated to explain on the evening news on TV but, with his methods, we can assert that there is a 99% probability that VE ≥ 69% and a 90% probability that VE ≥ 85%. By comparison, the US CDC has been happy with VEs of 60% for flu vaccines, so the Pfizer results on the COVID-19 vaccine on teenagers are excellent. They are just not VE = 100%.

## Graziani’s Method

Graziani’s assumptions are understandable if you have studied probability theory and Bayesian statistics:

- Infections in both the Vaccinated group and the Placebo group follow Poisson distributions.
- Prior to the study, all VEs between 0 and 100% were equally likely. This is the “ignorance prior” assumption, pessimistic but prudent.

Graziani’s formula provides a *posterior* distribution, refining the prior based on the results of the trial. This posterior distribution then provides the confidence intervals. Graziani made his Python code for these calculations available to download.

With two equally sized groups, 18 COVID-19 infections in the Placebo group and none in the Vaccinated group, Graziani’s formula for the probability distribution function of VE simplifies to

fleft ( x right ) = frac{17}{left ( 2-x right )^{18}} for

x between 0 and 100%

and its cumulative distribution function to

Fleft ( x right ) approx frac{1}{left ( 2-x right )^{17}}
## Trial Sizes

Surprisingly, the size of the groups disappears from the formulas but this size still matters, because it determines the number of infections in the Placebo group. If it were 10 times larger, there would be on the order of 10 times as many infections. Why was the trial not conducted on a larger group? The trial on adults involved 30,000 participants but they were *adults*, and able to decide to accept the risks. Clinical trials on children pose an ethical dilemma; applying the same techniques to validate a solution to a quality problem in Manufacturing doesn’t.

## References

- Graziani, C. (2020)
*A Simplified Bayesian Analysis Method for Vaccine Efficacy*, medRΧiv, https://doi.org/10.1101/2020.12.07.20244954
- Chu, H. & Halloran, M. E, (2004)
*Bayesian estimation of vaccine efficacy* Clinical Trials: Journal of the Society for Clinical Trials, 1(3), pp. 306-314

#covid19, #bayesianstatistics, #quality, #efficacy

Apr 2 2021

## COVID-19 Pfizer Vaccine Study On Teens

Pfizer-BioNTech just announced the results of a COVID-19 vaccine clinical trial of 12- to 15-year olds. Because the vaccinated group had 0 infections, the news media jumped to the conclusion that the vaccine has “100% efficacy” on 12- to 15-year olds. That is what the chyron said on NBC news. A look at the published trial results and quick analysis with current methods confirms that the vaccine works for that population but not that it eliminates 100% of the infections.

Contents

The COVID-19 Data From Pfizer

Testing The Proportions Of COVID-19 Infections

COVID-19 Vaccine Efficacy Confidence Interval

Graziani’s Method

Trial Sizes

References

## The COVID-19 Data From Pfizer

A look at the data shows that, yes, their COVID-19 vaccine works in that population but in no way does it show that it is “100% effective,” as can be seen from the study’s contingency table:

begin{matrix} textbf{Group} & textbf{Infected} & textbf{Not: Infected} & textbf{Total}\ Vaccinated& hphantom{1}0 & 1,131 & 1,131\ hphantom{100}Placebo& 18& 1,111& 1,129\ hphantom{1000}Total& 18 & 2,242 & 2,260 end{matrix}## Testing The Proportions Of COVID-19 Infections

The relative frequency of infections in the Vaccinated group is hat{p}_{vac} = frac{0}{1,131} = 0 and, in the Placebo group is hat{p}_{pla} = frac{18}{1,129} = 1.59%

If we apply the prop.test function from R to test whether these results contradict the hypothesis that the probability of COVID-19 infection is no better in the Vaccinated group than in the Placebo group, and it gives a p-value of 2.8times 10^{-5}, meaning that these numbers prove the vaccine works on the teenagers. How well it works, however, is a separate question.

## COVID-19 Vaccine Efficacy Confidence Interval

That hat{p}_{vac} = 0 on a sample of 1,131 in no way proves that it would still be 0 on a sample of 1,000,000. As in Acceptance Sampling In The Age of Low PPM Defectives, the logic of extrapolating from the frequency observed on the small sample breaks down when the number of occurrences is 0. As explained by Carlo Graziani, the parameter of interest is the Vaccine Efficacy (VE), which is the reduction in the probability of contracting COVID-19 due to the vaccine.

Based on a few assumptions on the data, he arrives at a confidence interval for VE. Perhaps it is too complicated to explain on the evening news on TV but, with his methods, we can assert that there is a 99% probability that VE ≥ 69% and a 90% probability that VE ≥ 85%. By comparison, the US CDC has been happy with VEs of 60% for flu vaccines, so the Pfizer results on the COVID-19 vaccine on teenagers are excellent. They are just not VE = 100%.

## Graziani’s Method

Graziani’s assumptions are understandable if you have studied probability theory and Bayesian statistics:

Graziani’s formula provides a

posteriordistribution, refining the prior based on the results of the trial. This posterior distribution then provides the confidence intervals. Graziani made his Python code for these calculations available to download.With two equally sized groups, 18 COVID-19 infections in the Placebo group and none in the Vaccinated group, Graziani’s formula for the probability distribution function of VE simplifies to

fleft ( x right ) = frac{17}{left ( 2-x right )^{18}} for x between 0 and 100%and its cumulative distribution function to

Fleft ( x right ) approx frac{1}{left ( 2-x right )^{17}}## Trial Sizes

Surprisingly, the size of the groups disappears from the formulas but this size still matters, because it determines the number of infections in the Placebo group. If it were 10 times larger, there would be on the order of 10 times as many infections. Why was the trial not conducted on a larger group? The trial on adults involved 30,000 participants but they were

adults, and able to decide to accept the risks. Clinical trials on children pose an ethical dilemma; applying the same techniques to validate a solution to a quality problem in Manufacturing doesn’t.## References

A Simplified Bayesian Analysis Method for Vaccine Efficacy, medRΧiv, https://doi.org/10.1101/2020.12.07.20244954Bayesian estimation of vaccine efficacyClinical Trials: Journal of the Society for Clinical Trials, 1(3), pp. 306-314#covid19, #bayesianstatistics, #quality, #efficacy

By Michel Baudin • Data science • • Tags: Bayesian Statitics, COVID-19, Efficacy, Quality