Have COVID vaccines saved lives? A cross-country approach

Steve Ohana (PhD), steve.ohana@gmail.com

Shahar Gavish, international-research@pecc.org.il

Abstract
We perform a cross-country analysis of the relation between the vaccination rate and post-vaccination COVID mortality, from the Our World in Data public database. Using a variety of problem specifications, we find no significant negative relation between vaccination rate and COVID mortality across countries. This finding does not lend support to the claim that COVID vaccines have saved lives, at least until now.

Objective
Assessments on the efficacy of the covid19 vaccines vary widely between the manufacturers’ research statements and others [1]. Assessing the claim that “many lives were saved by Covid19 vaccines” within a specific country is challenging due to the difficulty of building an objective counterfactual scenario in the absence of the vaccine, which requires the use of questionable data and assumptions.

The goal of this article is to revisit this problem through a cross-country analysis based on public data. Our premise is that, if Covid19 vaccines have indeed prevented a significant number of Covid19 infections and deaths, then we should observe a significant negative correlation between vaccination rate and post-vaccine Covid19 mortality across different countries, even more so considering the claimed high efficacy of the vaccines as published by the manufacturers.

Data
We use the Our World in Data (OWID) database [2] for vaccine doses administered and Covid19 deaths, which comprises 212 countries. The vaccination rate is measured through the total doses administered by hundred inhabitants up to the final date of the sample (May 20th, 2021). As for the Covid19 mortality, it is evaluated through the Covid19 deaths per million per year between two specified dates.

Methodology
Since we want to discard short-lived vaccination experiments where the vaccinated populations may not have the time to develop an immunity to the disease, we only select the countries which have experienced at least 120 days(1) of vaccination up to May 20th, 2021, hence, which have launched their vaccination campaigns not after Jan 20th, 2021. We obtain a sample of 54 countries (see Table 1 below). We further restrict the sample to the countries which have mainly used mRNA/DNA vaccines, as opposed to inactivated vaccines, because we seek homogeneity in the number of doses required and in the efficacy of the vaccination across the sample of countries.

The vaccination rate and Covid19 mortality are measured 120 days after the launching date of the vaccination campaign, defined as the first date when the total vaccinations per hundred inhabitants become positive in the OWID dataset. To account for the time needed for the vaccinated population to develop an immunity to the disease, the post-vaccination COVID mortality is computed from the 30th(2) to the 120th day after the launching date of the vaccination campaign.

Country-specific Covid19 mortality figures are determined by a number of country characteristics, such as, geography, population density, age structure, life expectancy, the quality of the healthcare system, the environmental conditions etc., which could confound the analysis of the effect of the vaccination itself. In order to control for all these factors, we normalize the mortality figures by computing, for each country, the ratio of post to pre vaccination COVID mortality (a variable we henceforth refer to as “post to pre vaccine mortality ratio”). This normalization allows us to analyze the way the vaccination rate is related to the variation in COVID mortality (post to pre vaccination) across different countries. For a given country, the pre-vaccination Covid19 mortality is defined as the Covid19 deaths from March 1st, 2020 to the vaccination launching date. Post to pre vaccine mortality ratios are reported in Table 1, together with pre and post vaccination Covid19 mortality rates.

Results
The rank correlation between vaccination rate and post to pre vaccine mortality ratio is -16%, the p-value of the test of significance of this correlation is 32%, hence this correlation is not statistically different from 0. To make the linear regression more robust to outliers (see e.g. Estonia and Slovakia in Table 1), we regress the log post to pre vaccine mortality ratio on the vaccination rate. The regression coefficient has a negative sign but is not statistically different from 0 and the regression R² very close to zero (Table 2). In particular, the high vaccination countries, such as Israel, Monaco, Malta, Singapore, UK, and the U.S. display an average log post to pre vaccine mortality ratio which is close to the one of the overall sample.

To check the robustness of this result, we have later relaxed all conditions on the length of the vaccination period and the vaccine type, only imposing a strictly positive terminal vaccination rate as a selection criterion. This leads to a sample of 166 countries, on which we find a significantly positive rank correlation of 23% as well as a significantly positive regression coefficient of the log post to pre vaccine mortality ratio to the vaccination rate, with an R² close to 5% (Table 3). Figure 2 suggests that the high vaccination countries display an average log post to pre vaccine mortality ratio which is higher than the one of the low vaccination countries. This is confirmed by a means difference test on the samples with vaccination rates higher and lower than 70.

Conclusions
Our conclusion that the vaccination rate is not related to the variation in (post to pre vaccination) COVID mortality across countries is therefore robust under very different kinds of methodological settings.

Our cross-country examination hence does not support the claim that “Covid19 vaccines have saved lives”, at least until now. An important avenue for future research is to investigate the cross-country relation of the vaccination rate to the global mortality rate for each age group.

References

[1] Fenton, N., Is the Pfizer vaccine as effective as claimed? Monday, 17 May 2021, http://probabilityandlaw.blogspot.com/2021/05/important-caveats-to-pfizer-vaccine.html

[2] Mathieu, E., Ritchie, H., Ortiz-Ospina, E. et al. A global database of COVID-19 vaccinations. Nat Hum Behav (2021)

Endnotes

(1) Our conclusions are robust to alternative specifications, e.g., a minimal vaccination period of 90 days.

(2) Again, our conclusions are robust to alternative specifications of the time needed for the vaccinated population to develop an immunity to the disease (e.g., 15 days).