'Et tu-John' – filleted, but now real

The stunning research that resulted in 14.8 million years of life saved thanks to Covid vaccines (see earlier short response here) has now also been put through the wringer on LinkedIn. Ronald Meester placed there a review by Bram Bakker and himself, in English (Download here). Below is an auto-translation from the Dutch translation.

One response to "Global estimates of lives and life-years saved by COVID-19 vaccination during 2020-2024", by John Ioannidis et al.

Bram Bakker and Ronald Meester
November 23, 2024

Introduction

In this article [1], John Ioannidis and colleagues estimate the number of lives (and years of life) saved worldwide by the Covid vaccinations, using very simple and straightforward mathematical modelling, based on rough estimates of the Infection Fatality Rate (IFR) and Vaccine Effectiveness (VE), among others. Based on their analysis, they estimate that in 2020-2024, around 2.5 million deaths were prevented worldwide and around 15 million years of life were saved. This is at least an order of magnitude lower than the most widely cited previous estimates (which is remarkable in itself and shows how variable these estimates can be), but nonetheless suggests large numbers of lives saved by the vaccines.

While we generally appreciate Ioannidis' work, this is by no means a convincing article. The most important thing to realize is that this is aPilot studyis based on many highly debatable assumptions, laid down in parameters such as assumed VE and assumed IFR per age stratum and so on. Everything depends on those assumptions and the model. In that sense, the model will simply tell you what you, as a researcher, put into it. Assume a positive VE and assume an IFR that is even slightly higher than 0, and you will 'find' (the model will say that) that the vaccines have saved many lives (and years of life).

Perhaps the best thing about this article is that Ioannidis et al. are at least open about all this and emphasize in several places that the assumptions can be incorrect and that many things are unclear and uncertain. But an experienced researcher like Ioannidis should know that these nuances will be lost in the overall coverage of the article, and that people (especially those who defend the vaccines and whose impartiality can be questioned, such as policymakers and their advisors) will simply say, "You see, even Ioannidis, always so critical of Covid policy, has shown that the vaccines have saved millions of lives!". We do not think that this is a justified conclusion at all, and we will explain that now.

As we mentioned, this is a modeling document. Assumptions and model choices are inevitable when making models, but for the model outcomes to be relevant and realistic, the assumptions and the model approach must both be reasonable. Are they? We argue that this is not the case. Again, to be fair, Ioannidis et al. discuss this in detail and admit the great uncertainties and their concerns about much of it. It's worth looking at some of those uncertainties in detail – to see how reasonable they actually are. We discuss four (interrelated) assumptions and choices, namely

1. IFR estimates;
2. VE estimates;
3. Ignorance of adverse events;
4. The (implicit) decision to model 'Covid deaths' as something that is independent of other (or all) causes of death.

IFR

Despite claiming in some places that their IFR estimates are "conservative," they themselves admit that their IFR estimates may in fact be overestimates for 2021 and beyond (when vaccines were available), because they are based on data from 2020. In their words:

"In addition, our IFR estimates are derived from national seroprevalence studies before vaccination. For unvaccinated individuals, IFR may have been lower in the second year (the pre-Omicron period that is important for calculating the number of lives saved) due to the availability of some effective treatments (e.g., dexamethasone), better organization of health care, and increased experience in dealing with severe COVID-19."

Far beyond these carefully worded comments, there is now strong evidence that certainly in the first few Covid waves (in 2020) the treatment of Covid was far from ideal; think of the discussion around (excessive use of) ventilation, the lack of proper antibiotic treatment of secondary pneumonia that was often accompanied by the Covid virus infection, et cetera. Regardless of possible criticism of the treatment in that early period (that's beside the point here), we can't generalize IFR from 2020 to 2021 and beyond, for many reasons:

1. In addition to vaccines, more knowledge and more effective treatments were available;
2. A significant portion of the most Covid-vulnerable portion of the population had died (e.g., in nursing homes);
3. The first waves in an epidemic are usually the worst;
4. There was already significant natural immunity from previous (often undetected) infections among many people.

It is therefore very likely that the IFR in 2021, and even more so in 2022 and later (Omicron period), was significantly lower than what was estimated in and before 2020 – and therefore should be lower in these model calculations.

That said, what is positive about this article is that it at least makes an attempt to account for much lower and more realistic IFR for young people in general, much lower IFR in the Omicron period (2022-2024), and the impact of natural immunity from previous infection on IFR.

VE

Regarding the VE estimates, the authors write:

"Vaccine effectiveness for death: we hypothesized VE = 75% during the pre-Omicron period and 50% during the Omicron period. This is an aggregate estimate given the large heterogeneity of vaccination experiences (different vaccines, some of which were likely to be less effective than others), waning effectiveness especially with long-term follow-up, and also different vaccination experiences including many people who received only one or two doses in the pre-Omicron period."

Are these reasonable estimates? Given the original study results of the Covid vaccines and many peer-reviewed observational studies, they may well appear to be, and even appear to be "conservative" as claimed. The original Covid vaccination tests claimed a very high VE, much higher than this 50 to 75%: the tests reported 80-95% against symptomatic infection and 95-100% against hospitalization and mortality from Covid (the highest for the new mRNA vaccines that ended up being the most widely used). However, subsequent reanalyses of these trials (e.g., by Christine Stabell-Benn [2]) suggest that their external validity is limited and questionable, especially when it comes to mortality. This is due to several factors (among others):

1. A very limited follow-up time of only a few months after vaccination (after which protection against infection decreases rapidly);
2. Counting the first few weeks after vaccination as 'unvaccinated' (or 'incompletely vaccinated'), when in fact the risk of infection seems to increase during that period;
3. A lack of focus on all-cause mortality (ACM; as opposed to symptomatic Covid infection or Covid death), while ACM metrics show little to no benefit for the vaccines in the trials;
4. Underrepresentation of vulnerable groups in the studies (those who are most vulnerable to Covid and have the weakest immune systems), making external validity questionable.

This means that these clinical trials almost certainly vastly overestimate VE in the real world.

Not surprisingly, retrospective observational studies, conducted later after large-scale real-world rollouts, show very mixed results and cast doubt on the actual protection against ACM in the real world. There's no consensus at all on what constitutes a realistic real-world VE, but there is consensus that it's much lower than those 90-100% figures and that the protection that is out there is rapidly waning (hence the fairly sudden introduction of repeated boosters). Reports vary widely, from the more optimistic ones of around 90% (which lasts a few months) to effectively 0%.

Equally important, there is evidence of serious statistical artifacts influencing the observational studies: systematic biases that distort the results and lead to artificial overestimation of VE.
These include strongHealthy Vaccineeeffects (HVE, a well-known effect in vaccine studies) as well as vaccination status artifacts and misclassification problems. In our own work ([3]), we find similar evidence of very strong HVE influencing the previously reported very high levels of VE ([4], analyses by government health authorities). It seems to be mainly short-term HVE, as a result of not vaccinating very vulnerable people who are about to die. This is evidenced, among other things, by the vaccine's very high apparent 'protection' against Non-Covid deaths (cancer, cardiovascular disease, dementia, etc.), much greater even than the protection against Covid deaths, especially in the 4 weeks immediately after large-scale vaccination in the spring of 2021.
Second, our results suggest significant misclassification bias, i.e., many people who were vaccinated were not registered as such, and therefore count as 'unvaccinated' – with a bias towards people who were highly vulnerable and/or died shortly after vaccination. Together, these biases almost completely explain the previously reported very high VE. Without these artifacts, there remains no evidence in our analysis for a VE greater than 0 when it comes to protection against death.

It is worth looking in detail at the three main studies referred to by Ioannidis et al., which they use to justify their estimates of a VE of 75% pre-Omicron and 50% during Omicron: [5], [6] and [7]. Do these studies support these VE numbers used in modelling? In our opinion, no.

The first [5] is a retrospective observational study and reports high (approximately 90%) and fairly long-lasting (measured up to 8 months after the first dose) protection againstCovid death. No all-cause mortality metrics are presented, making it possible that a large number of these deaths are 'cause of death substitute' deaths (as described above), or that secondary negative mortality effects are ignored or masked.

The second study [6] is a meta-study that reviewed only randomized clinical trials, not observational studies. They did assess all-cause mortality, but with very short follow-up times: "Median follow-up ranged from 35 to 92 days after randomization for all outcomes.
Interestingly, and similar to Stabell-Benn's work [2], they found evidence that the viral vector vaccines (AstraZeneca, J&J, Sputnik) did provide substantial (but limited time, see above) protection against all-cause mortality, with a VE of about 75% — but the mRNA vaccines (Pfizer, Moderna) did not, although they prevent Covid infection very well. Note that the mRNA vaccines were mainly used worldwide, so this is clearly important for estimates of the total number of lives and years of life saved.

The third study [7] looked at Covid deaths and non-Covid deaths and has Ioannidis as one of the authors. It concludes:

"VE estimates for COVID-19 deaths and reinfections exceeded 75% through the end of 2021, but decreased significantly with extended follow-up. The risk of non-COVID-19 deaths was lower in vaccinated versus unvaccinated people. [The extremely low COVID-19 mortality, regardless of vaccination, indicates strong protection of previous infection against COVID-19 mortality. The lower non-COVID-19 mortality in the vaccinated population could indicate a healthy vaccination bias."

So this study simultaneously suggests that Covid mortality as a share of total mortality was very low, that strong protection from previous infection probably played a very large role, that all-cause mortality was (therefore) not very reduced by the vaccines, and that whatever apparent protection there was may have been influenced by unresolved (healthy vaccinated) bias.

We don't see at all that these three studies support the assumption of a total, aggregate VE of 75% before Omicron and 50% during Omicron.

Adverse events

We quote from the article [1] (with ouremphasis):

"Assessment of absolute net benefits in these populations, if any, require careful consideration of potential additional benefits to non-lethal outcomes (e.g., hospitalizations and other symptomatic disease), as well as any deaths and other consequences of adverse events (not included in our calculations)."

It seems rather absurd not to take into account side effects when calculating net benefits and potential lives and years of life saved by the vaccines. Vaccines can have 'non-specific' effects [2] on the body and health, and can cause side effects that affect overall health and even cause death. It is now common knowledge that (among other things):

1. Very vulnerable, older people died relatively often after and from Covid vaccines (e.g. results from Norway), after which vaccination for those subgroups was largely discontinued;
2. The AstraZeneca vaccine is associated with a relatively high risk of thrombosis-related serious side effects, especially for relatively young women (after which the administration of AZ vaccines was discontinued in many countries);
3. The Pfizer and Moderna mRNA vaccines are associated with a relatively high probability of myocarditis-related serious adverse events, especially in young men, and with several other adverse events.

The total morbidity and mortality due to these and other secondary effects, including general non-specific effects on the immune system, are not yet fully known and are still being studied; but at least they reaffirm the importance of looking at mortality and morbidity across the board rather than just Covid mortality and Covid disease, when trying to make estimates of "lives and years of life saved by the Covid vaccines". In our opinion, this is therefore a crucial oversight, or error, in the approach of Ioannidis et al.

Is covid mortality independent of all-cause mortality?

This brings us to the important issue of considering "Covid mortality" as something that is completely independent of other, or all, causes of death (ACM), and the corresponding decision of Ioannidis et al. to model it as such. This is essentially an assumption that every Covid death is simply added to the total number of deaths and that if Covid hadn't occurred, these people would have continued to live. Is that reasonable? As argued by Stabell-Benn [2], ourselves [3], and many others, this is unfounded.

To a large extent, and even more so after the first Covid waves of 2020, many of the so-called 'Covid deaths' are actually people dyingwithCovid instead ofatCovid. During Covid waves, the number of deaths from other respiratory illnesses (especially influenza) decreases sharply, as does the number of deaths from other 'old' causes of death such as dementia (and related conditions), cardiovascular disease and cancer – suggesting that an official 'Covid death' is often simply another cause of deathReplaces. This phenomenon was exacerbated by the WHO's official guidelines that instructed government agencies to label every death as a Covid death if a positive Covid infection was identified or even (in some cases) just suspected. In addition, there is anecdotal evidence suggesting that after the rollout of vaccination, vaccinated people who died with/from Covid were much less likely to be labeled "Covid death" compared to unvaccinated people. The analyses by Stabell-Benn et al., ourselves, and others suggest that when we look at ACM rather than Covid mortality alone, most of the evidence of protection from the vaccines either disappears or appears to be largely based on statistical artifacts.

Ioannidis et al. seem to hint at these problems when they write in somewhat vague terms:

"Basically, if a disease/condition/event kills everyone regardless of health status, for example, an atomic bomb, then f=1; Conversely, for a condition that occurs precisely when a patient dies from other, co-existing conditions, F approaches infinity. The exact positioning of COVID-19 on that spectrum and the relative proportion of over- and undercount of COVID-19 deaths are still debated with significant implications for the estimated burden of disease and vaccination benefits."

Despite this, they continue to make assumptions about Covid deaths as separate and independent of ACM.

Conclusion

We conclude that this article contains serious errors in several respects. It models (Measurenot!) Total lives and years of life saved by the vaccines not only in an overly simplistic way, but more importantly, the assumptions are downright unrealistic and overly optimistic regarding the vaccines.

References

1. Global estimates of lives and life years saved by COVID-19 vaccination in 2020-2024 (preprint). John P.A. Ioannidis, Angelo Maria Pezzullo, Antonio Cristiano, Stefania Boccia, nov. 2024,https://www.medrxiv.org/content/10.1101/2024.11.03.24316673v1.
2. Randomized Clinical Trials of COVID-19 Vaccines: Do adenovirus vector vaccines have beneficial non-specific effects? Christine S. Benn, Frederik Schaltz-Buchholzer, Sebastian Nielsen, Mihai G. Netea and Peter Aaby. May 2023,iScience, Vol. 26(5).
3. Final report of the study on a possible relationship between Covid-19 vaccinations and excess mortality in the Netherlands 2021 – 2023 (technical report). Ronald Meester, Marc Jacobs, et al., Aug. 2024,https://www.researchgate.net/publication/383239838_Eindverslag_van_het_onderzoek_naar_een_mogelijke_relatie_tussen_Covid-19_vaccinaties_en_oversterfte_in_Nederland_2021_-_2023.
4. Effect of COVID-19 vaccination on mortality from COVID-19 and on mortality from other causes, Netherlands. Brechje de Gier, Liselotte van Asten, Tjarda M Boere, Annika van Roon, Caren van Roekel, Joyce Pijpers, C H Henri van Werkhoven, Caroline van den Ende, Susan J M Hahné, Hester E de Melker, Mirjam J Knol, Susan van den Hof (2023), January 2021-January 2022.Vaccine. 2023 Jul 12; 41(31):4488-4496.
5. Effectiviteit van Covid-19-vaccins over een periode van 9 maanden in North Carolina. Lin DY, Gu Y, Wheeler B, Young H, Holloway S, Sunny SK, Moore Z, Zeng D.N Engl J Med.2022 Mar 10; 386(10):933-41.
6. Vaccins ter voorkoming van COVID-19: A living systematic review with Trial Sequential Analysis and network meta-analysis of randomized clinical trials. Korang SK, von Rohden E, Veroniki AA, Ong G, Ngalamika O, Siddiqui F, Juul S, Nielsen EE, Feinberg JB, Petersen JJ, Legart C, Kokogho A, Maagaard M, Klingenberg S, Thabane L, Bardach A, Ciapponi A, Thomsen AR, Jakobsen JC, Gluud C.PLoS One. 2022 Jan 21; 17(1):e0260733.
7. Effectiveness of the first and second doses of the Severe Acute Respiratory Syndrome Coronavirus 2 vaccine: a nationwide cohort study from Austria on hybrid versus natural immunity. Chalupka A, Riedmann U, Richter L, Chakeri A, El-Khatib Z, Sprenger M, Theiler-Schwetz V, Trummer C, Willeit P, Schennach H, Benka B, Werber D, Høeg TB, Ioannidis JPA, Pilz S.Open Forum Infect Dis.2024 Sep 19; 11(10):ofae547.

32 Likes

2 Comments

1. JANVAN RUTHthe 23 / 11 / 2024 to the 15: 13(Edit)The real number of deaths from Covid can be counted on the fingers of a few hands.
   all the others have been killed by ignorance or willfulness.2Answer
2. C de vriesthe 23 / 11 / 2024 to the 15: 59(Edit)Now let's hope that the pre-print article will be placed in a reputable journal and that the text of Master and Baker will be accepted and placed as a healthy letter and that a scientific discussion will get off the ground in which the Lancet article will also be included. And Ioannidis openly acknowledges his mistake (I think he is capable of that. Of course, he may also decide to withdraw it after repentance, with arguments, after placement). Would be nice.5