The Cochrane report on excess mortality

A reaction to the report "Systematic literature analysis and international comparison 'Excess Mortality'", Cochrane Netherlands, April 3, 2023

Authors (in alphabetical order): Bram Bakker, Bastiaan Geelhoed, Maurice de Hond, Jillis Kriek, Fritsander Lahr, Ronald Meester, André Redert, Herman Steigstra, Anton Theunissen

Also found on the LinkedIn page from Ronald Meester.

Date: April 24, 2023

Introduction and context

On 14 April 2023, Minister Kuipers sent the above report[1]to the House of Representatives. This gave the report a status that warrants a response. The report concerns a meta-analysis, which describes or compares several international studies on excess mortality. In principle, meta-analyses are useful, although conclusions are often difficult or impossible to formulate, precisely because of the fact that causality is extremely difficult to establish in an epidemiological setting in such a large case.

In the accompanying letter to the House of Representatives, Kuipers provides the following summary of the results:

"The study describes that it shows a trend of higher excess mortality (the difference between observed and expected mortality (per 1 million population)) in Eastern European countries and lower excess mortality in Scandinavian countries, Australia and New Zealand. In addition, the study also includes a review of COVID-19 mortality. The highest COVID-19 mortality (per 1 million population) was seen in Western European countries, while COVID-19 mortality was also the lowest in Australia and New Zealand. The differences in the trends of excess mortality and specific COVID-19 mortality could possibly be explained by differences in the reporting of COVID-19-related deaths as well as differences in the ability to test individuals for COVID-19. The researchers also indicate that international differences in excess mortality may be explained by measures against coronavirus, organisation of and access to care, population characteristics, behaviour, geographical location and differences between countries in data quality. Finally, the researchers point out that a number of studies show that excess mortality is lower among the population of fully vaccinated people."

There is much to be said about this quote, and in principle it is good that the uncertainties and difficulties in comparing studies are mentioned. However, the last comment in particular is rather misleading because it suggests causality. Because what does the report say? We quote on page 8: "Vaccinations appear to be associated with lower excess mortality, but since higher vaccination rates were associated with the emergence of the Omikron variant, no causal relationship can be established." Cochrane's wording thus paints a different picture from the minister's suggestion. Not only that: as we will explain below, no conclusion about a relationship between vaccination coverage and the extent of excess mortality can be drawn from the report.

Of course, we are not just concerned with how Kuipers informs the House of Representatives. Rather, we make some observations about the report itself. We first present a list of some comments on the report. Our list of points of interest (which we present below) is by no means exhaustive - in fact, it would require a much larger time investment. We then show that the Cochrane report uses and cites sources incorrectly. We end with a brief conclusion.

Some comments on the report

1. The period 2020-2021 was probably chosen because of the Omtzigt motion. Since then, however, a wealth of information has emerged, in particular on the side effects and damage of vaccines and on the consequences of measures such as lockdowns. It is also precisely the over mortality that persists after vaccination that worries us, and which will not disappear after the arrival of the milder Omikron variant. Claims that may have seemed reasonable until 2021 do not hold up in the light of current insights. Thus, the report is already outdated when published and therefore of little or no use.
2. Comparisons between different studies are almost impossible if methodologies differ. There is no systematic methodological assessment of the various studies in the report. In any case, it is not clear what system was actually followed in selecting and assessing the various studies. So the study is not very systematic at all. If you don't look very deeply at the methodological strengths and weaknesses of individual studies, don't think very carefully and deeply about them including all the pitfalls and biases, and only say that "several studies say this", then you don't really have much use for it.
3. Even the term "excess mortality" itself has different definitions in different studies. [2] That in itself should be enough reason not to draw any conclusions from this report.
4. The report writes on page 6 that excess mortality has increased in Eastern Europe. This is completely false, seeExcess mortality in 34 countries, failing vaccines? (virusvaria.nl)for a discussion on this claim, which can only result from a naive and incomplete view of the data. Furthermore, countries like Belarus, Iceland and Greenland are not suitable for a study like this for various reasons. That analysis also shows that what was mistaken for a correlation between vaccination coverage and excess mortality cannot simply be that; based on historical data.
5. On page 7, we read that no studies were found in which vaccination and excess mortality were positively correlated. That can only mean that those studies were not included. Such studies do exist - in fact, there are many.[3]Indeed, here we see that the report was already dated when it was published.
6. It is completely unclear how is dealt with the various sources of bias, as for example recently described inSources of bias in observational studies of covid-19 vaccine effectiveness – PubMed (nih.gov), in the various studies as well as in this report.
7. Death certificates from at least CBS are shrouded in mystery, for various reasons. [4] For example, the WHO prescribes that when covid is listed somewhere on the death certificate, "Covid" must be recorded as the cause of death. There are many reasons why these data are not suitable as input for scientific research.
8. It is now also clear that there are large differences between batches of vaccines. In Denmark, about 70% of adverse events were found to come from 4% of the batches; a strong indication of a relationship between vaccines and mortality. [5] This may make a meta-analysis rather useless.
9. There are a lot of omissions in this report that are very relevant to this discussion on excess mortality, for instance with regard to the remarkably short protection against infection by the vaccines and the increasingly short protection by boosters. But we also mention the disappointing protection of the vaccines against new virus variants (a possibly important partial explanation of why the Omikron period was associated with high over-mortality rates), the now well-known side effects of the vaccines in the literature, which are quite worrying, the now well known differences between the various vaccine types and brands, and the striking over-mortality rates in the second half of 2021, all of 2022, and early 2023 in many highly vaccinated countries.
10. The exclusion criteria on pages 11 and 12 cannot be reconciled with research questions 5 and 6.
11. On page 13, we read, "In this way, different countries could be compared, and differences in follow-up duration between studies were corrected." The differences in follow-up duration are corrected by Cochrane by dividing the relevant variables (mortality/survival etc.) by the follow-up duration, so they were presented as rate (mortality per year). If a publication only zooms in on one wave, that wave gets hugely magnified when the values are converted to rates. A distinction should be made to individual waves. For each wave, you can convert values to rates so that countries can be compared. But now there is even the danger of comparing wave x in country A with wave y in country B on the basis of mortality rates, while in the meantime vaccinations may even have been rolled out.
12. For baseline question 5 (international differences), only studies that presented data from more than 5 countries were selected. However, even if studies cover only 1 country, these studies can still be used to compare countries, provided appropriate normalisations and standardisations are done. Thus, we think that with this selection criterion, there is an unnecessarily restrictive selection of studies.
13. Finally, we also note that there is an interesting difference between the version of the report sent to the House of Representatives and the version found on ZonMw's site.[6]On page 45, the ZonMw version reads: "Differences between excess mortality and COVID-19 mortality may be caused by different data quality." This sentence is not in the House of Representatives version. We find it confusing that there are apparently different versions in circulation, especially when it comes to comments that matter. This obviously does raise questions.

The use of resources

As a sample, we checked whether the Cochrane report used all sources correctly. This turned out not to be the case at all, and we illustrate this with some examples:

Example I

"One study presented excess mortality in relation to vaccination coverage for four regions in the United States and for three different time periods (Table 11 and Excel Appendix). [62] Mortality was highest for all regions in the period with the lowest vaccination coverage of about 7%."

The cited publication is (we follow the numbering from the Cochrane report) [62]Stoto MA, Schlageter S, Kraemer JD. COVID-19 mortality in the United States: It's been two Americas from the start. PLoS ONE 2022; 17(April 4) doi:https://dx.doi.org/10.1371/journal.pone.0265053.

However, the statement in the Cochrane report is incorrect because it probably overlooked the fact that there were periods when the vaccination rate is not given (a blank box in the table), but when it was in all likelihood lower than in the other periods in the table when the vaccination rate is given (because vaccination rates increase over time). So we then see that, for example, Northeast region has the lowest excess mortality (0.111) per capita in the period 31 May 2020 - 3 October 2020, while the vaccination rate for that period is not given, but was probably substantially lower than 7%, because mass vaccinations had not yet started then (so it will be 0% or at least close to 0%). This would then again point to the opposite conclusion (than the one in the Cochrane report). Thus, the Cochrane report suggests a conclusion (that lower vaccination coverage leads to higher excess mortality) that is not made by the authors of the article. Table 11 in the Cochrane report thus quotes selectively.

Example II

The Cochrane report continues on page 42 with: "Seven studies described the relationship between excess mortality and vaccinations. Multiple studies showed that increased vaccine availability and use correlated with lower excess mortality (Appendix 11B). [17, 31, 62, 70, 73]".

From reference [62] we have already shown that the Cochrane report does not interpret it properly. Do the other references show the connection that the authors of the Cochrane report point out? We'll check.

[17]Bell E, Brassel S, Oliver E, et al. Estimates of the Global Burden of COVID-19 and the Value of Broad and Equitable Access to COVID-19 Vaccines. Vaccine 2022; 10(8) doi:https://dx.doi.org/10.3390/vaccines10081320

The conclusion of the Bell paper does not state that increased vaccine availability and use are correlated with lower excess mortality. It may be that the authors of the Cochrane report have come to their conclusion based on the following statement in the Bell paper: "Our model's results under Scenario 1 show that in 2021 1.4 million direct deaths, 4.3 million excess (direct and indirect) deaths and 6.0 million hospitalisations have been averted and hospital resources worth USD 59 billion have been saved by COVID-19 vaccinations." Regardless of whether this conclusion is correct (after all, it is based on a model/scenario), it is simply different from what is claimed in the Cochrane report.

Source [31] then? This is aboutFazekas-Pongor V, Szarvas Z, Nagy ND, et al. Different patterns of excess all-cause mortality by age and sex in Hungary during the 2nd and 3rd waves of the COVID-19 pandemic. GeroScience 2022 Doi:https://dx.doi.org/10.1007/s11357-022-00622-3.

This study does not at all unambiguously show that (in the words of the Cochrane report) "increased vaccine availability and use would correlate with lower excess mortality". It shows a small decrease in "excess absolute mortality" from the second to the third wave while simultaneously increasing vaccination coverage. But this effect, the authors of the Fazekas paper also broke down by age, and seems to be driven mainly by the group aged 65 and above. The age groups 35 - 44 years, 45 - 54 years and 55 - 64 years all three showed an increase in excess absolute mortality, which was accompanied by a similarly increasing vaccination rate.

Source [70] then? That'sWatson OJ, Barnsley G, Toor J, et al. Global impact of the first year of COVID-19 vaccination: a mathematical modelling study. The Lancet Infectious Diseases 2022; 22(9):1293-302. Doi:https://dx.doi.org/10.1016/S1473-3099%2822%2900320-6.

This is a mathematical modelling of how COVID-19 vaccination can reduce excess mortality. The model does not take into account any deaths due to vaccination, adverse events or vaccine (in)safety. While interesting, this study should not be used to draw conclusions regarding mortality due to vaccination, adverse events or (in)safety of the vaccines, as these are assumed a priori to play no role by the model used.

Source [73] then? This is aboutZhou F, Hu TJ, Zhang XY, et al. The association of intensity and duration of non-pharmacological interventions and implementation of vaccination with COVID-19 infection, death, and excess mortality: Natural experiment in 22 European countries. Journal of Infection and Public Health 2022; 15(5):499-507. Doi:https://dx.doi.org/10.1016/j.jiph.2022.03.011.

In fact, this publication shows the opposite effect to what is claimed in the Cochrane report. In Figure 35, it clearly shows that the "vaccine effect" on excess mortality increases with increasing "vaccination coverage". That the relative risks (RR) are below 1 is irrelevant here. What matters for the correlation between excess mortality and vaccination coverage is only the fact that the RRs increase with increasing vaccination coverage.

Discussion and conclusion

We did not have time to study ALL sources of the Cochrane report, but where we did, there were invariably problems. Incidentally, in our research into problems with sources, we have encountered many more problems than those we mention here. Of course, the question is how this is possible, but we're unable to answer that question.

For context, it might be worth noting that Cochrane's Dutch team, including the director of Cochrane Netherlands, consists exclusively of employees of the UMC Utrecht. The director of contractor Cochrane Nederland also performs work for the final client (VWS).

Based on the foregoing, our quality judgment is clear: this Cochrane report hardly makes a meaningful contribution to research on excess mortality.

References

[1]Systematic literature analysis and international comparison 'Excess mortality' | Publication | Rijksoverheid.nl

[2] See for exampleComparison of pandemic excess mortality in 2020–2021 across different empirical calculations – ScienceDirect, as well as some other references in the Cochrane report.

[3] See for exampleIs there a Link between the 2021 COVID-19 Vaccination Uptake in Europe and 2022 Excess All-Cause Mortality? [v1] | Preprints.org,(16) (PDF) Australian COVID-19 pandemic: A Bradford Hill analysis of iatrogenic excess mortality (researchgate.net),(16) (PDF) Causal effect of covid vaccination on mortality in Europe (researchgate.net),(16) (PDF) A likelihood analysis of COVID-19 mRNA vaccine safety in the third booster campaign in The Netherlands (researchgate.net). There is much more to be found of course.

[4] See for exampleIs there really excess mortality? – Unexplained excess mortality.

[5]Batch-dependent safety of the BNT162b2 mRNA COVID-19 vaccine – PubMed (nih.gov)

[6]Mortality report (zonmw.nl)