\ Translation
This article was originally published in Danish and has been translated by Stephanie Lammers-Clark. Read the Danish version of this article at Videnskab.dk.
You can read more about Videnskab.dk in English here.
Only five Danish lives have been saved as a result of lockdowns and restrictions, concludes Jonas Herby, who is a specialist consultant at the liberal, free-market think tank CEPOS, on the basis of an analysis that he co-authored.
The analysis has not been published in a scientific journal.
Nevertheless, the startling conclusion has gained attention in the media both in Denmark and abroad: »Research: Lockdowns have only saved five Danes from death,« Danish newspaper Berlingske wrote on the front page as an advertisement for an opinion piece by Jonas Herby.
And the liberal special advisor has also visited foreign radio and television programmes with this exact message.
But the conclusion of the analysis from CEPOS is based on weak and insufficient evidence, according to scientists that Videnskab.dk has spoken to.
»I would fail my students if they had made this analysis,« says Mathias Heltberg, who is a postdoc in biocomplexity at the Niels Bohr Institute and part of the Division for Infectious for Disease Preparedness at Statens Serum Institut (SSI).
Professor Karsten Juhl Jørgensen also does not have much faith in the research on which the conclusion from CEPOS is based:
»I look at the methodology, and immediately I do not think this analysis would make it through a peer review,« says Karsten Juhl Jørgensen, professor of evidence synthesis and screening at Cochrane Denmark and Center for Evidence-Based Medicine Odense.
Part of the criticism is that the analysis ends up involving very few studies.
Jonas Herby replies that they have searched for everything that exists and that he does not think that new studies will change much in regards to the final conclusion.
»It may be that in two years time there will be some figures that are a little different, but I do not think there is anything in our results that indicates that it will be significantly different,« says Jonas Herby, who also does not believe that his conclusion about the five saved lives is too severe or that it rests on too thin a foundation. Read more of his answers below.
Four points of criticism
The analysis, written by Jonas Herby and Lars Jonung, professor emeritus affiliated with Lund University in Sweden, and Steve H. Hanke, professor of applied economics at the Johns Hopkins University in Baltimore, Maryland, USA, is a so-called meta-analysis.
In a meta-analysis, scientists systematically review the studies published on the topic they would like to investigate. In this specific case, the scientists looked at studies that examined the effect of lockdowns on the number of deaths.
The analysis concludes that lockdowns on average reduced mortality by 0.2 percent in Europe and the United States.
The controversial psychology professor Jordan B. Peterson has shared Jonas Herby's analysis with his 2.3 million followers on Twitter.
But Karsten Juhl Jørgensen and Mathias Heltberg do not believe that the conclusion that lockdowns have saved only five lives is supported in the analysis.
They underline a number of points of criticism after reading it for Videnskab.dk:
- The meta-analysis is based on very few studies
- A single study is dominating the conclusion
- The authors compare studies that cannot be compared nor transferred to Danish conditions
- The data is insufficient and weak
We dive into each point, and afterwards Jonas Herby responds to the criticism.
\ The researchers behind the study are strong opponents of lockdown
You work for the free-market think tank CEPOS, and you have produced the analysis with two researchers who have also been quite clear about where they stand in the debate about lockdown. Steve H. Hanke shared the analysis in a tweet with the text ‘lockdowns are for losers’. Should your analysis be equipped with a disclaimer that you are active in the debate?
»None of us have any conflict of interest. It will appear when the study is published in a journal. It is not normal to declare that you have taken part in a debate. It would then be strange if scientists could not participate in the debate in an area they understand,« says Jonas Herby.
»That’s why I did not consider it at all. But maybe. We write a little about what initially piqued our interest. It could well be that we should have written something there,« he says.
What should it say instead?
»I do not know, because I have not thought about it at all. But that is an interesting thought. There's a reason we're starting to investigate something like this in the first place - it's because there's something we think is strange. There must be an interest that is aroused.«
Is there a danger in looking for something that confirms one's interest?
»Of course, and that's why you have to do something like this systematically. That you search systematically in the scientific databases, so you are sure to get everything, no matter how you think.«
»We can see from some of these things that the authors of some of the studies were biased and thought that lockdowns work when they write that they are very effective. When you take a closer look at their numbers, it turns out that it is a very small effect. So I think we all suffer if we are not very careful to be systematic.«
And you have been that?
»Our meta-analysis is the essence of the research right now. There is not a lot of data, but it is, after all, a lot more data than those who locked the country down had.«
Those who shut the country down were politicians. When you make that comparison, and with your background in CEPOS in general, can you say whether you have a research or a political ‘cap’ on in this?
»The study is with the research cap, but when I communicate, I am liberal. I do not know if there is a problem in that,« he says.
Criticism point 1: There are far too few studies in the analysis
Meta-analyses, like the one written by Jonas Herby and the two economics professors, are usually perceived as a type of research that provides the most credible evidence, because scientists review many different studies and collect and summarise what the research as a whole points towards.
The weakness of a meta-analysis, however, is that it is relies on the studies on which it is based, and if it is based on too few studies, it is problematic.
In this particular meta-analysis, the authors found 19,000 studies in scientific databases. After three screenings, 34 studies qualified to be included in the meta-analysis.
»In the analysis, one can read that their main conclusion - that lockdowns have reduced mortality by 0.2 percent - is only based on seven articles,« says Mathias Heltberg, who is in the Ministry of Health's expert group for COVID-19, where he has been involved in providing projections to help the Ministry of Health assess the consequences of easing restrictions.
Seven studies are not many articles when it comes to assessing something so complex, he emphasises.
Criticism point 2: One study dominates the result
Of the seven studies that form the basis of the conclusion, one is given prominence in relation to the other studies.
As Mathis Heltberg says, it is 'effectively the study on which they (Jonas Herby and his co-authors, ed.) base their numbers'.
The study in question was written by scientists from the University of Pretoria in South Africa, and they actually come to the completely opposite conclusion, namely that lockdowns had a good effect in terms of reducing mortality.
»I would not be so self-assured on the basis of a review that has been boiled down to seven articles and where so much emphasis has been given to one study. At the same time it is also a study that Jonas Herby and his co-authors believe have concluded incorrectly, but they still want to use the numbers,« says Mathias Heltberg, who can hardly suppress a laugh as he continues:
»I think it's infinitely arrogant to attach maximum value to a study when you do not trust the conclusion.«
91.8 percent of the basis for the conclusion is attributed to the study in question while the remaining six studies are given 8.2 percent weight combined, the table below reveals.
Criticism point 3: The studies can neither be compared nor transferred to Danish conditions
Karsten Juhl Jørgensen is concerned that the authors believe that estimates concerning other European countries and the USA can be directly transferred to Denmark especially since lockdown have been so different in the respective countries.
»The timing has been quite different if we look at countries like Italy, France, Spain and the UK compared to Denmark. These countries had far stricter lockdowns than us, but they were probably also far harder hit by COVID-19 before the restrictions were introduced. It can be extremely important in a situation where the infection develops exponentially,« says Karsten Juhl Jørgensen.
He is backed by Mathias Heltberg, who believes that the crucial parameter is when the restrictions were put in place.
»It saved a lot of people in 2020, is my postulate. The most important thing was not how strict the lockdown was - we can easily discuss children and schools - but the curve rises exponentially in 2020, and it breaks quite quickly after lockdown,« says Mathias Heltberg.
Karsten Juhl Jørgensen remarks that Jonas Herby and his co-authors discuss the significance of when the lockdown was initiated, but exclude studies that actually examine the significance of timing.
In general, the seven studies, that Jonas Herby and his co-authors base their conclusion on, are very different, and it is difficult to compare them.
All sorts of things come into play when examining whether lockdowns work or not.
For example, how high the infection rate was when lockdown was imposed, how much trust the population has in the authorities, and to what extent they are willing to follow the guidelines, Karsten Juhl Jørgensen points out.
The many different factors affect how much of an impact studies made in and about different countries find.
»When they combine studies that are so different, as a systematic reviewer you have to ask yourself if it makes sense at all,« says Karsten Juhl Jørgensen.
Mathias Heltberg makes the same observation:
»Based on their professionalism, they have selected the most relevant studies and compared their averages. It's fine if you compare apples to apples. Here, I think it is a borderline case, where the numbers they add up should not be added up,« says Mathias Heltberg.
READ ALSO: 5 questions to ask yourself when reading about science
Criticism point 4: The evidence is too weak for such a conclusion
When you want to test whether something has an effect on humans, randomised trials deliver the most reliable results.
In a randomised trial, participants are randomly divided into two or more groups.
One group receives the treatment you want to test the effect of, and the other groups are control groups.
In the meta-analysis, the authors have selected studies from the social sciences that examine the effect of lockdown empirically by collecting experiences from, among other things, observations, sources and cases.
»It presents a significant limitation, especially when a complex intervention has to be assessed in a complicated context with many factors. The reliability of the studies included in the analysis is in all probability poor, and therefore you must be very careful when concluding,« says Karsten Juhl Jørgensen, who specialises in assessment evidence and the quality of systematic reviews and meta-analyses.
»But it does not seem that any care has been taken when I look at the article in Berlingske.«
Mathias Heltberg also points out a lack of humility in communicating:
»How to assess the significance of lockdown is complex, and you must therefore be extra careful when commenting on it. I do not want to say that I have the answer to what significance lockdown has had, but I want to say that their conclusion borders on being extremely arrogant,« he says.
You can read Jonas Herby's response to the criticism in the box below.
\ Jonas Herby from CEPOS responds to the criticism
Criticism point 1: There are far too few studies in the analysis
»We had hoped that there would be more studies. We have searched for everything that exists, and there are no more studies,« says Jonas Herby, when he is presented with the criticism.
He hopes that in time more studies will be produced.
»It may be that in two years' time there will be some figures that are a little different, but I do not think there is anything in our results that suggests that it will be significantly different,« he says.
Should you have waited to publish your analysis until more research was available?
»I have received that comment several times. It's funny that this starting point exactly hits a study that points out that lockdowns may not have had as big an effect as thought, but all those who say in the media that 'if we open, then thousands of people will die’ are not questioned,« says Jonas Herby.
Criticism point 2: One study dominates the result
The study that researchers Jonas Herby and the two economics professors place the most emphasis on was made by scientists from the University of Pretoria in South Africa. It is weighted at 7,390. In comparison, the second highest study is weighted at 256.
The scientists behind the South African study conclude that lockdown has saved many lives - a conclusion that Jonas Herby and his co-authors completely disagree with.
Why do you attach so much value to a study when you do not trust the conclusion?
»It is not us who attach great importance to the studio. That is the method behind meta-studies. The importance simply depends on the uncertainty of the estimate,« says Jonas Herby.
The scientists behind the individual studies report how much uncertainty is associated with their results. In the South African study, the uncertainty is set very low and therefore the importance increases.
»It is also quite normal in meta-studies that you do not look so much at the authors' conclusions, but look at their results, and here the results show something other than what they conclude. We do have objections to something in their data work,« says Jonas Herby.
Jonas Herby believes that the scientists behind the study are biased - that is, that they have an agenda: They want to show that lockdowns work. Therefore, they have - consciously or unconsciously - misinterpreted the data, he believes.
Videnskab.dk asked Jonas Herby how he himself relates to the fact that he publishes research on the effect of lockdowns, at the same time as he is an active opponent of lockdowns in the public debate. (See his answer in the box in the article).
However, he admits that it would be prudent if there were more studies. If they could instead put 50 studies into their chart, they would be beyond the problem that one study is so much more important than the others.
»But it is not that important either. If we remove that study and instead set it at 256 (like the study that is weighted second highest, ed.), then you get 2.5 percent. It is not because really changes the overall conclusion. The effect of lockdown is still very small,« he says and continues:
»I think most people do not care so much about whether lockdown has led to -0.2 percent fewer deaths or -2.5 percent,« says Jonas Herby.
According to his own calculations, the -2.5 percent, transferred to Danish conditions, will correspond to approximately 63 saved human lives.
Criticism point 3: The studies cannot be compared nor transferred to Danish conditions
Jonas Herby believes that the studies are comparable.
»They analyse the relationship between lockdown and the number of deaths due to COVID-19. This include different country groups and time periods, but they also do that in, for example, medical experiments,« he says.
The scientists, Videnskab.dk has spoken to, point out that differences between countries - and especially how different countries timed their respective lockdowns - create variations that are too great to be comparable.
Jonas Herby and his co-authors attach less value to timing and have excluded studies that look at the importance of timing because, according to them, they 'will systematically overestimate the effect of lockdowns’.
To the criticism of transferring the figures to Danish conditions, Jonas Herby says:
»It is an average, and there can be all sorts of reasons why it should be a little different in Denmark. It can be more or less. An estimate, I think one would say it is. We also know that smoking is extremely dangerous, but it is difficult to say whether it shortens the individual's life by 5, 10 or 15 years,« he says.
»I am also surprised that Heltberg and others so categorically postulate that lockdowns have saved a lot of people in 2020. What evidence do they have for that?«
Criticism 4: The evidence is too weak for such a conclusion
Videnskab.dk asks Jonas Herby whether there has been too sharp an angle on the effect of lockdown, when he writes in the comment that 'lockdown - before the vaccine rollout - prevented almost five deaths. Five. Total'.
»What I write in Berlingske is that if you transfer that number (the reduction of 0.2 percent, ed.) to Danish conditions, it is equivalent to saving five people. But I have obviously not claimed that it is exactly five. That's because it's a very small number. I cannot really see that it is a problem, to be completely honest,« he says.
He has seen that Berlingske has been criticised for overselling the message, says Jonas Herby, but »when you look at all the other things that have been brought up in the media,« it is not the case according to him.
It sounds as if you are saying that when others can write things that may/may not be based on research and that do not hold up, then you can do it too?
»Not at all. I'm just trying to defend Berlingske's headline. That they wrote so very categorically: Lockdown has saved five lives or something like that.«
It is also written very categorically in the commentary itself, and you are standing up for it, right?
»Definitely. But I cannot get upset about that. I write very clearly what I have done - that if you transfer it to Danish conditions, you will save five lives,« says Jonas Herby.
