Economists Behaving Badly: Proving Epidemiology is Much Harder than It Looks

And why it could be very bad for public health

“The only function of economic forecasting is to make astrology look respectable.”
– JK Galbraith

COVID, it seems, has the bizarre neurologic effect of making everyone believe they are epidemiologists. Among the demographic groups most susceptible to this delusion are those with statistical training in other fields. Economists it seems, are particularly vulnerable. Epidemiology looks simple to these quants. When they try to do it, however, they prove one thing definitively.

It's not.

Apparently, when you are an economist with a searing epidemiological insight about a new pandemic, you immediately seek to publish your theory in a journal. Their preferred biomedical journal appears to be the Wall Street Journal. That is where two economists, Neeraj Sood and Jay Bhattacharya went with separate opinion pieces in the early days of the pandemic to publicly announce their suspicions that COVID might be far less dangerous than the public health community believed. From there, they went on to conduct two highly influential and wildly flawed studies to prove their point.

This would be nothing more than a piece of pandemic history, something to be reviewed as we seek to learn the public health lessons of COVID, but for two things. First was that, during the pandemic, those who preferred policies based on the belief that COVID was a relatively mild disease seized on these studies and amplified them without regard to their quality. Second is that one of those economists has just been picked to lead the NIH.

To be clear, economists have a critical role to play in assessing the impact of public health interventions. Travel restrictions, school closures and constraints on businesses and public gatherings all have significant social and economic impacts. Decisions about public health interventions must be made in the context of those costs. However, the essential public debate over the right balance of these costs and benefits requires weighing financial costs against human morbidity and mortality and raises daunting ethical questions. The one thing that makes the debate easier is evidence that the benefits of public health interventions are smaller than believed, because either the interventions are less effective, or the disease is less severe than commonly believed. These were exactly the conclusions of many of the epidemiological studies attempted by economists.

Economists as Epidemiologists

Sood and Bhattacharya collaborated on the first such study , perfectly demonstrating the pitfalls in assuming one can treat public health data and financial data interchangeably. The authors used market survey techniques to assemble what they believed was a representative sample of the population of Santa Clara County. The problem was that they were not doing a market survey. They were collecting blood samples to test for antibodies to SARS-COV2. With no one on the team who had ever done a seroprevalence study, they failed to account for the motivation required to drive across Santa Clara County to give a blood sample. If people who have felt ill are more likely to do so, they have a problem. It now seems clear that they dramatically overestimated the rate of undiagnosed COVID infections by a factor of three or more. They conducted a similar, second study in LA County with equally flawed methodology and similar results.

Those studies fueled the belief that the public health response to COVID was overblown, gave rise the protection-by-infection strategy of the Great Barrington Declaration, and laid the foundation for resistance to everything from masks to vaccines.  Their results were widely interpreted to mean that COVID is no worse than the flu. This comparison was not at all arbitrary. Creating the illusion that the two diseases were comparable eliminated the need for a debate about the need for non-pharmaceutical interventions (NPI’s) such as travel restrictions or business closures. We, as a society, had implicitly decided that we did not need any NPI’s to manage COVID, since our response to the flu included only a voluntary vaccination.

Bhattacharya and Sood are not alone in presuming the shift from epidemiology to economics is seamless. Brown University economist, Emily Oster became the champion of school reopening, conducting assessmentsof the impact of masking, distancing, and school closures on COVID incidence with no pre-COVID experience in epidemiology. British economist, Sourafel Girma, used machine learning models to conclude that the benefits of COVID vaccination were “relatively small”. The fact that he published the results in European Economic Review suggests that a substantial element of the community of economists buys into the conceit that epidemiology is simply an extension of economics.

Estimates from Europe, France, and the United States suggest that NPI’s prevented hundreds of thousands of COVID deaths, even when one accounts for deaths related to their economic impact. In addition, vaccines are estimated to have cut deaths by more than 50% in the US. The minimization of the risk of COVID and/or the effectiveness of interventions seems likely to have costs hundreds of thousands of lives by reducing implementation of and eroding compliance with interventions. Failure to fully vaccinate alone is estimated to have cost two to three hundred thousand lives in the US. The impact on hospitalization and morbidity would be many multiples of these estimates.

We All Make Mistakes

The Santa Clara Study mentioned above, concluded that 0.17% of those with COVID-19 infections would die from the disease. If that were corrected, the 1.2 million COVID deaths in the US would require over 700 million infections, even without vaccination. Accounting for the more than three-fold reduction in IFR after vaccination would put this number over two billion cases in a country with 330 million people. Last summer, I had the opportunity to ask Bhattacharya in person if these numbers made him reconsider his conclusions. He insisted that they had been exactly right in their seroprevalence studies.

Celebrated economist, JK Galbraith, once said, “There are two kinds of economists: those who don't know the future, and those who don't know they don't know.”  Bhattacharya may represent a third type of economist, those who refuse to acknowledge that they didn’t know even after reality has proven them wrong.

All of this suggests that a cadre of economists conducted poorly designed epidemiological studies that may have had a major negative impact on global and US public health. As the Republicans roll out an alternate reality depicting Jay Bhattacharya as a brave voice railing against the subject matter experts in public health, we must loudly insist that they fully consider the impact of that advice and the far more devastating consequences that would have ensued had we followed it more closely.