Wednesday, April 17, 2013

The Scholarly Defense of Austerity is Seriously Flawed, Included a Significant Excel Error

The main peer reviewed journal article that provides the academic credibility for proponents of austerity has serious flaws, including a mistaken Excel calculation. Mike Konczol:
In 2010, economists Carmen Reinhart and Kenneth Rogoff released a paper, "Growth in a Time of Debt." Their "main result is that...median growth rates for countries with public debt over 90 percent of GDP are roughly one percent lower than otherwise; average (mean) growth rates are several percent lower." Countries with debt-to-GDP ratios above 90 percent have a slightly negative average growth rate, in fact.

This has been one of the most cited stats in the public debate during the Great Recession. Paul Ryan's Path to Prosperity budget states their study "found conclusive empirical evidence that [debt] exceeding 90 percent of the economy has a significant negative effect on economic growth." The Washington Post editorial board takes it as an economic consensus view, stating that "debt-to-GDP could keep rising — and stick dangerously near the 90 percent mark that economists regard as a threat to sustainable economic growth." 
Is it conclusive? One response has been to argue that the causation is backwards, or that slower growth leads to higher debt-to-GDP ratios. Josh Bivens and John Irons made this case at the Economic Policy Institute. But this assumes that the data is correct. From the beginning there have been complaints that Reinhart and Rogoff weren't releasing the data for their results (e.g. Dean Baker). I knew of several people trying to replicate the results who were bumping into walls left and right - it couldn't be done.

In a new paper, "Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff," Thomas Herndon, Michael Ash, and Robert Pollin of the University of Massachusetts, Amherst successfully replicate the results. After trying to replicate the Reinhart-Rogoff results and failing, they reached out to Reinhart and Rogoff and they were willing to share their data spreadhseet. This allowed Herndon et al. to see how how Reinhart and Rogoff's data was constructed.

They find that three main issues stand out. First, Reinhart and Rogoff selectively exclude years of high debt and average growth. Second, they use a debatable method to weight the countries. Third, there also appears to be a coding error that excludes high-debt and average-growth countries. All three bias in favor of their result, and without them you don't get their controversial result. Let's investigate further:

Selective Exclusions. Reinhart-Rogoff use 1946-2009 as their period, with the main difference among countries being their starting year. In their data set, there are 110 years of data available for countries that have a debt/GDP over 90 percent, but they only use 96 of those years. The paper didn't disclose which years they excluded or why.

Herndon-Ash-Pollin find that they exclude Australia (1946-1950), New Zealand (1946-1949), and Canada (1946-1950). This has consequences, as these countries have high-debt and solid growth. Canada had debt-to-GDP over 90 percent during this period and 3 percent growth. New Zealand had a debt/GDP over 90 percent from 1946-1951. If you use the average growth rate across all those years it is 2.58 percent. If you only use the last year, as Reinhart-Rogoff does, it has a growth rate of -7.6 percent. That's a big difference, especially considering how they weigh the countries.

Unconventional Weighting. Reinhart-Rogoff divides country years into debt-to-GDP buckets. They then take the average real growth for each country within the buckets. So the growth rate of the 19 years that the U.K. is above 90 percent debt-to-GDP are averaged into one number. These country numbers are then averaged, equally by country, to calculate the average real GDP growth weight.

In case that didn't make sense, let's look at an example. The U.K. has 19 years (1946-1964) above 90 percent debt-to-GDP with an average 2.4 percent growth rate. New Zealand has one year in their sample above 90 percent debt-to-GDP with a growth rate of -7.6. These two numbers, 2.4 and -7.6 percent, are given equal weight in the final calculation, as they average the countries equally. Even though there are 19 times as many data points for the U.K.

Now maybe you don't want to give equal weighting to years (technical aside: Herndon-Ash-Pollin bring up serial correlation as a possibility). Perhaps you want to take episodes. But this weighting significantly reduces the average; if you weight by the number of years you find a higher growth rate above 90 percent. Reinhart-Rogoff don't discuss this methodology, either the fact that they are weighing this way or the justification for it, in their paper.

Coding Error. As Herndon-Ash-Pollin puts it: "A coding error in the RR working spreadsheet entirely excludes five countries, Australia, Austria, Belgium, Canada, and Denmark, from the analysis. [Reinhart-Rogoff] averaged cells in lines 30 to 44 instead of lines 30 to 49...This spreadsheet responsible for a -0.3 percentage-point error in RR's published average real GDP growth in the highest public debt/GDP category." Belgium, in particular, has 26 years with debt-to-GDP above 90 percent, with an average growth rate of 2.6 percent (though this is only counted as one total point due to the weighting above).
As Dean Baker asks, just exactly how much of the world's unemployment crisis can we attribute to the mistakes of these authors? But also:
If facts mattered in economic policy debates, this should be the cause for a major reassessment of the deficit reduction policies being pursued in the United States and elsewhere. It should also cause reporters to be a bit slower to accept such sweeping claims at face value.
Ideally this would lead to a new look at all of these papers and the peer review process at journals. You simply can't imagine any other science where errors of this magnitude would have gone ignored. Something else that can't be ignored is the incentives. This paper made the authors famous in the policy making world, and famous for the right reasons on the side of it that has all the money. There is a tremendous benefit to your career if you write academic papers that back up the solutions our elites have already endorsed. We have no way of knowing if these incentives caused the authors to fudge their data or leave out countries that contradicted their thesis, but you damn sure have to wonder.

Either way, these monsters are responsible for human suffering by giving it's peddlers intelectual backing, which considering the success rate of austerity in generating economic growth (0 for history), is not surprising in the least.

As Dean says, this should be the prefect time to point out the insanity of our president and congress deciding to purposely hurt the economy, but we all know (just like R+R's research) this whole austerity endeavor has never been about the facts. 


  1. "You simply can't imagine any other science where errors of this magnitude would have gone ignored"

    Ignored... I don't know. Occur? Are published? Persist in the literature/public consciousness for years? I don't have to imagine. Shit happens all the time. Remember the "link" between autism and vaccines?

    "It should also cause reporters to be a bit slower to accept such sweeping claims at face value"
    Reporters should be slower to accept a lot of things at face value. Just look at the coverage of the Boston marathon bomb investigations.

  2. The link between autism and vaccines were published in one journal, and I'm pretty sure they printed a retraction. Of course the damage was done because people continued to cite the article.

    Though, the point stands errors are published at an alarming rate. Still, someone should have been suspicious when they refused to share their data after they published.

  3. Two other issues here are the failure of the peer review process and the misconception that all journals and the quality of the work published in them are equal (particularly in the vaccine/autism example). It seems to me that R+R should've been pressed harder by their reviewers to justify the data that they chose to exclude or at the very least comment on how the inclusion of those data points may or may not have changed the overall result. This solution doesn't get around the issue of their "coding error," and the only way to shine a light on that error would've been for the researchers trying to replicate the result to repeatedly PUBLISH their contrasting findings. The way the publication system works today, those researchers probably would've had a difficult time publishing their "negative results" because of the presumed strength of the R+R analysis. Journals generally try to avoid publishing papers that aren't deemed novel enough, and performing the same analysis on the same or a similar dataset certainly falls into that category.

  4. Really interesting stuff guys, although I figured this is the type of discussion we might have with the number of people who read this blog that are very immersed in this world.

    I guess my main take is not that these errors can't happen in other disciplines, it's just that econ academia has more incentive to allow/push for this type of flawed research because (sorry to you all, because this is not right...) there is a shitload of money behind the right kind of research in econ, whereas in other academic disciplines not so much.