Reinhart, Rogoff, and Austerity

[zoogs]

LINK (article)

 

I'm unable to believe the summary of this piece.

 

A graduate student (Thomas Herndon) at UMass Amherst made waves when he turned an important study's "stylized facts" conclusion upside down by finding.....a formula error in an excel spreadsheet.

 

LINK (paper pdf)

 

How is this possible?

 

The original study by Reinhart and Rogoff claimed some sort of brick wall at 90% publicdebt-GDP ratio as far as GDP growth potential goes. But because they only averaged rows 30 through 44 instead of 30 through 49 by accident, they messed up the numbers. GDP growth for countries over the 90% ratio was +2.2%, instead of -0.1%.

 

Billed in this article as "one of the most politically influential economic papers of the last decade", the Reinhart/Rogoff paper was cited by proponents of austerity such as Paul Ryan and Tim Geithner and used as support for policy. The 90% threshold figure had apparently, taken center stage in the public discourse. Then the MSEXCEL screwup was caught.

 

By a graduate student. Doing an assignment for class. THREE YEARS later.

 

And only because he asked for (and eventually received) the dataset from Reinhart.

 

How in the world is this possible? How did the original paper come into circulation and common acceptance without a fully released data set or anyone who bothered checking the math? How can academia shield itself from such utter incompetency in the future, and how often has this happened, yet unnoticed, in the past?

[JJ Husker]

I don't want to turn this into a "faith" in the scientific method discussion but this example is exactly what I mean when I say it takes a certain amount faith to believe studies like this. Not many people are going to or even have the ability to check the math or calculations that lead to these types of widely accepted studies and conclusions. I would bet this one is nowhere near the lone exception. Pure science is one thing but when you venture into economics you can bet there is going to be some political influence even with the best intentions. It sure does seem this one should have been rooted out much sooner. Kind of scary in a way.

[strigori]

Its simply ridiculous that no one found the error sooner.

 

However, economics is hardly a true science. Too much room for interpretation, and too many people take correlation for cause and effect. And in some cases its arguing the chicken and the egg.

[zoogs]

Yeah, the social sciences are certainly a lot less exact. This wasn't mainly about interpretation though, just a straight up calculation error. Does peer review not have access to this data? Does the data itself never get published (it seems so), meaning we look at some charts and have to say, "Oh yeah, I'm sure they got those numbers right."

 

It all seems so easily correctable and something that should not be possible in academia.

[BigRedBuster]

I don't want to turn this into a "faith" in the scientific method discussion but this example is exactly what I mean when I say it takes a certain amount faith to believe studies like this. Not many people are going to or even have the ability to check the math or calculations that lead to these types of widely accepted studies and conclusions. I would bet this one is nowhere near the lone exception. Pure science is one thing but when you venture into economics you can bet there is going to be some political influence even with the best intentions. It sure does seem this one should have been rooted out much sooner. Kind of scary in a way.

 

 

Now I'm not sure which to actually believe. Do I believe RR was wrong and possibly politically motivated to give incorrect info and this kid is right, or do I believe RR is right and this kid's paper was politically motivated to swing opinion the other way?

[carlfense]

Now I'm not sure which to actually believe. Do I believe RR was wrong and possibly politically motivated to give incorrect info and this kid is right, or do I believe RR is right and this kid's paper was politically motivated to swing opinion the other way?

You don't have to rely on beliefs, you know . . . you can look at the data and the statements from Reinhart and Rogoff.

[Someone]

I don't want to turn this into a "faith" in the scientific method discussion but this example is exactly what I mean when I say it takes a certain amount faith to believe studies like this. Not many people are going to or even have the ability to check the math or calculations that lead to these types of widely accepted studies and conclusions. I would bet this one is nowhere near the lone exception. Pure science is one thing but when you venture into economics you can bet there is going to be some political influence even with the best intentions. It sure does seem this one should have been rooted out much sooner. Kind of scary in a way.

Actually, taking them on faith is the problem.

Reinhart et al continue to compound their errors. They are guilty of coming up with an idea and this looking for data to support that idea. They keep tossing aside good data that disproves their idea until they find a data point that agrees with them.

[BigRedBuster]

Now I'm not sure which to actually believe. Do I believe RR was wrong and possibly politically motivated to give incorrect info and this kid is right, or do I believe RR is right and this kid's paper was politically motivated to swing opinion the other way?

You don't have to rely on beliefs, you know . . . you can look at the data and the statements from Reinhart and Rogoff.

 

 

Where is the data?

[carlfense]

Now I'm not sure which to actually believe. Do I believe RR was wrong and possibly politically motivated to give incorrect info and this kid is right, or do I believe RR is right and this kid's paper was politically motivated to swing opinion the other way?

You don't have to rely on beliefs, you know . . . you can look at the data and the statements from Reinhart and Rogoff.

Where is the data?

You must not have read the link.

[zoogs]

What concerns me is that the original data was never released (at least the grad student had to email the original authors for it). So in the general case, if that's common practice, you can't simply look at the data, right?

 

The only way you'd get to is if you are a graduate student doing an assignment on replicating experiments and are lucky enough to get an email response with an attached .XLSX.

 

Reinhart and Rogoff do appear to be guilty of shoddy science, jumping at the chance to tout the conclusions they wanted to based on numbers they arrived at erroneously but wanted to find. Somewhere, someone along the line should have caught this. It shouldn't even have a chance of slipping through the cracks.

[zoogs]

Ha, per the Colbert Report I learned tow things I missed before:

 

-the original paper wasn't submitted to a peer-review journal

-apparently there was some economist who claimed they 'refused' to share their data initially.

 

And yet it was a highly influential paper cited by politicians from the US to the EU.

[BigRedBuster]

Now I'm not sure which to actually believe. Do I believe RR was wrong and possibly politically motivated to give incorrect info and this kid is right, or do I believe RR is right and this kid's paper was politically motivated to swing opinion the other way?

You don't have to rely on beliefs, you know . . . you can look at the data and the statements from Reinhart and Rogoff.

Where is the data?

You must not have read the link.

 

 

Actually, I did read the entire paper. I have gone back to it now three times and I don't see the spreadsheets with the data. Where am I missing it?

[carlfense]

Actually, I did read the entire paper. I have gone back to it now three times and I don't see the spreadsheets with the data. Where am I missing it?

pg. 14 footnote 9.

 

You should be able to follow the breadcrumbs from there.

[zoogs]

So the numbers themselves are public information you can find from various sources. But their spreadsheet where they tabulated the data and performed the calculations, that was not released. The UMass guy, Herndon, was able to look up the information publicly but couldn't reproduce the results and couldn't figure out why.

 

I feel, peer reviewed journal or not, those spreadsheets and other stuff like this, ought to be released if you're going to publish your research. Otherwise we're taking it on faith that you didn't do something stupid like mistype one of several hundred numbers, or I don't know, forget to sum up a few rows.

 

Something from which even Harvard economists are not, apparently, immune.

[carlfense]

So the numbers themselves are public information you can find from various sources. But their spreadsheet where they tabulated the data and performed the calculations, that was not released. The UMass guy, Herndon, was able to look up the information publicly but couldn't reproduce the results and couldn't figure out why.

 

I feel, peer reviewed journal or not, those spreadsheets and other stuff like this, ought to be released if you're going to publish your research. Otherwise we're taking it on faith that you didn't do something stupid like mistype one of several hundred numbers, or I don't know, forget to sum up a few rows.

 

Something from which even Harvard economists are not, apparently, immune.

Exactly right.