These are not the stats you’re looking for: Abel Brodeur on publication bias, economic research and the science of Uber

Berkeley Initiative for Transparency in the Social Sciences awards Abel Brodeur the Leamer-Rosenthal Prize for Open Social Science

During a ceremony at UC Berkeley on December 15th, 2016, Professor Abel Brodeur was awarded the Leamer-Rosenthal Prize for Open Social Science, Emerging Researcher. This cash award of 10,000 USD is in honor of Professor Edward Leamer (Economist- UCLA) and Professor Robert Rosenthal (Psychology – UC Riverside).

Professor Brodeur won this prize for his work “Star Wars” (American Economic Journal: Applied Economics) and two current projects on research transparency.

Congratulations!

*********************************************************************************

Let’s say you’re looking at survey results to help you make a decision. Does post-secondary education of a particular kind guarantee a higher salary? Are more people going to vote for Hillary Clinton or Donald Trump? If the results showed, for example, that 19 out of 20 students with a degree found higher-paying jobs, you’d likely be convinced of the correlation (and also pretty pleased). However, if that number were only 12 out of 20, you might not be as convinced. Researchers are aware that a study is more likely to be published if they show convincing and attractive results – which may encourage them to adjust their research data to show more favourable results.

And that is where Professor Abel Brodeur comes in. While pursuing his Ph.D. in economics, he found that nearly everyone was aware of this practice of massaging data”: “It was never something we discussed in class, and I found that there had been almost no research done on this in the field of economics.” Professor Brodeur, along with Mathias Lé, Marc Sangnier and Yanos Zylberberg, has recently published a paper entitled Star Wars: The Empirics Strike Back, examining publication and research bias.

The paper examined the p-values of hundreds of economic studies published in major journals between 2005 and 2011 and analyzed their characteristics. The p-values exhibited a pattern suggesting that lower values had been adjusted to show clearer, more convincing results.

“Unfortunately,” says Brodeur, “you might lose the validity of the study by trying to make it more appealing for publication.”

Brodeur acknowledges that this is common practice, and isn’t out to condemn researchers. Hence, the eye-catching Star Wars reference in the title. “We’ve gotten a lot of attention on the paper just based on the title – this is a serious topic, but that was our way of lightening it up, saying we aren’t blaming economists. They’re just responding to incentives to publish. Besides, researchers have a sense of humour. We’re a bunch of geeks.”

Although some editors are now encouraging researchers to submit negative results (i.e., less convincing results), professor Brodeur admits he is not expecting an overnight revolution in publication and research practices. “It’s going to take a lot of time to change this. Norms are hard to change. But, we are seeing the emergence of new practices such as pre-analysis plans, where the entire study is laid out beforehand, so the results being good or bad becomes irrelevant. We’re just trying to force researchers to be more honest with their results, but it’s going to take a while.”

In the meantime, Professor Brodeur has plenty of other projects to keep him busy. Aside from his research on prostitution in Thailand and the economic effects of terrorism, he’s also examining the way in which Uber is solving a classic problem in economics.

“There are few well-known papers that examined taxis in New York. The hardest time to find a taxi is when it’s raining, both because of high demand and the fact that taxi drivers don’t like to work in the rain. Taxi drivers are not compensated to work in these conditions since they cannot adjust their prices. But, with the emergence of Uber, for which prices can be adjusted when demand increases, it’s now easier to get a ride in the rain. Uber drivers are willing to work in the rain because of surge pricing, and it’s thus Uber made it easier to get a ride in the rain in New York.”

To economics students beginning their research, Brodeur says, the world is at their fingertips. “For any kind of topic you’re interested in, the data are out there. Whether you want to research systems of government, prostitution, whether education matters…the data are out there. Younger researchers have the possibility to study empirically any kind of economic questions, just because of the volume of data online. You can research anything you want to.”

Back to top