Reader Comments

Post a new comment on this article

Clarification on Study 3

Posted by kimmoeriksson on 18 Sep 2013 at 10:30 GMT

As the authors of this paper we would like to offer some clarification on Study 3, in which participants were asked for their judgments of the validity of some findings in a published study on economic inequality by Norton and Ariely. After their first judgment, participants were given information about contradictory evidence from a study by Eriksson and Simpson.

In our paper we did not include the complete information that was given to participants, and it may therefore not be clear to a reader of the paper in what sense the original findings were contradicted by the new evidence. We therefore want to underscore that the replication study replicated the original results when using the same measure of economic inequality. The contradiction lay in that another measure of economic inequality gave different results, suggesting that (at least) one of the measures lacks validity. The authors of the replication study argued that data on inequality across different domains pointed to a specific problem with the validity of the original measure. See the cited papers for the various arguments in this debate.

The main aim of Study 3 was to see whether respondents’ judgments of the validity of the original findings depended on in which journal they were told the contradictory evidence was published.

The complete instructions to this study are included below.

COMPLETE INSTRUCTIONS TO STUDY 3
Evaluation scale
We are interested in how you judge the credibility of findings in social science research. Answers are given on a subjective likelihood scale from 0% to 100%. Thus,
0% means I am utterly convinced the finding is invalid
100% means I am utterly convinced the finding is valid.
________________________________
Evaluation #1
A couple of years ago a study was conducted, which can be summarized as follows:
• A nationally representative sample of Americans first estimated the current distribution of wealth in the United States, then constructed distributions with their ideal level of inequality.
• Finding: Americans dramatically underestimate the current level of wealth inequality in United States. Moreover, their ideal wealth distributions are nearly egalitarian.
This study, by researchers of distinction in the field, was published in one of the most prestigious journals in the field.
My subjective likelihood that this finding is valid is: __ %
Please provide some motivation in the text box below.

________________________________
Evaluation #2
Last year, other researchers conducted a study which seemed to contradict the previous study:
Using another question than the previous study, the researchers found
• that Americans did not tend to underestimate current levels of inequality, nor did they construct distributions that were nearly egalitarian.
The researchers argued that their question had higher validity than the question used in the previous study and concluded that the findings of that study were invalid.
This paper was published in {MANIPULATED journal}.
My subjective likelihood that this second, contrary, finding is valid is: __ %
My subjective likelihood that the original finding is valid is now: __ %
Please provide some motivation in the text box below.

________________________________
Evaluation #3
Here we present more detailed information about what the two studies actually did. This may of course affect your evaluation of the credibility of their findings.
The original study asked questions on the following "Percent format":
• What percent of the United States’ total wealth is [should be] controlled by the richest 20% of Americans? (The same question was then asked about the 2nd richest 20%, etc., down to the poorest 20%.)
The second paper suggested that respondents may find it extremely difficult to estimate United States' total wealth and percentages of it. Instead of actually using their knowledge about rich and poor households in such estimations, respondents might answer these questions by resorting to intuitions about percentages in general. If this is the case, then findings using this type of question lack validity. To relieve respondents of the computation of percentages of total wealth, the authors of the second paper instead suggested the "Average format":
• What is [should be] the average household wealth, in dollars, among the 20% richest households in the United States? (The same question was then asked about the 2nd richest 20%, etc.,)
The second paper reported a replication study where respondents were asked questions both on "Percent format" and "Average format". Answers to questions on the "Percent format" were similar to those obtained in the original study. Indeed, answers tended to be similar even if questions were not about wealth but instead asked about the percentage of total teacher income received by the 20% highest income teachers, or about the percentage of total number of web page visits received by the 20% most popular web pages.
If these answers are valid, they indicate that Americans are generally out of touch with reality, because there is actually very little inequality in the distribution of teacher incomes, but extremely large inequality in the popularity of web pages. However, when respondents answered questions on the "Average format" instead, they correctly identified that inequality is small for teacher salaries, but huge for the distribution of wealth and for popularity of web pages. The authors of the second paper therefore concluded that the "Percent format" used in the original study gives invalid results, that is, results that do not reflect what people actually know about inequality.
After reading these details of the studies, my subjective likelihood that the second, contrary, finding is valid is now: __ %
After reading these details of the studies, my subjective likelihood that the original finding is valid is now: __ %
Please provide some motivation in the text box below.

No competing interests declared.