Author: Gunther Eysenbach
Position: Associate Professor, Senior Scientist
Institution: Universtity of Toronto, UHN
E-mail: geysenba@uhnres.utoronto.ca
Submitted Date: May 17, 2006
Published Date: May 17, 2006
This comment was originally posted as a “Reader Response” on the publication date indicated above. All Reader Responses are now available as comments.

The introduction of the article and two accompanying editorials [1-3] already answer Harnads questions why author, editors, and reviewers were critical of the methodology employed in previous studies, which all only looked at "green OA" (self-archived/online-accessible papers) (hint 1: "confounding") (hint 2: arrow of causation: are papers online because they are highly cited, or the other way round?). The statement in the PLoS editorial has to be seen against this background. None of the previous papers in the bibliography mentioned by Harnad employed a similar methodology, working with data from a "gold-OA" journal.

The correct method to control for problem 1 (multiple confounders) is multivariate regression analysis, not used in previous studies. Harnads statement that "many [of the confounding variables] are peculiar to this particular [...] study" suggests that he might still not fully appreciate the issue of confounding. Does he suggest that in his samples there are no differences in these variables (for example, number of authors) between the groups? Did he even test for these? If he did, why was this not described in these previous studies?

The correct method to address problem 2 (the "arrow of causation" problem) is to do a longitudinal (cohort) study, as opposed to a cross-sectional study. This ascertains that OA comes first and THEN the paper is cited highly, while previous cross-sectional studies in the area of "green OA" publishing (self-archiving) leave open what comes first -- impact or being online.

Harnad -- who usually carefully distinguishes between "green" and "gold" OA publishing -- ignores that open access is a continuum, much as publishing is a continuum [4]. Publishing in an open access journal is a fundamentally different process from putting a paper published in a toll-access journal on the Internet. In analogy, printing something on a flyer and handing it out to pedestrians on the street, and publishing an article in a national newspaper can both be called "publishing", but they remain fundamentally different processes, with differences in impact, reach, etc. A study looking at the impact of publishing a newspaper can not be replaced with a study looking at the impact of handing out a flyer to pedestrians, even though both are about "publishing".

Finally, Harnad says that "prior evidence derived from substantially larger and broader-based samples showing substantially the same outcome". I rebut with two points here.

Regarding "larger samples" I think rigor and quality (leading to internal validity) is more important than quantity (or sample size). Going through the laborious effort to extract article and author characteristics for a limited number of articles (n=1492) in order to control for these confounders provides scientifically stronger evidence than doing a crude, unadjusted analysis of a huge number of online accessible vs non-online accessible articles, leaving open many alternative explanations.

Secondly, contrary to what Harnad said, this study is NOT at all "showing substantially the same outcome". On the contrary, the effect of green-OA -- once controlled for confounders -- was much less than what others have claimed in previous papers. Harnad, a self-confessed "archivengalist", co-creator of a self-archiving platform, and an outspoken advocate of self-archiving (speaking of vested interests) calls the finding that self-archived articles are [...] cited less often than [gold] OA articles from the same journal "controversial". In my mind, the finding that the impact of non-OA is less than green-OA is less than gold-OA is less than green+gold-OA is intuitive and logical: The level of citations correlates with the level of openness and accessibility.

Sometimes our egos stand in the way of reaching a larger common goal, and I hope Harnad and other sceptics respond with good science rather than with polemics and politics to these findings. Unfortunately, in this area a lot more people have strong opinions and beliefs than those having the skills, time, and willingness to do rigorous research. I hope we will change this, and I reiterate a "call for papers" in that area [3].

References

1. Eysenbach G. Citation Advantage of Open Access Articles. PLoS Biol. 2006;4(5) p. e157.
OPEN ACCESS http://dx.doi.org/10.1371...

2. MacCallum CJ, Parthasarathy H. Open Access Increases Citation Rate. PLoS Biol. 2006;4(5) p. e176.
OPEN ACCESS http://dx.doi.org/10.1371...

3. Eysenbach G. The Open Access Advantage. J Med Internet Res 2006 (May 15); 8(2):e8
OPEN ACCESS: http://www.jmir.org/2006/...

4. Smith R. What is publication? [editorial]. BMJ 1999;318:142
OPEN ACCESS: http://bmj.bmjjournals.co...