Conceived and designed the experiments: CP FT ID PA CA. Performed the experiments: HA CP. Analyzed the data: HA CP. Contributed reagents/materials/analysis tools: HA CP PA. Wrote the paper: HA CP FT PA ID CA.
The authors have declared that no competing interests exist.
Peer review is the most widely used method for evaluating grant applications in clinical research. Criticisms of peer review include lack of equity, suspicion of biases, and conflicts of interest (CoI). CoIs raise questions of fairness, transparency, and trust in grant allocation. Few observational studies have assessed these issues. We report the results of a qualitative study on reviewers’ and applicants’ perceptions and experiences of CoIs in reviews of French academic grant applications.
We designed a qualitative study using semi-structured interviews and direct observation. We asked members of assessment panels, external reviewers, and applicants to participate in semi-structured interviews. Two independent researchers conducted in-depth reviews and line-by-line coding of all transcribed interviews, which were also subjected to Tropes® software text analysis, to detect and qualify themes associated with CoIs. Most participants (73/98) spontaneously reported that non-financial CoIs predominated over financial CoIs. Non-financial CoIs mainly involved rivalry among disciplines, cronyism, and geographic and academic biases. However, none of the participants challenged the validity of peer review. Reviewers who felt they might be affected by CoIs said they reacted in a variety of ways: routine refusal to review, routine attempt to conduct an impartial review, or decision on a case-by-case basis. Multiple means of managing non-financial CoIs were suggested, including increased transparency throughout the review process, with public disclosure of non-financial CoIs, and careful selection of independent reviewers, including foreign experts and methodologists.
Our study underscores the importance of considering non-financial CoIs when reviewing research grant applications, in addition to financial CoIs. Specific measures are needed to prevent a negative impact of non-financial CoIs on the fairness of resource allocation. Whether and how public disclosure of non-financial CoIs should be accomplished remains debatable.
Peer review of grant applications is the most widely used method for evaluating clinical research and has been used in industrialized countries to allocate research resources since 1950
Many charges have been made against peer review of grant applications
The French Ministry of Health funded a research project to assess the grant-application review process in France. As part of this project, we conducted a qualitative study to investigate the perceptions and experience of the various stakeholders in the process. Of the topics investigated, CoIs emerged as a significant concern among internal reviewers, external reviewers, and applicants.
The objective of this report is to describe the perceptions and experience of stakeholders regarding non-financial CoIs potentially affecting the grant-application review process, to describe the management of these non-financial CoIs, and to suggest possible solutions.
This qualitative observational study did not involve patients and written consent was not required. Anonymity and confidentiality of the interviews were guaranteed to all participants. An information sheet on the research objectives and confidentiality of study participation was read to each participant at the beginning of each interview. The participant was then asked to give oral consent and to allow audio recording of the interview. The Institutional Review Board of the Paris North Hospitals, Paris 7 University, AP-HP, approved the study protocol, including the information sheet and oral consent procedure (N° IRB00006477).
The French Health Ministry grant program for hospital-based clinical research (
For the national program, the president of the panel assigns an internal reviewer to each grant application. Then, each application is reviewed and rated by at least two external reviewers selected by the internal reviewer. Grant applicants do not know the identities of their internal or external reviewers; each external reviewer knows the identities of the applicant and internal reviewer but not of the other external reviewer; only the internal reviewer knows the identity of all four people involved. The internal reviewer writes a report on the grant application based on the assessment of the external reviewers. Then, the president of the panel and all the internal reviewers meet to discuss all grant applications. Based on the scientific quality of each project, funding decisions are made during this meeting.
For the Paris regional program, a board composed of the president, several internal reviewers, and the regional director of research assigns two internal and three external reviewers to each application. Grant applicants do not know who reviews their applications. In addition, internal reviewers are masked to external reviewers and each external reviewer is masked to the other external reviewers and to the internal reviewers. Thus external reviewers report anonymously to the internal reviewers. The applications given the highest ratings by the reviewers are then discussed by a panel composed of all the internal reviewers and the board.
In 2009, the national and Paris regional programs had 56 internal reviewers and asked 192 external reviewers to review applications submitted by 487 applicants. Eligibility criteria for participation in our study were as follows:. For internal reviewers, having been an active member of either the national or the Paris regional committee in 2008 or 2009; For external reviewers, having been asked, and having accepted or refused, to review at least one grant application for the national or Paris regional program in 2009 and having reviewed at least one grant application in the last three years; For grant applicants, having submitted at least one grant application to the national or Paris regional program in 2009.
All eligible internal reviewers were asked to participate, whereas external reviewers and grant applicants were selected by stratified randomization in order to obtain a broad spectrum of views. Stratification criteria were medical specialty and academic experience (i.e., junior vs. senior university-hospital physician), geographic location (Paris region versus rest of the country), type of stakeholder and, for applicants, rejection of a previous application. Interviews were conducted until the saturation point was reached, i.e., until additional interviews produced no new information
One of us (CP) attended the 2009 national and Paris regional committee meetings (a three-day meeting for French National PHRC and a two-day meeting for Paris Regional PHRC) to observe the interactions and to make notes about the debates. No audio recordings were obtained. The notes provided direct information on the review process, as opposed to the rationalized reconstruction of events provided by the reviewers in post hoc interviews.
During the observation sessions, we obtained access to the abstracts of the grant applications that were given to the panel members and discussed in the meetings. For the Paris regional program, we also had access to the reports by the external and internal reviewers.
We designed semi-structured interviews based on key themes identified from an analysis of the medical and sociological literature, French grant-application procedures, and official documents. The final interview guide included open questions on seven topics (
|
Employment status, past and current |
History of applicant/internal or external reviewer |
Reasons for being an applicant/internal or external reviewer |
|
Experience in PHRCs and other French institutions as applicant or as internal or external reviewer |
Experience in other grant applications as applicant or as internal or external reviewer |
|
Methodology and conception of peer review |
|
Perception of biases in the grant-application peer review process |
|
Strengths of the PHRC review process and of grant-application review in general |
Weaknesses of the PHRC review process and of grant-application review in general and specific question regarding conflicts of interests or other peer review weaknesses (perception and experience) |
|
Suggestions for improvement |
Specific questions about blinded peer review, compensation of reviewers, selection of peer reviewers |
|
Experience with personal applications: failures and successes |
Experience with other grant applications |
Interviews were conducted face-to-face at the participants’ workplace or by telephone (36 [37%] interviews) by two of us (CP, a science sociologist; and HA, an epidemiologist trained in semi-structured interviewing by CP). Neutrality of the interviews was ensured by the fact that neither interviewer was involved in the grant-application review process. The interviews began after the panel meetings, in June 2009, and ended in November 2010. They varied in length from 15 to 90 minutes (median, 31 minutes.
The interviews were audio taped and transcribed verbatim anonymously by an individual who was not otherwise involved in the study. Two interviewees refused to be recorded during the interview, and two recordings were of insufficient quality to allow transcription. The written notes taken during these four interviews allowed us to analyze them nevertheless. Biographical information for each participant was collected at the beginning of each interview.
The transcribed interviews were analyzed and coded by CP and HA, who used both case-oriented and variable-oriented methods
Of the 205 individuals who were asked to participate, 79 did not reply, 8 refused (usually because of lack of time), 1 was unavailable for participation, and 117 were included. Of those, 98 were interviewed, including 38 internal reviewers, 27 external reviewers, and 33 grant applicants; none declined participation after receiving oral information on the study. The remaining 19 individuals (2 internal reviewers, 9 external reviewers, and 8 applicants) either canceled or failed to attend the interview appointment.
N | N (%) | Internal reviewers, n (N=38) | External reviewers, n (N=27) | Grant applicants, n (N=33) | |
|
|
||||
30 – 39 | 8 (8) | 0 | 1 | 7 | |
40–49 | 38 (39) | 18 | 9 | 11 | |
50–59 | 33 (34) | 10 | 14 | 9 | |
60–69 | 9 (9) | 5 | 2 | 2 | |
Unknown | 10 (10) | 5 | 1 | 4 | |
|
|
||||
Male | 70 (71) | 29 | 19 | 22 | |
Female | 28 (29) | 9 | 8 | 11 | |
|
|
||||
Paris area | 65 (66) | 31 | 14 | 20 | |
Other regions | 33 (34) | 7 | 13 | 13 | |
|
|
||||
Medicine | 37 (38) | 14 | 7 | 16 | |
Surgery | 6 (6) | 2 | 3 | 1 | |
Methodology | 11 (11) | 9 | 1 | 1 | |
Psychiatry | 6 (6) | 1 | 2 | 3 | |
Obstetrics and gynecology | 4 (4) | 1 | 1 | 2 | |
Biology | 21 (22) | 8 | 9 | 4 | |
Anesthesia | 10 (10) | 2 | 3 | 5 | |
Other | 3 (3) | 1. | 1 | 1 | |
|
|
||||
Senior teaching-hospital physician | 79 (81) | 37 | 23 | 19 | |
Junior teaching-hospital physician | 3 (3) | 0 | 0 | 3 | |
Physician not working in a teaching hospital | 14 (14) | 1 | 4 | 9 | |
Other | 2 (2) | 0 | 0 | 2 | |
|
|
||||
0–2 | 13 (20) | 11 | 2 | - | |
3–5 | 21 (32) | 9 | 12 | - | |
> 5 | 19 (29) | 6 | 13 | - | |
Unknown | 12 (19) | 12 | 0 | - | |
|
|
||||
Yes | 10 (30) | - | - | 10 | |
No | 23 (70) | - | - | 23 | |
|
|
||||
Accepted | 14 (42) | - | - | 14 | |
Refused | 19 (58) | - | - | 19 |
During the interviews, most participants (79/98) spontaneously voiced concerns about non-financial CoIs and listed them ahead of all other biases such as those related to scoring, expertise, or notoriety. Industrial or financial CoIs were rarely mentioned by participants and were often viewed as minor or nonexistent in the PHRC review process:
All participants, n (N=98) | Internal reviewers, n (N=38) | External reviewers, n (N=27) | Applicants, n (N=33) | |
|
||||
Yes | 73 | 28 | 22 | 23 |
No | 25 | 10 | 5 | 10 |
|
||||
Disciplinary | 49 | 17 | 17 | 15 |
Rivalry or cronyism | 28 | 7 | 10 | 11 |
Geographic | 7 | 4 | 1 | 2 |
Academic | 4 | 0 | 1 | 3 |
|
||||
Yes (personal or not) | 60 | 14 | 22 | 24 |
Personal experience | 39 | 11 | 15 | 13 |
|
||||
CoIs viewed as unacceptable | 12 | 4 | 4 | 4 |
CoIs viewed as unavoidable | 15 | 6 | 5 | 4 |
|
||||
Always refuses to review | 9 | 5 | 4 | - |
Case-by-case decision | 8 | 2 | 6 | - |
Accepts to review while directing special attention to impartiality | 9 | 4 | 5 | - |
Disciplinary conflicts (i.e., competition among specialties or schools of thought) were unanimously listed as the most frequently occurring CoIs, ahead of personal or institutional rivalries, political considerations, and cronyism:
During the national PHRC meeting, CP noted a strong reaction of the group against some of the internal reviewers who defended their disciplines too strenuously. The president pointed out to an internal reviewer that not all the projects he/she had reviewed could be perfect.
Rivalry or cronyism was mentioned by both the applicants and the reviewers.
Geographic and (4) academic CoIs were particularly likely to occur in competitions among universities or between the Paris region and the rest of France and were less often mentioned by the participants.
Applicants could not formally prove the existence of non-financial CoIs in the grant-application peer review process, but one-third of them (13/38) reported having personal experience with such CoIs. Their suspicion that non-financial CoIs had affected the review process originated occasionally in personal convictions, interpretations, and hearsay and more often in discordances between reviewers’ reports.
Applicant 29 suspected that an idea was stolen from a previous application he had submitted:
While non-financial CoIs were considered either unacceptable or unavoidable by the various stakeholders, opinions about the feasibility of preventing CoIs were more contrasted. Some interviewees were fatalistic (
Other participants considered that CoIs were too variable in nature to be properly managed:
Interestingly, even when CoIs were suspected, they were not always perceived as important by the internal reviewers.
In addition, external reviewers had no knowledge of the reporting and management of CoIs during the grant-application review process. More generally, most of them were unaware of how their reviews were considered in the final assessment:
We found that several mechanisms were used to limit CoIs, although they were not explicitly described in an official policy statement. Grant applicants could list the names of experts they did not want as reviewers of their projects. Experts could, but were not mandated to, refuse to review projects they felt might involve CoIs. Each grant application was reviewed by three (national PHRC) or five (Paris regional PHRC) internal and external reviewers, whose names were masked to the applicants. Moreover, the panel members were chosen from a variety of geographic areas and specialties to ensure that the panel represented the diversity of the grant applications. Grant applications were discussed collectively during the panel meeting, and panel members were free to voice their opinions, although the discussions were influenced by individual factors such as effectiveness in public speaking, scientific expertise, desire to share personal convictions, and willingness to risk expressing disagreement. The panel president ensured that internal reviewer(s) who were involved with an application as investigator were not present when the application was discussed. During the panel meeting, the president played an important role in identifying and managing CoIs, for example by ensuring that internal reviewers did not place excessive emphasis on applications in their own disciplines to the expense of those in other disciplines:
Personal experience with non-financial CoIs was reported by 26 reviewers. Based on the interviews of these reviewers, approaches to CoI management were divided into three evenly represented categories.
First, some reviewers routinely refused to review grant applications if they felt they might be biased in favor of or against the applicant:
Second, some reviewers felt that non-financial CoIs were unavoidable and should be managed by conducting the reviews in a strictly impartial manner. They only refused to review applications for which they felt unable to remain impartial:
The third group of reviewers adopted a case-by-case approach to decide whether or not to review each application according to their subjective understanding of potential non-financial CoIs. For example, two reviewers said that they refused reviews if they were biased against the applicant or project, but not if their bias was positive:
Among the numerous suggestions for improving the peer review process (
Suggestions | All participants, n (N=98) | Internal reviewers, n (N=38) | External reviewers, n (N=27) | Applicants, n (N=33) |
No improvements are possible | 7 | 4 | 1 | 2 |
Masking applicant’s identity | 26 | 7 | 7 | 12 |
Careful selection of independent reviewers | 21 | 6 | 6 | 9 |
International reviewers | 18 | 10 | 5 | 3 |
Possibility for an applicant to challenge a reviewer | 6 | 4 | 0 | 2 |
Open peer review | 2 | 2 | 0 | 0 |
Enhancement of general transparency procedures | 17 | 5 | 5 | 7 |
Interactions with the grant applicant during the reviewing process | 12 | 1 | 4 | 7 |
Public disclosure of conflicts of interest | 6 | 4 | 1 | 1 |
Training of peer reviewers | 1 | 0 | 1 | 0 |
Regarding the overall reviewer selection process, many interviewees voiced major concerns about reviewer selection, especially for specific specialties or research topics:
Another suggestion was to give the applicants the opportunity to challenge the report of the reviewers:
Improving transparency was also suggested:
Other suggestions were made, such as interactions between applicants and reviewers:
Disclosure of CoIs, particularly of a non-financial nature, was mentioned by many interviewees as an important transparency procedure that did not appear to be part of PHRC policy:
Finally, one reviewer suggested training of peer reviewers in the identification and management of CoIs and improved uniformity of the peer review process
Direct observation of panel meetings and interviews with various stakeholders identified non-financial CoIs as a major concern of all parties involved in the process of academic grant-application review. Most of the interviewees spontaneously reported that non-financial CoIs were a major source of bias in the review process and had a greater influence than did financial or industrial CoIs. This high level of concern about non-financial CoIs was in striking contrast to the absence of a formal procedure for non-financial CoI disclosure and management. Although the various stakeholders usually felt that non-financial CoIs were so protean and ubiquitous as to be unavoidable, they also felt that peer review was the best possible evaluation method. The applicants were generally prepared to accept that the review process was not perfect. Among the suggested methods for CoI management two may deserve particular attention, namely, the careful selection of independent reviewers, particularly from other countries and among methodologists; and increased transparency throughout the review process, including a requirement to disclose non-financial CoIs
To our knowledge, this is the first empirical study that used observational scientific methods to investigate non-financial CoIs potentially affecting the grant-application review process.
Few previous studies assessed the opinions of the various stakeholders in the grant-application review process
Non-financial CoIs are often listed by editors and scientists as important biases
Our study was not designed specifically to explore CoIs but addressed instead the overall academic grant-application evaluation process. Nevertheless, as most participants spontaneously mentioned CoIs, this point probably had little impact on information saturation in this qualitative study. Our focus on a single country may limit the external validity of our findings. It could be argued that the French system lacks transparency compared to those used in the UK (Medical Research Council) and US (National Institutes of Health), which are often used as models for European grant-awarding processes. This could be due to the fact that until recently most of the academic research conducted in France relied on permanent structural funds from the government
Efforts to minimize the impact of CoIs focus chiefly on disclosure, as shown in a recent study of several international grant organizations
In France, structural public funds available for research have diminished substantially in recent years, leaving a greater role for funding via grants. Consequently, specific measures designed to minimize the impact of non-financial CoIs are required to ensure trust in the grant-application review process and fairness of grant allocation
Proposals drawn from study results and review of the literature | Pros | Cons | Authors’ comments |
Masking of applicant’s identity | Requested by the majority of applicants | Useless according to some reviewers May be harmful (because the identity of the applicant provides information on the feasibility and chances of success of the research project) |
Studies of manuscript and grant-application reviews produced conflicting data |
Enhancement of general transparency procedures | Requested by the majority of applicants Might restore applicants’ trust in grant institutions | May be costly and time consuming | Grant institutions should provide more information about their process (via the Internet for example |
Public disclosure of Conflicts of Interests (CoI) | Requested by the majority of applicants May restore applicants’ trust in grant review institutions | Difficulty in defining non-financial CoIs |
Need to develop requirements for disclosure of non-financial CoIs |
Open peer review | Requested by a few applicants | May impact reviewers’ work and objectivity |
Further studies are needed to assess this method in grant review |
Interactions with the grant applicant during the reviewing process | Requested by some participants Would allow applicants to challenge the review of their project Already used in some grant institutions |
Could be costly and time consuming | Need to assess the impact and feasibility of this method in grant review |
Elimination of grant review | Bibliometrics to evaluate the applicants ability to successfully conduct useful research | Not requested by the reviewers or applicants Bibliometric methods have several limitations |
Need to assess the impact and the feasibility of these methods. |
Improvement of reviewer selection | Selection of international reviewers, for example with no or few CoIs | Difficulty in finding the best reviewer as “there is no such thing as the perfect reviewer" |
Need to recognize the importance of reviewers’ work |
Training of reviewers | Requested by a few external reviewers May increase recognition of reviewers’ work | May be costly or time consuming May have no impact on grant review |
Need to assess the impact and feasibility of this method in grant review |
First, mandatory CoI disclosure would considerably improve the transparency of the review process. CoI disclosure could be required of all internal and external reviewers and of applicants, allowing crosschecking of the information. In practice, most CoI disclosures are related to financial interests, notably with the industry
Second, transparency could be improved by giving applicants free access to reports by external and internal reviewers, as well as to the panel meeting discussions. Audio recordings or verbatim transcripts of the meetings may improve the objectivity of the review process and have been assessed in some grant organizations, including the NIH
A third means of improving transparency is open peer review, i.e., the unmasking of reviewers and applicants. This suggestion has generated considerable controversy
Fourthly, assignment of applications to reviewers also deserves attention as a means of minimizing non-financial CoIs. Masking of applicant identity is often suggested by applicants as a means of improving the objectivity of the review process. However, the feasibility of a research project may be difficult to assess without knowledge of which principal investigator and research group are involved. In manuscript submission to journals, the masking of applicants’ identities has been shown to reduce biases, particularly geographic biases and academic CoIs
Peer review, although often criticized, is the most widely used method of research grant allocation. Our results indicate the presence of non-financial CoIs in the grant-application peer review process used to allocate academic funds. We believe there is an urgent need to improve transparency, trust, and fairness, particularly by issuing uniform requirements for non-financial CoI disclosure.
There is still a paucity of data on the efficacy and quality of the grant-application peer review process
(DOC)
(DOC)
(DOC)
We thank all those who accepted to participate in our study, Claire Ughetto for transcribing the interviews, and Karen Brigham for reading the manuscript.