Introduction

The role of Institutional Review Boards (IRBs), otherwise known as Human Research Ethics Committees (HRECs) or Research Ethics Committees (RECs), in biomedical research remains central to ensuring that research is designed and conducted in accordance with high ethical and scientific standards, that research participants are adequately protected from harms inherent in some research and that their dignity and rights are protected [1]. IRBs were first established by law in the US in the 1970s, partly in response to revelations about harmful research [2, 3]. While the functions of IRBs around the world and across institutions may differ slightly, their general aim is to review proposed research projects involving humans to ensure that the research is ethically justifiable along a number of dimensions before it commences. IRBs will, for instance, assess the potential harms of a proposed research project and ways to minimise such potential harms, the anticipated benefits (to society and/or to participating individuals), the vulnerability of the subject population, the recruitment process, the consent process, researchers’ experienced and expertise, potential conflicts of interest and, in some institutions for some proposals, other aspects of a proposal, such as study design. IRBs also have oversight responsibilities in relation to approved research as well as an educational function in some institutions to which they belong. (See for example: [4–7]).

In recent decades, the research landscape has significantly changed as a result of scientific advances, substantial diversification of research, and increases in the overall volume of biomedical research [8]. As a result, IRBs have made considerable efforts to refine and revise their processes to keep pace and, in many cases, to redefine their role in the research ethics process to extend beyond approval and oversight functions [9]. Nevertheless, research continues to require appropriate ethical oversight provided by IRBs, which, in the context of Singapore, is duly acknowledged in national laws and guidelines [5, 6].

Despite the central role IRBs play in ensuring the ethical conduct of research, it is recognized internationally that IRBs may not be functioning at the high level expected of them [10]; this may be due to the intricate operational facets which characterize them and, in part, due the financial constraints to which some are subject [8, 11], which can impact on the provision of adequate training for IRB members and therefore the quality of review processes and IRB functioning. Moreover, the relationship between IRBs and the research community is often strained and impacted by perceptions and expectations that the research community have of the role and function of IRBs [12–14], which have been shown to diverge from perceptions held by IRBs [15]. For example, researchers view IRB interventions as means of controlling them but IRBs view their involvement as helpful and accommodating; researchers may view IRBs as unfriendly and adversarial as they adopt a stance of interrogation when seeking justifications for aspects of research proposals; researchers may view IRBs as impeding research, as they sometimes forego certain studies for fear of having it rejected but IRBs appear to view this as an inevitable but unintended consequence of processes; researchers view IRBs as exerting considerable power but IRBs do not share this view because they see themselves as simply implementing research regulations; finally, researchers seem to lack an understanding of the research approval machinery which some IRBs view as the reason for delays and lengthy approval processes [15]. Researchers have a range of expectations given that their work is dependent on ethics clearance; they expect, for example, timely reviews, requisite IRB expertise, and IRB ability to appropriately assess ethical issues, including risks and benefits, rather than simply focusing on regulatory compliance [10, 16]. Insight into how researchers view IRB practices can be helpful in overcoming impediments to a good working relationship between the research community and IRBs.

Internationally, a number of empirical studies have examined how biomedical researchers perceive the IRB review process. Systematic literature reviews [10, 17] indicate that these studies vary considerably in terms of sampling characteristics, data collection procedures, survey tools, and analyses, making their results difficult to compare. However, one survey instrument that has enjoyed repeated uptake is the IRB-Researcher Assessment Tool (IRB-RAT), which measures researchers’ perceptions of how 45 IRB characteristics and functions relate to their ideal IRB and their actual IRB [18]. Ideal IRB is to be understood as an IRB that displays features that are most central to enabling researchers to achieve their best work while actual IRB is to be understood to relate to IRB features that researchers feel their IRB actually displays. An initial study of 886 US biomedical and social behavioral scientists [19] generated a baseline ‘US National Validation Sample’ (USNV Sample) to validate the IRB-RAT. Subsequent studies have used the IRB-RAT to capture the perceptions of medical researchers at specific institutions [20, 21] and to guide quality improvement of IRB functioning by identifying the areas in which perceived actual IRB processes particularly fall short of the identified ideal processes [22, 23].

However, to date there has been no systematic attempt to identify and measure how researchers perceive IRB functions and characteristics in Singapore. Indeed, so far only one published study has used the IRB-RAT to measure researchers’ perceptions outside the US [24]. Notably, this is also the only published study originating from the broader Asian region that explicitly explores researchers’ perceptions of IRBs. While there are published studies on IRB processes and characteristics in Asian countries, these tend to describe only the structure and composition of IRBs [25–27] or only measure their operational efficiency [28]. Our study addresses this gap and uses the IRB-RAT to measure how IRB users in Singapore rate their IRB’s review processes and characteristics and what IRB characteristics they would view as improving the quality of IRB functioning. In addition to the IRB-RAT, which collects information about perceptions of the particular IRB researchers are most familiar with, our survey also collected information about the general perception of the impact of IRB review in various areas, such as maintaining public trust in research.

Our study contributes to a general understanding of perceptions around IRB functions and characteristics in Singapore. It aims to generate preliminary evidence that may be useful in guiding institutions and policymakers to improve IRB functioning in Singapore, and to compare Singaporean findings with international findings. The study also provides a platform for further analyses and research to develop a comprehensive understanding of the Singaporean research ethics landscape with the aim of generating more specific suggestions for improvements to local IRB processes and functions in the future. Finally, through the use of the IRB-RAT, the study allows us to compare Singaporean findings with international findings.

Methods

Data was collected via an anonymous online survey incorporating the IRB-RAT. The survey was launched on the e-Survey platform of the National University of Singapore (NUS) and IP addresses were not collected.

Survey tool

The survey comprised a welcome message and information about the research followed by four sections, as illustrated in Table 1. The full survey can be found in S1 Appendix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of survey structure.

https://doi.org/10.1371/journal.pone.0241783.t001

Section 1 screened potential participants to ensure that they were either a PI/Co-I on a biomedical research project that has undergone IRB review in the past 12 months, or research support staff who had been substantially involved in the drafting of a biomedical research project that had undergone IRB review in the past 12 months. This ensured participation only by individuals who had recent first-hand experience with IRB submissions. Section 2 consisted of questions about the general impact of IRB review in four areas. Section 3 of the survey comprised the core component and was an adaptation of the IRB-RAT, a validated tool serving as a proxy for the quality of IRB functions and characteristics [23]. The survey prompted respondents to indicate on a 7-point Likert-type scale how well 45 statements about IRB characteristics and functions (‘items’) described their ideal and their actual IRB [18]. If participants had experience with more than one IRB, they were asked to respond about only one of the IRBs.

Three variations were made to the original IRB-RAT. First, minor adjustments were made to the wording of some statements to ensure they were applicable to the local context, e.g. ‘federal regulations’ became ‘relevant laws or national guidelines’. The second variation concerned the structure of the published tool. Instead of asking participants to first give views about their ideal IRB for all 45 questions and, then, in a second round asking for their views about the actual IRB rating across all 45 items, we converted this to a “single-pass” version which contained all 45 items and required consecutive responses, one about the actual and one about the ideal IRB. The rationale for this change was that the single-pass version would decrease the cognitive load for participants and the time required to complete the survey. The original IRB-RAT, developed by Keith-Spiegel and Koocher allowed for this variation but had not implemented it [29]. The third variation concerned the introduction of the option “I don’t know/ I have no experience”, for responses relating to actual IRBs. This option was included because we judged that researchers and research support staff could not possibly be personally familiar with some internal IRB processes (e.g. “An IRB composed of members who arrive at meetings well-prepared”). We notified the developers of the IRB-RAT that we would be employing their tool and informed them of the modifications that we intended to make.

Section 4 of the survey collected non-identifying demographic data relating to institutional affiliation, IRB membership, type of research engaged in, years of involvement in biomedical research and involvement in international research. The research was approved by the National University of Singapore IRB (S-18-289E) and participants were not reimbursed for their involvement.

Results

Among 149 individuals who answered at least 1 screening question, 117 individuals met the inclusion criteria. Ninety-eight individuals started and completed the questions on the general perception of the impact of IRB review and 82 individuals then progressed to start the IRB-RAT. Seventy-one completed it, with the remaining 11 providing responses to only some of the items (Table 2). All responses to the IRB-RAT survey questions were included in the final analysis regardless of whether the respondent completed the rest of the survey. Of the 35 RIs approached, 16 (42%) are represented in our sample with at least one response (S1 Table). The response rate could not be calculated as there was no information on how many researchers and research support staff had received the invitation to participate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Responses received per survey section.

https://doi.org/10.1371/journal.pone.0241783.t002

Demographic information was only available for 71 respondents who completed the survey since the section was located at the end of the survey. Table 3 lists these demographic details and shows that among the respondents, 53 (75%) were PI/Co-Is while the remaining 18 (25%) were research support staff. The vast majority (93%) were not IRB members. A large proportion of participants (58%) indicated involvement in research involving data only. Respondents could choose multiple affiliations and the most common affiliations clustered around two organisations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Characteristics of survey respondents.

https://doi.org/10.1371/journal.pone.0241783.t003

Table 4 shows the results from Section 1 of the survey, which asked participants about their general perceptions of the impact of IRB review processes. The 98 participants who completed this section generally thought that the IRB review process has a very positive/positive influence on the following four areas examined: the overall quality of research (59%); establishing/maintaining public trust in research (74%); the protection of research participants (86%); and the scientific validity of research (50%). Few respondents stated that the IRB review process has a negative/very negative impact on these areas. However, a considerable percentage of respondents viewed IRB review processes as having no impact, particularly in relation to contributing to the scientific validity of research (45%) and on the overall quality of research (29%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. General perception of the IRB review process.

https://doi.org/10.1371/journal.pone.0241783.t004

The ideal and actual ratings of each IRB-RAT item are summarized in Table 5. The items are ranked by the mean ratings for the ideal IRB in descending order, highlighting which items respondents in our study regarded as most and least important. A wide range of IRB features was viewed as crucial for an IRB, the most important of which included: timely reviews, the promotion of participants’ rights in conjunction with facilitation of research, good collaborative relations with researchers, IRB responsiveness, the absence of reviewer and committee biases, IRB accountability, procedurally and legally knowledgeable members and good record-keeping.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Ideal and Actual ratings, ranked in descending order by mean ideal ranking.

https://doi.org/10.1371/journal.pone.0241783.t005

There were fewer responses in the actual dimension than in the ideal dimension for all items except item 33 (“An IRB that is open and pleasant in its interactions with investigators”), as participants could select the option “I don’t know/ I have no experience” for actual (but not for ideal) items. For some items, the number of participants who selected this option was significant. For instance, 41/71 respondents who answered the question did not know whether the members of the IRB that they were most familiar with arrived at meetings well prepared (item 45) and 37/73 respondents who answered the question did not know whether the IRB members would abstain from evaluating protocols whenever a real or apparent conflict of interest arose (item 21).

Fig 1 below plots the mean ideal and mean actual ratings. The importance of an ideal IRB possessing an item was rated higher than the actual IRB performance on all items except for item 39 (“An IRB that is composed of more than one lay person”). Item 39 also had the lowest mean ideal rating. This suggests that respondents may not see much value in having lay people on IRBs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Mean ideal and actual ratings for each IRB-RAT item.

https://doi.org/10.1371/journal.pone.0241783.g001

Ideal and actual scores for each of the 45 IRB-RAT items were compared, and the results show that all items in our study had significantly higher mean ideal scores than the mean actual scores, except items 9 and 39 for which the difference was not statistically significant (S2 Table contains further details).

We compared our ideal scores with those in Keith-Spiegel and colleagues’ USNV sample [19], Reeser et al. [20] and Chenneville et al. [24] and compared our actual scores and the difference between the ideal and actual scores with those in Reeser et al. [20] and Chenneville et al. [24]. We found a significant strong correlation between the mean ideal scores in our study and those in the USNV sample (p < 0.001, Spearman’s Rho = 0.817, df = 43). A plot of our ideal responses against those of Keith-Spiegel et al. [19] is shown in Fig 2 below. This result is consistent with findings in Hall et al. [23].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Plot of the ideal mean found in our study against that in the USNV Sample in Keith-Spiegel et al. [19].

https://doi.org/10.1371/journal.pone.0241783.g002

Moreover, three of the top five ranked ideal items (3, 4, 41) and three of the last five ranked ideal items (9, 37, 39) reported in the USNV sample were the same as those in our data. For example, timely reviews, IRB commitment to both participants’ rights and research, and not allowing personal biases to affect the evaluation of protocols featured as the most important ideal IRB features for both our respondents and the respondents in the USNV sample (S4 Table). Similarly, both the Singaporean cohort in our study and the USNV sample [19] viewed lay membership and diversity of membership as well as editorial suggestions from the IRB as some of the least important features of an IRB (S5 Table). Our ideal scores for all 45 IRB-RAT items were significantly higher than those in the USNV sample [19] (S6 Table). The range of the difference between the mean ideal scores in our study and the mean ideal scores in the USNV sample [19] was 1.63.

Forty-one actual IRB-RAT items in our study scored significantly lower than those in Reeser and colleagues’ study [20]. However, only 17 ideal items showed significant difference between our data and those in Reeser and colleagues’ study [16].

With regard to Chenneville and colleagues’ study [24], only six ideal items and ten actual items differed significantly. The correlation between the mean ideal scores in our study and the mean ideal scores in Chenneville and colleagues’ study [24] was weak (p < 0.001, Spearman’s Rho = 0.476, df = 43). Only one of the five most important ideal items (item 4) and two of the five least important ideal items (items 9 and 39) were the same in both our study and Chenneville and colleagues’ study [19, 24].

Respondents in Chenneville and colleagues’ study [24] reported a significantly smaller expectation gap (i.e. the difference between the ideal and actual IRB ratings) than respondents in our study in a number of areas. A smaller expectation gap indicates that an IRB more closely met the respondents’ expectation. In Chenneville and colleagues’ study a significantly smaller expectation gap was observed in areas that would promote science and research, for example: being open to innovative ways of performing research (item 14); being viewed as an ally rather than as a hurdle to clear (item 40); showing considerable evidence that the advancement of science is part of its mission (item 7); being willing to work with investigators to find mutually satisfying solutions whenever disagreements exist (item 8); ensuring that at least one member is knowledgeable about the content domain and discipline of submitted protocols (item 17). In these areas, respondents in our survey perceived IRBs as falling shorter of their expectations compared with respondents in Chenneville and colleagues’ study [24]. On the other hand, respondents in our study had a significantly smaller expectation gap, indicating that IRBs more closely met their expectations in adhering to certain rules and formalities compared with what was reported in Chenneville and colleagues’ study [24]. The S7 Table summarizes the different areas in which we observed expectation gaps in these two studies. Compared to that of Reeser and colleagues’ study [20], the mean difference between the ideal and actual IRB ratings in our study was significantly greater for 38 items. Further details can be found in S3 Table.

Table 6 groups the IRB-RAT items along two rating dimensions: their relative importance as measured by their ideal scores and their relative performance as measured by the paired difference between the ideal and actual scores. IRB performance in this part of the analysis was operationalized using the paired difference rather than the actual scores because the former better reflects which items require improvement. Our analysis used paired difference to reduce the effect of confounding variables. For example, respondents could not know how an actual IRB rates on an item for which they had no direct knowledge because they were not present during IRB deliberations. This may have affected how important they thought that factor was in an ideal IRB.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. The classification of IRB-RAT items according to performance and importance.

https://doi.org/10.1371/journal.pone.0241783.t006

The left column in Table 6 shows items of ‘low importance’, understood as items whose ideal score is at or below overall (weighted) average of the ideal scores and the right column shows the ‘high importance’ items. The top half of this quadrant shows items whose performance was at or above average compared to the overall (weighted) average of the paired difference between the ideal and actual scores, while the bottom half of the quadrant shows items whose performance was below the (weighted) average. This method of analysis is derived from Hall and colleagues’ 2015 study [23].

Category 4 in Table 6 consists of items that were regarded as above average importance but below average performance. These 16 items include the 5 lowest performing IRB-RAT items, as measured by the paired difference between ideal and actual scores (items 3, 40, 22, 32, and 5). This suggests that attention should be paid to the items in category 4 as items most in need of improvement. Further analysis of these 16 items in category 4 is taken up in a separate forthcoming publication.

There was no statistically significant effect of experience of submissions to foreign IRBs, years of experience in biomedical research, or type of researcher (PI/Co-I vs. research support staff). Furthermore, the effect of IRB membership was indeterminable because only five participants were IRB members. However, there was a statistically significant difference between respondents who had experience performing data-only research and respondents without experience in data-only research for 14 items (see S8 Table.) Most of these items (12 of 14) concerned ratings on the actual IRB. In all of the 12 items on the actual IRB, respondents with experience in data-only research rated the actual IRBs lower than respondents who did not have experience performing data-only research. On the other hand, respondents with experience in data-only research provided significantly higher ratings for the two items that concerned ideal IRBs: an IRB that views its role as being an investigator’s ally rather than as being a hurdle to clear (item 40); and an IRB that is empathetic with the difficulties that can present themselves during the design or conduct of the research (item 42).

S1 and S2 Figs show the distribution of ideal ratings on IRB-RAT items 40 and 42 respectively, grouped by respondents with and without experience performing data-only research. In S1 Fig we see that respondents with experience in data-only research selected only ratings 6 (27%) and 7 (73%). In contrast, respondents without experience in data-only research selected a wider range of ratings, and the proportion of those selecting ratings 6 (23%) and 7 (53%) was lower. In S2 Fig we show that of all respondents with experience in data-only research, 34% selected rating 6 and 59% selected rating 7. In contrast, of all respondents without experience in data-only research, 30% selected rating 6 and 40% selected rating 7.

Discussion

This is the first study conducted in Singapore that aimed to gain an understanding of a. how local researchers and research support staff view the impact of IRB review generally; b. how they perceive the IRBs they are in contact with; and c. which IRB features and functions they view as important for IRBs to display, as these impact on their work. Despite the small sample and the inability to ascertain the total number of such researchers and research support staff in Singapore, the findings are important as they provide the only insight we have into perceptions of the characteristics and functioning of Singaporean IRBs. Due to the nature of the findings, IRBs could still consider improvements in the areas identified, particularly because the findings are supported by international literature. This is an opportunity for Singaporean IRBs to pause and consider how they could more generally attend to perceptions about their role and function in the research ethics process and the value of cohesive working relationships.

There were at least four key insights that can be gleaned from our study. Firstly, there was an overall positive perception of the impact of the ethics review process on a number of dimensions. Secondly, gaps between ideal and actual scores in our results highlight to IRBs and institutions where they may want to focus reform efforts. Thirdly, where there are strong areas of overlap between perspectives among Singaporean researchers and those in the US, the solutions to concerns adopted in the US may be particularly worthy of attention for possible adaptation in Singapore. And finally, appropriate reforms to IRB processes to address researchers’ concerns may be substantially different for data-only research compared with other forms of research, reflecting the divergent priorities and needs of different types of researchers. We will now discuss each of these insights in more detail.

Limitations

Limitations of our study should also be acknowledged. The response rate could not be calculated, which means that we cannot calculate the extent to which our results are representative. A related point is that even though recruitment was aimed at researchers and research support staff from all biomedical RIs in Singapore, participants came from only 16 of these. Finally, an issue which impacted on recruitment numbers was the fact that we did not have direct access to researchers’ contact details to recruit directly but rather relied on third parties to disseminate the invitation to participate. This method always yields lower participation rates.

Conclusions

Respondents in this study generally consider ethics review as a process that adds value in key areas such as the quality of research, establishing and maintaining public trust in research, the protection of research participants, and the scientific validity of research. There remains, however, some work to be done to improve such perceptions, as a proportion do not consider the ethics review process as value-adding; in fact, a small minority view the process as detrimental to research.

Singaporean respondents reported highly valuing IRB functions and characteristics that positively impact on the efficiency of review, including timeliness and a balance between protecting research participants and facilitating the conduct of research, as well as IRB willingness to work with investigators to find mutually acceptable solutions where disagreements emerge. Conversely, matters relating to IRB composition were rated as being of relatively less importance. We established that these Singaporean researchers and research support staff hold similar views to each other as well as similar views to those surveyed in some international studies about the most important features and functions of an ideal IRB; they also hold similar perceptions about the IRBs they most deal with. There was a greater gap between our respondents’ conception of an ideal IRB and the reality of the IRBs they deal with. Local differences also emerged between those dealing with data-only research and those engaged with recruitment of participants, which is an area that warrants additional research given the increasing dependence on big data, particularly in Singapore.

This research is a first step in shedding light on researcher and research support staff perceptions. While there are current gaps, the ideal would be for researchers to view IRBs as a resource rather than an obstacle and for IRBs to view researchers as valuable contributors to scientific, medical, and social progress. Further, it would be mutually beneficial for IRBs to view researchers as a group whom they should assist to improve research protocols both to better protect research participants but also to enhance the scientific integrity of research. An added benefit of such collaborative efforts would also be a reduction in approval timelines, an issue of prime importance to this and other research communities internationally.

Supporting information

Acknowledgments

We are very grateful to the PICs and POCs in all RIs in Singapore as well as the Council of the Singapore College of Clinician Scientists who kindly disseminated the invitation to researchers and research support staff.