10 Mar 2021: Labude MK, Shen L, Zhu Y, Schaefer GO, Ong C, et al. (2021) Correction: Perspectives of Singaporean biomedical researchers and research support staff on actual and ideal IRB review functions and characteristics: A quantitative analysis. PLOS ONE 16(3): e0248613. https://doi.org/10.1371/journal.pone.0248613 View correction
Biomedical research is overseen by numerous Institutional Review Boards (IRBs) in Singapore but there has been no research that examines how the research review process is perceived by the local research community nor is there any systematic data on perceptions regarding the review process or other research ethics processes and IRB characteristics. The aim of this study was to ascertain general views regarding the overall perceived value of ethics review processes; to measure perceptions about local IRB functions and characteristics; to identify IRB functions and characteristics viewed as important; and to compare these views with those of other international studies.
An online survey was used with the main component being the IRB-Researcher Assessment Tool (IRB-RAT), a validated tool, to evaluate perceptions of ideal and actual IRB functions and characteristics held by Singaporean researchers and research support staff. Data were analysed descriptively first, with mean and SD of each item of IRB-RAT questionnaire reported, excluding the respondents whose answers were unknown or not applicable. The Wilcoxon Sign Rank test was used to compare the ideal and actual ratings of each IRB-RAT item, while the Mann-Whitney U test was used to compare the ratings of each IRB-RAT item between respondents with different characteristics. The Z-test was used to compare the mean ratings of our cohort with the mean ratings reported in the literature. The correlation between our mean ideal scores and those of two international studies also employing the IRB-RAT was examined.
Seventy-one respondents completed the survey. This cohort generally held positive views of the impact of the ethics review process on: the quality of research; establishing and maintaining public trust in research; the protection of research participants; and on the scientific validity of research. The most important ideal IRB characteristics were timeliness, upholding participants’ rights while also facilitating research, working with investigators to find solutions when there are disagreements, and not allowing biases to affect reviews. For almost all 45 IRB-RAT statements, the rating of the importance of the characteristic was higher than the rating of how much that characteristic was descriptive of IRBs the respondents were familiar with. There was a significant strong correlation between our study’s scores on the ideal IRB characteristics and those of the first and largest published study that employed the IRB-RAT, the US National Validation (USNV) sample in Keith-Spiegel et al. .
An understanding of the perceptions held by Singaporean researchers and research support staff on the value that the ethics review process adds, their perceptions of actual IRB functions and characteristics as well as what they view as central to high functioning IRBs is the first step to considering the aspects of the review process that might benefit from improvements. This study provides insight into how our cohort compares to others internationally and highlights strengths and areas for improvement of Singapore IRBs as perceived by a small sample of the local research community. Such insights provide a springboard for additional research and may assist in further enhancing good relations so that both are working towards the same end.
Citation: Labude MK, Shen L, Zhu Y, Schaefer GO, Ong C, Xafis V (2020) Perspectives of Singaporean biomedical researchers and research support staff on actual and ideal IRB review functions and characteristics: A quantitative analysis. PLoS ONE 15(12): e0241783. https://doi.org/10.1371/journal.pone.0241783
Editor: Daniel Jeremiah Hurst, Rowan University School of Osteopathic Medicine, UNITED STATES
Received: June 19, 2020; Accepted: October 20, 2020; Published: December 31, 2020
Copyright: © 2020 Labude et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data area available from Figshare with the DOI: https://doi.org/10.6084/m9.figshare.13093466.
Funding: This research was supported by the Singapore Ministry of Health’s National Medical Research Council under its NMRC Funding Initiative grant (NMRC/CBME/2016). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: IRB, Institutional Review Board (i.e. Human Research Ethics Committee); PIC, Principle-Person-in-Charge; POC, Person-of-Contact; RI, Research Institution; USNV sample, US National Validation sample
The role of Institutional Review Boards (IRBs), otherwise known as Human Research Ethics Committees (HRECs) or Research Ethics Committees (RECs), in biomedical research remains central to ensuring that research is designed and conducted in accordance with high ethical and scientific standards, that research participants are adequately protected from harms inherent in some research and that their dignity and rights are protected . IRBs were first established by law in the US in the 1970s, partly in response to revelations about harmful research [2, 3]. While the functions of IRBs around the world and across institutions may differ slightly, their general aim is to review proposed research projects involving humans to ensure that the research is ethically justifiable along a number of dimensions before it commences. IRBs will, for instance, assess the potential harms of a proposed research project and ways to minimise such potential harms, the anticipated benefits (to society and/or to participating individuals), the vulnerability of the subject population, the recruitment process, the consent process, researchers’ experienced and expertise, potential conflicts of interest and, in some institutions for some proposals, other aspects of a proposal, such as study design. IRBs also have oversight responsibilities in relation to approved research as well as an educational function in some institutions to which they belong. (See for example: [4–7]).
In recent decades, the research landscape has significantly changed as a result of scientific advances, substantial diversification of research, and increases in the overall volume of biomedical research . As a result, IRBs have made considerable efforts to refine and revise their processes to keep pace and, in many cases, to redefine their role in the research ethics process to extend beyond approval and oversight functions . Nevertheless, research continues to require appropriate ethical oversight provided by IRBs, which, in the context of Singapore, is duly acknowledged in national laws and guidelines [5, 6].
Despite the central role IRBs play in ensuring the ethical conduct of research, it is recognized internationally that IRBs may not be functioning at the high level expected of them ; this may be due to the intricate operational facets which characterize them and, in part, due the financial constraints to which some are subject [8, 11], which can impact on the provision of adequate training for IRB members and therefore the quality of review processes and IRB functioning. Moreover, the relationship between IRBs and the research community is often strained and impacted by perceptions and expectations that the research community have of the role and function of IRBs [12–14], which have been shown to diverge from perceptions held by IRBs . For example, researchers view IRB interventions as means of controlling them but IRBs view their involvement as helpful and accommodating; researchers may view IRBs as unfriendly and adversarial as they adopt a stance of interrogation when seeking justifications for aspects of research proposals; researchers may view IRBs as impeding research, as they sometimes forego certain studies for fear of having it rejected but IRBs appear to view this as an inevitable but unintended consequence of processes; researchers view IRBs as exerting considerable power but IRBs do not share this view because they see themselves as simply implementing research regulations; finally, researchers seem to lack an understanding of the research approval machinery which some IRBs view as the reason for delays and lengthy approval processes . Researchers have a range of expectations given that their work is dependent on ethics clearance; they expect, for example, timely reviews, requisite IRB expertise, and IRB ability to appropriately assess ethical issues, including risks and benefits, rather than simply focusing on regulatory compliance [10, 16]. Insight into how researchers view IRB practices can be helpful in overcoming impediments to a good working relationship between the research community and IRBs.
Internationally, a number of empirical studies have examined how biomedical researchers perceive the IRB review process. Systematic literature reviews [10, 17] indicate that these studies vary considerably in terms of sampling characteristics, data collection procedures, survey tools, and analyses, making their results difficult to compare. However, one survey instrument that has enjoyed repeated uptake is the IRB-Researcher Assessment Tool (IRB-RAT), which measures researchers’ perceptions of how 45 IRB characteristics and functions relate to their ideal IRB and their actual IRB . Ideal IRB is to be understood as an IRB that displays features that are most central to enabling researchers to achieve their best work while actual IRB is to be understood to relate to IRB features that researchers feel their IRB actually displays. An initial study of 886 US biomedical and social behavioral scientists  generated a baseline ‘US National Validation Sample’ (USNV Sample) to validate the IRB-RAT. Subsequent studies have used the IRB-RAT to capture the perceptions of medical researchers at specific institutions [20, 21] and to guide quality improvement of IRB functioning by identifying the areas in which perceived actual IRB processes particularly fall short of the identified ideal processes [22, 23].
However, to date there has been no systematic attempt to identify and measure how researchers perceive IRB functions and characteristics in Singapore. Indeed, so far only one published study has used the IRB-RAT to measure researchers’ perceptions outside the US . Notably, this is also the only published study originating from the broader Asian region that explicitly explores researchers’ perceptions of IRBs. While there are published studies on IRB processes and characteristics in Asian countries, these tend to describe only the structure and composition of IRBs [25–27] or only measure their operational efficiency . Our study addresses this gap and uses the IRB-RAT to measure how IRB users in Singapore rate their IRB’s review processes and characteristics and what IRB characteristics they would view as improving the quality of IRB functioning. In addition to the IRB-RAT, which collects information about perceptions of the particular IRB researchers are most familiar with, our survey also collected information about the general perception of the impact of IRB review in various areas, such as maintaining public trust in research.
Our study contributes to a general understanding of perceptions around IRB functions and characteristics in Singapore. It aims to generate preliminary evidence that may be useful in guiding institutions and policymakers to improve IRB functioning in Singapore, and to compare Singaporean findings with international findings. The study also provides a platform for further analyses and research to develop a comprehensive understanding of the Singaporean research ethics landscape with the aim of generating more specific suggestions for improvements to local IRB processes and functions in the future. Finally, through the use of the IRB-RAT, the study allows us to compare Singaporean findings with international findings.
Data was collected via an anonymous online survey incorporating the IRB-RAT. The survey was launched on the e-Survey platform of the National University of Singapore (NUS) and IP addresses were not collected.
Section 1 screened potential participants to ensure that they were either a PI/Co-I on a biomedical research project that has undergone IRB review in the past 12 months, or research support staff who had been substantially involved in the drafting of a biomedical research project that had undergone IRB review in the past 12 months. This ensured participation only by individuals who had recent first-hand experience with IRB submissions. Section 2 consisted of questions about the general impact of IRB review in four areas. Section 3 of the survey comprised the core component and was an adaptation of the IRB-RAT, a validated tool serving as a proxy for the quality of IRB functions and characteristics . The survey prompted respondents to indicate on a 7-point Likert-type scale how well 45 statements about IRB characteristics and functions (‘items’) described their ideal and their actual IRB . If participants had experience with more than one IRB, they were asked to respond about only one of the IRBs.
Three variations were made to the original IRB-RAT. First, minor adjustments were made to the wording of some statements to ensure they were applicable to the local context, e.g. ‘federal regulations’ became ‘relevant laws or national guidelines’. The second variation concerned the structure of the published tool. Instead of asking participants to first give views about their ideal IRB for all 45 questions and, then, in a second round asking for their views about the actual IRB rating across all 45 items, we converted this to a “single-pass” version which contained all 45 items and required consecutive responses, one about the actual and one about the ideal IRB. The rationale for this change was that the single-pass version would decrease the cognitive load for participants and the time required to complete the survey. The original IRB-RAT, developed by Keith-Spiegel and Koocher allowed for this variation but had not implemented it . The third variation concerned the introduction of the option “I don’t know/ I have no experience”, for responses relating to actual IRBs. This option was included because we judged that researchers and research support staff could not possibly be personally familiar with some internal IRB processes (e.g. “An IRB composed of members who arrive at meetings well-prepared”). We notified the developers of the IRB-RAT that we would be employing their tool and informed them of the modifications that we intended to make.
Section 4 of the survey collected non-identifying demographic data relating to institutional affiliation, IRB membership, type of research engaged in, years of involvement in biomedical research and involvement in international research. The research was approved by the National University of Singapore IRB (S-18-289E) and participants were not reimbursed for their involvement.
An e-mail invitation to complete the online questionnaire was sent to potential participants via two channels and was open for approximately 6 weeks. The first channel made use of the fact that biomedical research in Singapore may only be conducted under the purview of a Research Institution (RI) that has notified the Ministry of Health of its intention to perform such research. By Singapore law, RIs engaging in biomedical research are required to each appoint a ‘Principle-Person-in-Charge’ (PIC) and a nominated representative, ‘Point-of-Contact (POC)’, and the Ministry of Health maintains a public list of PIC/POCs’ contact details. PICs and POCs at all registered RIs were requested to forward the study invitation to relevant Principal Investigators (PIs), Co-Investigators (Co-Is) and research support staff. During the research period there were 38 registered RIs and the 35 institutions whose PICs/POCs agreed to be contacted to disseminate our research invitation can be found in S1 Table. Two reminder emails were sent. The second recruitment channel involved assistance from Singapore’s College of Clinician Scientists, whose Council forwarded the e-mail invitation to members of the College.
The survey was piloted with three individuals who were either biomedical researchers or research support staff. Participants in the pilot phase provided responses to the survey, time taken for their involvement, general feedback on the ease with which the survey was completed, their interest in completing the survey, and comments on the text provided at the beginning of the survey. Minor adjustments to the layout and the wording in the introduction were made in light of the feedback provided.
Statistical analyses were performed using SPSS v.25, R v.3.5.1 and v.3.6.2. Data were analysed descriptively first, with mean and SD of each item of the IRB-RAT questionnaire reported, excluding the respondents whose answers were unknown or not applicable. The Wilcoxon Sign Rank test was used to compare the ideal and actual ratings of each IRB-RAT item, while the Mann-Whitney U test was used to compare the ratings of each IRB-RAT item between respondents with different characteristics. The Z-test was used to compare the mean ratings of our cohort with the mean ratings reported in the literature. The correlation between our mean ideal scores and those in the USNV sample  and Chenneville and colleagues’  studies was also examined. The correlation between our mean ideal scores and the mean ideal scores in Reeser et al.  is not reported because the Reeser et al. sample was not a national sample. However, the correlation between our mean ideal scores and those in Chenneville and colleagues’ study  was reported, even though the Chenneville et al. study sample was not a national sample, because it is the only study outside the US to have used the IRB-RAT.
Among 149 individuals who answered at least 1 screening question, 117 individuals met the inclusion criteria. Ninety-eight individuals started and completed the questions on the general perception of the impact of IRB review and 82 individuals then progressed to start the IRB-RAT. Seventy-one completed it, with the remaining 11 providing responses to only some of the items (Table 2). All responses to the IRB-RAT survey questions were included in the final analysis regardless of whether the respondent completed the rest of the survey. Of the 35 RIs approached, 16 (42%) are represented in our sample with at least one response (S1 Table). The response rate could not be calculated as there was no information on how many researchers and research support staff had received the invitation to participate.
Demographic information was only available for 71 respondents who completed the survey since the section was located at the end of the survey. Table 3 lists these demographic details and shows that among the respondents, 53 (75%) were PI/Co-Is while the remaining 18 (25%) were research support staff. The vast majority (93%) were not IRB members. A large proportion of participants (58%) indicated involvement in research involving data only. Respondents could choose multiple affiliations and the most common affiliations clustered around two organisations.
Table 4 shows the results from Section 1 of the survey, which asked participants about their general perceptions of the impact of IRB review processes. The 98 participants who completed this section generally thought that the IRB review process has a very positive/positive influence on the following four areas examined: the overall quality of research (59%); establishing/maintaining public trust in research (74%); the protection of research participants (86%); and the scientific validity of research (50%). Few respondents stated that the IRB review process has a negative/very negative impact on these areas. However, a considerable percentage of respondents viewed IRB review processes as having no impact, particularly in relation to contributing to the scientific validity of research (45%) and on the overall quality of research (29%).
The ideal and actual ratings of each IRB-RAT item are summarized in Table 5. The items are ranked by the mean ratings for the ideal IRB in descending order, highlighting which items respondents in our study regarded as most and least important. A wide range of IRB features was viewed as crucial for an IRB, the most important of which included: timely reviews, the promotion of participants’ rights in conjunction with facilitation of research, good collaborative relations with researchers, IRB responsiveness, the absence of reviewer and committee biases, IRB accountability, procedurally and legally knowledgeable members and good record-keeping.
There were fewer responses in the actual dimension than in the ideal dimension for all items except item 33 (“An IRB that is open and pleasant in its interactions with investigators”), as participants could select the option “I don’t know/ I have no experience” for actual (but not for ideal) items. For some items, the number of participants who selected this option was significant. For instance, 41/71 respondents who answered the question did not know whether the members of the IRB that they were most familiar with arrived at meetings well prepared (item 45) and 37/73 respondents who answered the question did not know whether the IRB members would abstain from evaluating protocols whenever a real or apparent conflict of interest arose (item 21).
Fig 1 below plots the mean ideal and mean actual ratings. The importance of an ideal IRB possessing an item was rated higher than the actual IRB performance on all items except for item 39 (“An IRB that is composed of more than one lay person”). Item 39 also had the lowest mean ideal rating. This suggests that respondents may not see much value in having lay people on IRBs.
Ideal and actual scores for each of the 45 IRB-RAT items were compared, and the results show that all items in our study had significantly higher mean ideal scores than the mean actual scores, except items 9 and 39 for which the difference was not statistically significant (S2 Table contains further details).
We compared our ideal scores with those in Keith-Spiegel and colleagues’ USNV sample , Reeser et al.  and Chenneville et al.  and compared our actual scores and the difference between the ideal and actual scores with those in Reeser et al.  and Chenneville et al. . We found a significant strong correlation between the mean ideal scores in our study and those in the USNV sample (p < 0.001, Spearman’s Rho = 0.817, df = 43). A plot of our ideal responses against those of Keith-Spiegel et al.  is shown in Fig 2 below. This result is consistent with findings in Hall et al. .
Moreover, three of the top five ranked ideal items (3, 4, 41) and three of the last five ranked ideal items (9, 37, 39) reported in the USNV sample were the same as those in our data. For example, timely reviews, IRB commitment to both participants’ rights and research, and not allowing personal biases to affect the evaluation of protocols featured as the most important ideal IRB features for both our respondents and the respondents in the USNV sample (S4 Table). Similarly, both the Singaporean cohort in our study and the USNV sample  viewed lay membership and diversity of membership as well as editorial suggestions from the IRB as some of the least important features of an IRB (S5 Table). Our ideal scores for all 45 IRB-RAT items were significantly higher than those in the USNV sample  (S6 Table). The range of the difference between the mean ideal scores in our study and the mean ideal scores in the USNV sample  was 1.63.
Forty-one actual IRB-RAT items in our study scored significantly lower than those in Reeser and colleagues’ study . However, only 17 ideal items showed significant difference between our data and those in Reeser and colleagues’ study .
With regard to Chenneville and colleagues’ study , only six ideal items and ten actual items differed significantly. The correlation between the mean ideal scores in our study and the mean ideal scores in Chenneville and colleagues’ study  was weak (p < 0.001, Spearman’s Rho = 0.476, df = 43). Only one of the five most important ideal items (item 4) and two of the five least important ideal items (items 9 and 39) were the same in both our study and Chenneville and colleagues’ study [19, 24].
Respondents in Chenneville and colleagues’ study  reported a significantly smaller expectation gap (i.e. the difference between the ideal and actual IRB ratings) than respondents in our study in a number of areas. A smaller expectation gap indicates that an IRB more closely met the respondents’ expectation. In Chenneville and colleagues’ study a significantly smaller expectation gap was observed in areas that would promote science and research, for example: being open to innovative ways of performing research (item 14); being viewed as an ally rather than as a hurdle to clear (item 40); showing considerable evidence that the advancement of science is part of its mission (item 7); being willing to work with investigators to find mutually satisfying solutions whenever disagreements exist (item 8); ensuring that at least one member is knowledgeable about the content domain and discipline of submitted protocols (item 17). In these areas, respondents in our survey perceived IRBs as falling shorter of their expectations compared with respondents in Chenneville and colleagues’ study . On the other hand, respondents in our study had a significantly smaller expectation gap, indicating that IRBs more closely met their expectations in adhering to certain rules and formalities compared with what was reported in Chenneville and colleagues’ study . The S7 Table summarizes the different areas in which we observed expectation gaps in these two studies. Compared to that of Reeser and colleagues’ study , the mean difference between the ideal and actual IRB ratings in our study was significantly greater for 38 items. Further details can be found in S3 Table.
Table 6 groups the IRB-RAT items along two rating dimensions: their relative importance as measured by their ideal scores and their relative performance as measured by the paired difference between the ideal and actual scores. IRB performance in this part of the analysis was operationalized using the paired difference rather than the actual scores because the former better reflects which items require improvement. Our analysis used paired difference to reduce the effect of confounding variables. For example, respondents could not know how an actual IRB rates on an item for which they had no direct knowledge because they were not present during IRB deliberations. This may have affected how important they thought that factor was in an ideal IRB.
The left column in Table 6 shows items of ‘low importance’, understood as items whose ideal score is at or below overall (weighted) average of the ideal scores and the right column shows the ‘high importance’ items. The top half of this quadrant shows items whose performance was at or above average compared to the overall (weighted) average of the paired difference between the ideal and actual scores, while the bottom half of the quadrant shows items whose performance was below the (weighted) average. This method of analysis is derived from Hall and colleagues’ 2015 study .
Category 4 in Table 6 consists of items that were regarded as above average importance but below average performance. These 16 items include the 5 lowest performing IRB-RAT items, as measured by the paired difference between ideal and actual scores (items 3, 40, 22, 32, and 5). This suggests that attention should be paid to the items in category 4 as items most in need of improvement. Further analysis of these 16 items in category 4 is taken up in a separate forthcoming publication.
There was no statistically significant effect of experience of submissions to foreign IRBs, years of experience in biomedical research, or type of researcher (PI/Co-I vs. research support staff). Furthermore, the effect of IRB membership was indeterminable because only five participants were IRB members. However, there was a statistically significant difference between respondents who had experience performing data-only research and respondents without experience in data-only research for 14 items (see S8 Table.) Most of these items (12 of 14) concerned ratings on the actual IRB. In all of the 12 items on the actual IRB, respondents with experience in data-only research rated the actual IRBs lower than respondents who did not have experience performing data-only research. On the other hand, respondents with experience in data-only research provided significantly higher ratings for the two items that concerned ideal IRBs: an IRB that views its role as being an investigator’s ally rather than as being a hurdle to clear (item 40); and an IRB that is empathetic with the difficulties that can present themselves during the design or conduct of the research (item 42).
S1 and S2 Figs show the distribution of ideal ratings on IRB-RAT items 40 and 42 respectively, grouped by respondents with and without experience performing data-only research. In S1 Fig we see that respondents with experience in data-only research selected only ratings 6 (27%) and 7 (73%). In contrast, respondents without experience in data-only research selected a wider range of ratings, and the proportion of those selecting ratings 6 (23%) and 7 (53%) was lower. In S2 Fig we show that of all respondents with experience in data-only research, 34% selected rating 6 and 59% selected rating 7. In contrast, of all respondents without experience in data-only research, 30% selected rating 6 and 40% selected rating 7.
This is the first study conducted in Singapore that aimed to gain an understanding of a. how local researchers and research support staff view the impact of IRB review generally; b. how they perceive the IRBs they are in contact with; and c. which IRB features and functions they view as important for IRBs to display, as these impact on their work. Despite the small sample and the inability to ascertain the total number of such researchers and research support staff in Singapore, the findings are important as they provide the only insight we have into perceptions of the characteristics and functioning of Singaporean IRBs. Due to the nature of the findings, IRBs could still consider improvements in the areas identified, particularly because the findings are supported by international literature. This is an opportunity for Singaporean IRBs to pause and consider how they could more generally attend to perceptions about their role and function in the research ethics process and the value of cohesive working relationships.
There were at least four key insights that can be gleaned from our study. Firstly, there was an overall positive perception of the impact of the ethics review process on a number of dimensions. Secondly, gaps between ideal and actual scores in our results highlight to IRBs and institutions where they may want to focus reform efforts. Thirdly, where there are strong areas of overlap between perspectives among Singaporean researchers and those in the US, the solutions to concerns adopted in the US may be particularly worthy of attention for possible adaptation in Singapore. And finally, appropriate reforms to IRB processes to address researchers’ concerns may be substantially different for data-only research compared with other forms of research, reflecting the divergent priorities and needs of different types of researchers. We will now discuss each of these insights in more detail.
Perceptions of ethics review as value-adding
In terms of perceptions of the general impact of IRB reviews, the majority of respondents reported that IRB review has an overall positive impact. This generally positive attitude towards IRB review provides confidence that there is a general appreciation in this small cohort of researchers and research support staff of the intended role and purpose of IRBs.
However, a sizeable proportion of respondents felt that IRB review has no impact and an even smaller proportion felt that IRB review has a negative impact on the quality of research; establishing and maintaining public trust in research; the protection of research participants; and on the scientific validity of research. The perspectives of this minority who viewed IRB review negatively, while not representative of the entire Singaporean research community, are nevertheless cause for serious reflection. There is now considerable research ethics guidance available and awareness of legislation which regulates the conduct of research via IRBs and other relevant bodies but the justification for such controls may not be apparent to all. For example, unless researchers are familiar with the historical and scientific context within which research ethics developed, it is possible that the extensive focus on human participant protections, to give one example, may appear excessive and even obstructive. On the other hand, some of the lowest performing actual IRB-RAT items, which were viewed as important in this study, related to IRBs’ level of content knowledge and ability to conduct informed analyses. Therefore, negative views of the limited value that the ethics review process adds may also reflect these participants’ views of committee members’ competence; that is, their ability to correctly identify ethical issues that require consideration and suggest amendments that appropriately address these issues rather than focusing on unnecessary changes to protocols. Such issues echo researcher concerns in other parts of the world . If some in the research community fail to see the value in the review process, they are more likely to view it as a hurdle to clear rather than appreciating both the underlying reasons requiring review and the potential benefits arising from quality reviews. Such neutral and negative perceptions of the general value of ethics review have the potential to influence other researchers and adversely impact on researcher-IRB relationships [11, 13].
Highlighting potential areas for improvement
Concentrating more specifically on the findings of the IRB-RAT survey itself, IRBs in Singapore may wish to focus on the expectation gaps identified in this study. Identifying wider expectation gaps may help highlight areas where there is greater perceived need for reform while smaller expectation gaps may reveal areas where systems are perceived to be functioning in accordance with researchers’ needs and expectations. For the purpose of generating evidence to guide improvements to IRB review processes and functioning in Singapore, the results listed in Table 6 hold important insights, especially the areas where local IRBs are viewed as underperforming but which the respondents regarded as important features and functions of an IRB. The five areas identified by our participants as requiring most attention cover a range of procedural and substantive IRB characteristics and functions and align with concerns expressed in the literature.
Timely review of protocols was the issue identified as requiring most attention, an issue repeatedly identified in a range of research areas internationally [10, 19, 22, 30] including disaster research where timeliness is of central importance . Timely reviews are not simply a matter of inconvenience for researchers; they create significant delays in accessing findings that are of potential great public benefit . In addition, the detrimental impact IRB delays may have on interpersonal relationships has multiple dimensions, as members of IRBs are also members of the research community. It should, however, also be acknowledged that researchers also contribute to the protracted review period with delays in responding to IRB requests for clarification or amendment of the protocol .
IRBs as allies.
Of great importance to the conduct of research but lacking in these participants’ experience with IRBs was a collaborative and supportive relationship with IRBs, an issue also identified elsewhere [11, 15]. When researchers and IRBs do not view themselves as sharing a common goal in promoting quality research, this not only frustrates important interpersonal relationships but may ultimately impact negatively on scientific advances and clinical innovation . IRBs have the potential to play a significant role in guiding researchers, in a collaborative rather than authoritarian way, to achieve ethically, legally, and scientifically sound research thus contributing to greater societal benefits but also to more cohesive working relationships.
Need for IRB resources.
In order to be in a position to guide the research community and function efficiently, IRBs require sufficient resources, an issue acknowledged by our participants and others . Unless resources are available to ensure that adequate sustained training is available, it is unlikely that quality ethics reviews can be achieved. An environment where superficial reviews that focus on bureaucratic requirements yet overlook important issues of ethical import is likely to develop in the absence of appropriate training for IRB members.
Truthfulness and accountability.
It is unlikely that IRBs could ever avoid making errors but our participants felt that truthfulness and accountability for IRB errors (and processing delays) were important but lacking. Such perceived deficiencies also impact on interpersonal relationships and could be perceived as a lack of respect towards researchers. Making improvements in this area would attend to issues of justice but would perhaps require an attitudinal shift regarding the roles and functions of IRBs within institutions. For example, IRBs would need to view one of their primary roles as supporting researchers, in line with the issue raised above.
The fifth area that our participants viewed as most important yet most deficient related to IRB competency levels; more specifically, to IRBs being knowledgeable and discerning regarding the appropriate yet flexible application of informed consent requirements. Arguably, one of the greatest challenges for researchers and IRBs alike is striking the right balance between adhering to consent requirements that demonstrate appropriate consideration of and respect for research participants without being excessively burdensome and ultimately ineffective. IRBs have received harsh criticism regarding their perceived shifting objectives: “Rather than protecting research subjects from harm, they now seem especially focused on protecting universities and research centers.”p. 611 .
The discrepancy between these participants’ perceptions of IRB functions and characteristics of great value where IRBs are deficient may be important for IRBs to consider, particularly if they value a collaborative relationship of trust with the research community. As previously indicated, these findings have been subjected to additional analysis and will be presented in a separate forthcoming paper.
Unlike several other published studies that have employed the IRB-RAT, our recruitment pool was not limited to a single institution or health system nor to researchers only; we aimed to include any biomedical researcher or research support staff within Singapore. The only other published study that reports national data is the USNV sample , which involved biomedical and social behavioural scientists from a wide range of institutions. The high degree of correlation between the mean ideal scores in our study and those in the USNV study , as opposed to the lower degree of correlation between the mean ideal scores in our study and those in the Chenneville et al. study , are important to consider given that much guidance on IRB operations and functioning comes out of the US and may be relevant to IRB functioning in Singapore. If researcher perceptions are similar, it may be that solutions to address the gaps are also similar. We will therefore now discuss and highlight the implications of several areas of overlap between our study and the USNV sample.
In the assessment of functions and characteristics that are most important for the ideal IRB, the views of our respondents were broadly in line with the USNV sample [18, 19], indicating some apparent degree of alignment between Singaporean and US researcher perspectives on what constitutes an ideal IRB. Timeliness of IRB review processes was ranked as the most important IRB characteristic in both studies, reflecting researchers’ desire to avoid unnecessary delays in carrying out their research. Indeed, in Singapore, two of the top five ideal features of an IRB related to timeliness. We have already raised some implications arising from delays in the ethics review process above. In addition to the practicalities of receiving timely responses and ethics review outcomes, procedural timeliness carries an important ethical dimension; research is publicly funded and extensive delays in the approval process arising from deficiencies in IRB processes and functions contribute to wasting valuable and limited public funds . Another similarity between the two studies included the fact that both US and Singaporean cohorts rated balancing protecting participant rights whilst also facilitating research very highly suggesting that researchers are not eager to ignore research participants’ interests, but prefer a proportionate approach to ethics review.
Conversely, in both studies, items relating to having IRBs that are diverse and contain multiple lay members were regarded as less important. In fact, for our study, the only item where the actual average rating exceeded the ideal average rating was item 39, which states “an IRB that is composed of more than one lay person.” This may indicate that researchers in both Singapore and the US regard lay persons as being unqualified to evaluate their research proposals. It might also show lack of understanding about why the inclusion of lay members is often mandated, i.e. to voice concerns of potential research participants, who are themselves likely to be non-scientists, and to assist in ensuring that information that is conveyed to potential participants is accessible to laypersons . The absence of scientific training is, in this regard, the asset of a lay person. Another commonly low-ranked ideal item was the IRB’s role in offering editorial suggestions to protocols and participant information sheets. Given that IRBs are tasked with evaluating ethical and regulatory aspects of a study, correcting typographical and grammatical errors may be seen as trivial or irrelevant to their remit even though errors do, in fact, impact on the accuracy of the message conveyed, readability, and comprehension of a text. Respondents in both studies did, however, value IRBs’ input to overcome disagreements, (3rd highest ideal score in our study, 10th highest ideal scores in the USNV sample) which may indicate that they viewed such collaboration as essential to timely approvals.
There were also a few relevant differences between our study and other studies. Our ideal scores for all 45 IRB-RAT items were higher than those of Keith-Spiegel and colleagues’ study . While both cohorts valued similar items most highly, the Singaporean cohort seemingly attached greater importance to all listed IRB features. At the same time, the majority of our study’s ratings on the actual performance of an IRB were significantly lower than those of Reeser and colleagues’ study . On the surface, this would indicate that our cohort viewed the quality of local IRBs as lower. However, we do need to bear in mind the different composition of respondents in the various surveys, and the difference in contextual factors. For instance, 64% of all respondents in Keith-Spiegel and colleagues’ study  were social/behavioural scientists, while all of our respondents were involved in biomedical research. Furthermore, the respondents in Reeser and colleagues’ study  came from a single, not-for-profit medical centre in rural United States, while our respondents were spread out across multiple institutions in an Asian city-state. The fact that our study was conducted more than 10 years after other studies [19, 20] may also have contributed to the difference in results.
Type of research and respondents
An unexpected finding was that there was no statistically significant difference in responses from those leading the research (PIs/Co-Is) and those assisting with research protocols and submissions (research support staff). This is in contrast to other studies that have observed a difference . Our findings suggest that in Singapore both researchers and research support staff have similar perceptions when it comes to the actual performance of IRBs and also consider the same kinds of characteristics and functions identified as ideal for an IRB as being important. This is a positive finding, as such concordance is beneficial to the development of research proposals for submission to IRBs.
The relationship between the type of research conducted and the assessment of the actual performance of the IRB also merits consideration, particularly given the increasingly prominent role big data is playing in research and development around the world. We hypothesized that researchers who conduct research involving data only would have a more negative view of the actual functioning of their IRB since they may not as readily see the value in IRB review given the use of data rather than interacting directly with participants. There is evidence, for example, that some researchers do not consider local ethics approval a requirement when accessing previously collected data, such as clinical trial data deposited in the ClinicalStudyDataRequest.com repository . Our findings may support this hypothesis. Respondents with experience in data-only research rated actual IRB performance significantly lower than respondents without experience in data-only research in areas such as: IRB members being knowledgeable about IRB procedures and legal requirements; IRBs being allocated sufficient resources to carry out functions efficiently and thoroughly; and IRBs showing considerable evidence that the advancement of science is part of their mission. On the other hand, respondents with experience in performing data-only research placed significantly higher importance on an IRB which views itself as an ally rather than as a hurdle for researchers and an IRB which is empathetic with difficulties that occur during the design or conduct of research.
The results suggest that researchers who have experience performing data-only research may have different perceptions on some aspects of IRB functioning compared to researchers who do not have experience performing data-only research. These findings may relate to perceived challenges IRBs face with the methodological and technical complexities involved in some data-only research, the complexity in identifying the ethical and legal issues in such research, and in applying guidelines that do not fully consider new research methods. These perceptions may also point to the perceived need for IRB members to receive appropriate training in related areas such as data science. A scoping review conducted in 2018 identified such issues as being considerable challenges for IRBs in relation to the review of big data in health research . The importance that the respondents in our study placed on a close collaborative relationship with IRBs may point to what they view as the way forward to communicating the intricacies of their research and finding mutually acceptable solutions. Additional research in this area may be useful in further clarifying the identified difference between those conducting data-only research and those engaging with research participants directly.
Our research also led to the improvement of the IRB-RAT, which is relevant for those wishing to employ the tool in future studies. The original design did not give participants the option to select “I don’t know” in response to questions about how their IRB was faring along the actual dimension. Although the IRB-RAT developers raised the possibility that such an option could be added , it was not included in previously published studies. Our results confirm that for a number of the IRB-RAT items a substantial proportion of respondents could not provide a response on the 7-point Likert-type scale, as several of these items presuppose either a degree of familiarity with the inner workings of specific IRBs or intimate knowledge of the features of specific IRBs that respondents could not possibly have first-hand unless they were members of that IRB.
Limitations of our study should also be acknowledged. The response rate could not be calculated, which means that we cannot calculate the extent to which our results are representative. A related point is that even though recruitment was aimed at researchers and research support staff from all biomedical RIs in Singapore, participants came from only 16 of these. Finally, an issue which impacted on recruitment numbers was the fact that we did not have direct access to researchers’ contact details to recruit directly but rather relied on third parties to disseminate the invitation to participate. This method always yields lower participation rates.
Respondents in this study generally consider ethics review as a process that adds value in key areas such as the quality of research, establishing and maintaining public trust in research, the protection of research participants, and the scientific validity of research. There remains, however, some work to be done to improve such perceptions, as a proportion do not consider the ethics review process as value-adding; in fact, a small minority view the process as detrimental to research.
Singaporean respondents reported highly valuing IRB functions and characteristics that positively impact on the efficiency of review, including timeliness and a balance between protecting research participants and facilitating the conduct of research, as well as IRB willingness to work with investigators to find mutually acceptable solutions where disagreements emerge. Conversely, matters relating to IRB composition were rated as being of relatively less importance. We established that these Singaporean researchers and research support staff hold similar views to each other as well as similar views to those surveyed in some international studies about the most important features and functions of an ideal IRB; they also hold similar perceptions about the IRBs they most deal with. There was a greater gap between our respondents’ conception of an ideal IRB and the reality of the IRBs they deal with. Local differences also emerged between those dealing with data-only research and those engaged with recruitment of participants, which is an area that warrants additional research given the increasing dependence on big data, particularly in Singapore.
This research is a first step in shedding light on researcher and research support staff perceptions. While there are current gaps, the ideal would be for researchers to view IRBs as a resource rather than an obstacle and for IRBs to view researchers as valuable contributors to scientific, medical, and social progress. Further, it would be mutually beneficial for IRBs to view researchers as a group whom they should assist to improve research protocols both to better protect research participants but also to enhance the scientific integrity of research. An added benefit of such collaborative efforts would also be a reduction in approval timelines, an issue of prime importance to this and other research communities internationally.
S1 Table. Research Institutions that have notified the Ministry of Health, Singapore, of their operations (as at 4 September 2018).
S2 Table. Paired difference between our ideal and actual scores.
S3 Table. Comparison between our paired difference of ideal and actual scores with the difference of ideal and actual scores in other studies.
S4 Table. The 5 most important ideal IRB characteristics.
S5 Table. The 5 least important ideal IRB characteristics.
S6 Table. Ideal and actual scores from other studies and whether they differ significantly from our study cohort.
S8 Table. Items where respondents with experience in data-only research score significantly differently from those without experience in data-only research.
We are very grateful to the PICs and POCs in all RIs in Singapore as well as the Council of the Singapore College of Clinician Scientists who kindly disseminated the invitation to researchers and research support staff.
- 1. Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council. Tri-Council Policy Statement Ethical Conduct for Research Involving Humans. December 2018.
- 2. Moon MR, F. K-M-G. The History and Role of Institutional Review Boards. AMA Journal of Ethics. 2009;11(4):311–21. pmid:23195065
- 3. Emanuel EJ, Grady CC, Crouch RA, Lie RK, Miller FG, Wendler DD, editors. The Oxford Textbook of Clinical Research Ethics. New York, US: Oxford University Press, Inc.; 2008.
- 4. US Department of Health and Human Services. Regulations 45 CFR part 46. Available at: https://www.hhs.gov/ohrp/regulations-and-policy/guidance/faq/45-cfr-46/index.html.
- 5. Beskow LM, Botkin JR, Daly M, Juengst ET, Lehmann LS, Merz JF, et al. Ethical issues in identifying and recruiting participants for familial genetic research. American Journal of Medical Genetics. 2004;130 A(4):424–31. pmid:15455364
- 6. Bioethics Advisory Committee. Ethics Guidelines for Human Biomedical Research. Singapore: MOH, 2015.
- 7. National Health and Medical Research Council, Australian Research Council, Universities Australia. National Statement on Ethical Conduct in Human Research. Canberra: Commonwealth of Australia, 2007 (Updated 2018).
- 8. Grady C. Institutional Review Boards: Purpose and Challenges. Chest. 2015;148(5):1148–55. https://doi.org/10.1378/chest.15-0706. pmid:26042632
- 9. Lapid MI, Clarke BL, Wright RS. Institutional Review Boards: What Clinician Researchers Need to Know. Mayo Clinic Proceedings. 2019;94(3):515–25. pmid:30832791.
- 10. Abbott L, Grady C. A Systematic Review of the Empirical Literature Evaluating IRBs: What We Know and What We Still Need to Learn. Journal of Empirical Research on Human Research Ethics. 2011;6(1):3–19. pmid:21460582
- 11. Guillemin M, Gillam L, Rosenthal D, Bolitho A. Human Research Ethics Committees: Examining Their Roles and Practices. Journal of Empirical Research on Human Research Ethics. 2012;7(3):38–49. pmid:22850142
- 12. Giles J. Researchers break the rules in frustration at review boards. Nature. 2005;438(7065):136–7. pmid:16280994
- 13. Keith-Spiegel P, Koocher GP. The IRB Paradox: Could the Protectors Also Encourage Deceit? Ethics & Behavior. 2005;15(4):339–49. pmid:16578924
- 14. Angell EL, Bryman A, Ashcroft RE, Dixon-Woods M. An analysis of decision letters by research ethics committees: the ethics/scientific quality boundary examined. Quality & Safety in Health Care. 2008;17(2):131. pmid:18385408.
- 15. Klitzman R. The Ethics Police?: IRBs’ Views Concerning Their Power. PLoS ONE. 2011;6(12). pmid:22174893
- 16. Paul C. Health researchers' views of ethics committee functioning in New Zealand. New Zealand Medical Journal. 2000;113(1111):210–4. pmid:10909934.
- 17. Nicholls SG, Hayes TP, Brehaut JC, McDonald M, Weijer C, Saginur R, et al. A Scoping Review of Empirical Research Relating to Quality and Effectiveness of Research Ethics Review. PloS one. 2015;10(7):e0133639–e. pmid:26225553.
- 18. Keith-Spiegel P, Koocher G. Institutional Review Board Researcher Assessment Tool (IRB-RAT) Users Guide. 2005.
- 19. Keith-Spiegel P, Koocher GP, Tabachnick B. What Scientists Want from Their Research Ethics Committee. Journal of Empirical Research on Human Research Ethics. 2006;1(1):67–81. pmid:19385866
- 20. Reeser JC, Austin DM, Jaros LM, Mukesh BN, McCarty CA. Investigating Perceived Institutional Review Board Quality and Function Using the IRB Researcher Assessment Tool. Journal of Empirical Research on Human Research Ethics. 2008;3(1):25–34. pmid:19385780
- 21. Gerber A. An assessment of IRB member values using the IRB researcher assessment tool. Available from ProQuest Dissertations & Theses Global. (2240027002) 2019.
- 22. Hall DE, Feske U, Hanusa BH, Ling BS, Stone RA, Gao S, et al. Prioritizing initiatives for institutional review board (IRB) quality improvement. AJOB Empirical Bioethics. 2016;7(4):265–74. pmid:32775502
- 23. Hall DE, Hanusa BH, Ling BS, Stone RA, Switzer GE, Fine MJ, et al. Using the IRB Researcher Assessment Tool to Guide Quality Improvement. Journal of Empirical Research on Human Research Ethics. 2015;10(5):460–9. pmid:26527369
- 24. Chenneville T, Menezes L, Bylsma LM, Mann A, Kosambiya J, Baxi R. Assessing Institutional Ethics Committees in India Using the IRB-RAT. Journal of Empirical Research on Human Research Ethics. 2014;9(4):50–9. pmid:25747296
- 25. Akabayashi A, Slingsby BT, Nagao N, Kai I, Sato H. An eight-year follow-up national study of medical school and general hospital ethics committees in Japan. BMC Medical Ethics. 2007;8(1):8. pmid:17598923
- 26. Hyder AA, Wali SA, Khan AN, Teoh NB, Kass NE, Dawson L. Ethical review of health research: a perspective fromdeveloping country researchers J Med Ethics 2004;30:68–72. pmid:14872079
- 27. Kim O-J, Park B-J, Sohn D-R, Lee S-M, Shin S-G. Current status of the institutional review boards in Korea: constitution, operation, and policy for protection of human research participants. J Korean Med Sci. 2003;18(1):3–10. pmid:12589079.
- 28. Adams P, Kaewkungwal J, Limphattharacharoen C, Prakobtham S, Pengsaa K, Khusmith S. Is your ethics committee efficient? Using "IRB Metrics" as a self-assessment tool for continuous improvement at the Faculty of Tropical Medicine, Mahidol University, Thailand. PloS one. 2014;9(11):e113356–e. pmid:25406085.
- 29. Keith-Spiegel P, Koocher GP. Institutional Review Board Researcher Assessment Tool (IRB-RAT) Users Guide. 2005.
- 30. Petrova M, Barclay S. Research approvals iceberg: how a ‘low-key’ study in England needed 89 professionals to approve it and how we can do better. BMC Medical Ethics. 2019;20(1):7. pmid:30678668
- 31. Hunt M, Tansey CM, Anderson J, Boulanger RF, Eckenwiler L, Pringle J, et al. The Challenge of Timely, Responsive and Rigorous Ethics Review of Disaster Research: Views of Research Ethics Committee Members. PLoS ONE. 2016;11(6). pmid:27327165
- 32. Duszynski KM, Pratt NL, Lynch JW, Braunack-Mayer A, Taylor LK, Berry JG, et al. Process trumps potential public good: better vaccine safety through linked cross-jurisdictional immunisation data in Australia. Australian and New Zealand Journal of Public Health. 2019;43(5):496–503. pmid:31535432
- 33. Stryjewski TP, Kalish BT, Silverman B, Lehmann LS. The Impact of Institutional Review Boards (IRBs) on Clinical Innovation: A Survey of Investigators and IRB Members. Journal of Empirical Research on Human Research Ethics. 2015;10(5):481–7. pmid:26564945
- 34. Heimer CA, Petty J. Bureaucratic Ethics: IRBs and the Legal Regulation of Human Subjects Research. Annual Review of Law and Social Science. 2010;6(1):601–26.
- 35. Klitzman R. Institutional Review Board Community Members: Who Are They, What Do They Do, and Whom Do They Represent? Acad Med. 2012;87(7):975–81. pmid:22622206
- 36. So D, Knoppers BM. Ethics approval in applications for open-access clinical trial data: An analysis of researcher statements to clinicalstudydatarequest.com. PloS one. 2017;12(9):e0184491–e. pmid:28886178.
- 37. Ienca M, Ferretti A, Hurst S, Puhan M, Lovis C, Vayena E. Considerations for ethics review of big data health research: A scoping review. PloS one. 2018;13(10):e0204937–e. pmid:30308031.