The open science landscape: A whistle-stop tour

Today’s open science initiatives aim to address issues that range from very precise (such as providing nonambiguous identifiers to biological reagents in lab studies) to overarching (like embedding an appreciation for data sharing into a complex research ecosystem). Table 1 outlines 4 distinct topics that demonstrate the diversity of open science initiatives and convey the need for efforts across various fronts. We selected these topics based on our expertise; they are not intended to be exhaustive. Below, we unpack these examples and highlight where some have succeeded and others have fallen short (see also Box 2 for a personal perspective of open science milestones).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Examples of past developments and future directions in open science.

https://doi.org/10.1371/journal.pbio.3002362.t001

Methods transparency

The methods section of many publications lacks key information that would be necessary to repeat an experiment. In response to this lack of transparency, researchers across a range of health disciplines have come together to develop standardized reporting guidelines. The EQUATOR Network (Enhancing the QUAlity and Transparency Of health Research) now includes over 500 reporting guidelines for different types of health research. Some of the highly adopted checklists include CONSORT (Consolidated Standards of Reporting Trials) [19,20], ARRIVE (Animal Research: Reporting of In Vivo Experiments) [13,14], and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) [21]. To achieve their current impact, these guidelines have gone through updates informed by wide-reaching consensus processes. For example, despite the first iteration of the ARRIVE guidelines being endorsed by over a thousand journals [22], they had limited impact on improving transparent reporting, even when authors were explicitly requested to use the ARRIVE checklist [23]. Guidelines were then revised and updated to focus on feasibility and include educational resources and examples. Development of reporting standards is an ongoing process, and some are now being harmonized through initiatives such as the MDAR Checklist (Materials, Design, Analysis, and Reporting) [24,25] and the alignment of guidelines for reporting trial protocols (SPIRIT) and results (CONSORT) [26].

Beyond guidelines that outline what details to include in a publication, research transparency also depends on standardized structures for how to report this information. A few decades ago, catalogs of reagents for biological experiments contained a few hundred listings. A company name and antibody target were generally sufficient to unambiguously identify a reagent. Today, a catalog from a single company can list over 100,000 antibodies, with hundreds of antibodies targeting the same protein. Simply citing a company name and target leaves much ambiguity and, in a surprisingly large percentage of cases, leads scientists to waste money and time trying to optimize the wrong reagent [27–29].

To address the issue, researchers convened meetings and workshops with the editors-in-chief of 25 major neuroscience journals, officers from the US National Institutes of Health (NIH), and representatives of several nonprofit organizations to work on a plan to address the underreporting of reagents. They then proposed a 3-month pilot project in which journals requested that antibodies, organisms, and other tools listed in publications contain the reagent name, catalog or stock number, company name, and Research Resource Identifier (RRID), a reagent identifier that persists regardless of whether companies merge or stock centers move. This RRID initiative [30] is now in its ninth year and over a thousand journals request RRIDs. In 2020, nearly half of published references to antibodies included sufficient information to track the antibody down, a big shift from 15% in the 1990s [31]. By asking researchers to publish RRIDs, researchers were also inadvertently encouraged to double-check their reagents, reducing not only errors in antibodies but also the use of problematic cell lines, with no additional effort on the part of journals [29].

The success of the RRID initiative depended on a dedicated group of volunteers who worked for nearly a decade to overcome an initial unwillingness from actors who held power to make change. The initiative was initially contentious because it added to the workload of journal editors and simply updating author guidelines to request RRIDs proved ineffective. Achieving greater compliance required convincing journals to take an active approach, which depended on the persistence of the RRID Initiative leadership, alongside sufficient infrastructure for authors to easily find their reagents and a helpful helpdesk for when the infrastructure fails to perform as expected. When prominent journals such as Cell began to visibly request RRIDs, the conversation shifted. While we could celebrate the success of the RRID initiative as an example of the benefits of grassroots initiatives, an alternative argument can be made: that similar initiatives would be far more common if supported by standard funding mechanisms and greater stakeholder involvement.

Scholarly communication

Publishing technology has undergone remarkable transformations, and scientists can now instantaneously share nearly all aspects of their scholarship with a worldwide audience. However, the academic research community continues to treat journal articles as the principal way of sharing research and efforts for change generally remain tied to this journal-centric system. One unfortunate legacy of the print era—when publishing was expensive and limited in length and structure—is that publications often serve as an advertisement of research rather than a complete record of the research process and outcomes [32]. This state of affairs, combined with an incentive structure that rewards groundbreaking and positive findings, has led to a muddled scientific record that entails irreproducible studies and wasted resources.

The past few decades, however, have seen several open science initiatives making stepwise progress toward sharing the components of research. These efforts include preregistration of study design and outcome measures, as well as open sharing of materials, protocols, data, and code. Some disciplines have been much more successful than others in these endeavors.

ClinicalTrials.gov and the International Standard Randomised Controlled Trial Number (ISRCTN) were launched in the year 2000 and now contain over half a million registrations. These registries brought transparency to the research process by allowing anyone with access to the internet not only to see what clinical trials were being run but also to have information on the methods, including the study intervention, the inclusion criteria, the outcomes measures of interest, and, increasingly, the results. Their uptake was made possible by funded infrastructure from key organizations such as the US NIH, the European Commission, and the World Health Organization (WHO), and their adoption was fostered by 2 decades of policies from the International Committee of Medical Journal Editors [33], the Declaration of Helsinki [34], and the US Food and Drug Administration (FDA), among others. While the purpose of trial registration was initially to recruit participants and reduce duplication, the infrastructure was iteratively updated. First to make study plans transparent and later to serve as a database of clinical trial results with the aim to reduce selective reporting and wasted research efforts. These updates came with new policies from regulatory agencies, including a requirement for researchers to post their trial results. Notably, policies alone were not enough, and advocacy and external monitoring have been key to press researchers to adhere [35]. Today, most clinical trials are registered and report their results [36–38].

In disciplines beyond clinical trials, preregistration has yet to become standard practice. In psychology, recent estimates for the prevalence of preregistration are lacking, but it likely remains around or below 10% [39,40]. In the social sciences, preregistration prevalence is much lower [41], and in preclinical research, one of the main registries has only 161 registrations as of September 2023 [42–44]. This low prevalence may stem from research protocols in more exploratory fields being less strictly defined in advance as compared to clinical trials. Nevertheless, these disciplines could draw on the experience of clinical trial registration to encourage uptake where applicable and also explore alternative interventions that may prove more viable (e.g., blinded data analysis of electronic health records, as done on OpenSAFELY) [33].

Beyond increasing the uptake of preregistration, we can benefit from ensuring that preregistration is serving its intended purpose. One study found that 2 researchers could only agree on the number of hypotheses in 14% of the preregistrations they assessed [45]. A meta-analysis also found that about one-third of clinical trials published at least 1 primary outcome that was different than what was registered and that these deviations were rarely disclosed [46]. These data underscore the need to acknowledge that, although conversations about preregistration appear to have reached the mainstream, concerted and persistent efforts are needed to ensure their uptake and achieve their intended impacts.

Sharing of research data and code has also recently entered mainstream discussions. At the more advanced end of the spectrum, some manuscripts are now entirely reproducible with a button press [47]. However, a recent meta-analysis of over 2 million publications revealed that while 5% to 11% (95% confidence interval) of publications declared to have publicly available data, only 1% to 3% actually had publicly available data [48]. For code sharing, the estimate was <0.5%. The meta-analysis also found that only declarations of data sharing increased over time. Whether shared data are findable, accessible, interoperable, and reusable (FAIR) is yet another question, and some evidence, at least in the field of psychology, suggests that this is often not the case [49,50]. Meanwhile, several national-level funding agencies are quickly moving towards mandating the open sharing of data (US NIH, Canada’s Tri-Agency). While these policies are a step in the right direction, ensuring their success will take substantial effort beyond the policy alone [51,52].

Team science

To improve methods transparency and data sharing, we could benefit from employing individuals specialized in these tasks. The predominant model of academic research—where a senior researcher supervises several more junior researchers who each lead almost every aspect of their own project [53]—remains a vestige of an outdated apprenticeship model of scientific research. In practice, each aspect of a research project can benefit from distinct expertise, including domain-specific knowledge (e.g., designing a study), technical capabilities (e.g., statistical analysis), and procedural proficiencies (e.g., data curation and data deposit). Poor distribution of labor and lack of task specialization may be part of the reason data and code sharing remain rare [48,54], publications regularly overlook previous research conducted on the same topic [55], and the majority of studies in some disciplines use sample sizes too small to reasonably answer their research question [56].

Efforts to recognize diverse research contributions are helping usher in a new research model that fosters open science. The Contributor Roles Taxonomy (CRediT), launched in 2014, brings attention to the need for diverse contributions by outlining 14 standardized contributor roles, such as conceptualization, data curation, and writing (review and editing). Dozens of notable publishers have adopted CRediT, and some (e.g., PLOS) require a CRediT statement when submitting a manuscript [57]. While the concept of authorship continues to overshadow “contributorship,” the widespread adoption of CRediT is a first step in recognizing diverse research inputs: including efforts related to open science and reproducibility by including roles in data curation and validation. CRediT statements also provide a dataset that meta-researchers can use to study the research ecosystem and realign incentives [53,58]. The US National Academy of Sciences has taken a step towards this goal by establishing the TACS (Transparency in Author Contributions in Science) website, which will list journals committed to setting authorship standards, defining corresponding authors’ responsibilities, requiring ORCID identifiers, and adopting the CRediT taxonomy.

Promoting role specialization can also help foster the creation of large research teams and, in turn, valuable large-scale research resources. For example, the UK Biobank contains detailed genetic, biological, and questionnaire data from over 500,000 individuals and has been analyzed by over 30,000 researchers in about 100 countries [59–61]. Another initiative, the Brain Imaging Data Structure (BIDS) is a standard for file structure and metadata that allows results from expensive brain imaging studies to be more easily reproduced and meta-analyzed [62]. These efforts, however, require large specialized groups: The UK Biobank includes 15 distinct teams, including imaging, executive, data analyst, laboratory, study administration, and finance [63]; BIDS credits over 250 contributors across 26 roles [64].

Academic funding schemes, however, mainly support small to medium size teams. When larger teams are funded, they generally comprise several smaller teams and sometimes lack the organizational structure and efficiency that specialization can entail, including staff dedicated to human resources, information technology, and project management. Several exceptions exist across the biological sciences where large consortia are becoming more common (e.g., the European Commission Human Brain Project, the US NIH’s Knockout Mouse Program), and in high-energy physics, where CERN has served as a model for large-scale scientific collaboration. Consortia in other disciplines, however, continue to have difficulty securing funding and largely comprise volunteers with their main responsibilities elsewhere (e.g., the Psychological Science Accelerator) [65].

In the absence of mainstream funding opportunities for large and enduring research teams, the possibility of answering certain questions is left to those who can afford it, such as industry, government, and exceptional philanthropists. These actors may not prioritize the advancement of science and betterment of society in the same way one would hope that impartial academics do. For academia to remain competitive across the landscape of research questions, we envision a future where the systems for funding, hiring, and promotion prioritize the flourishing of large and long-lasting research teams.

Research culture

To embed open science and team science into our research system, we can benefit from considering our research culture—the behaviors, expectations, and norms of our research communities [66] (see Box 3 for a personal account). In the absence of a culture that prioritizes openness, tasks like accessing data that support a key finding can remain impossible and sharing your own data can be far from trivial.

Despite increasing awareness of the need for transparent and reproducible research practices, there remains a disconnect between ideals, formal policies, and the actual behavior of researchers. Reproducibility Networks are one example of a collective bottom-up effort to address these gaps. They comprise national consortiums of researchers distributed across universities who can work collaboratively with policy makers from research institutions, government, funders, and the broader research community to drive rigorous and transparent research. First launched in the United Kingdom, Reproducibility Networks now exist in over a dozen countries [67,68]. The UK Reproducibility Network‘s (UKRN) unified voice led to a major strategic investment of £4.5M from Research England to roll out a coordinated effort for training in open science across 18 institutions. UKRN creates a cohesive and consistent message of open science practices that is helping to establish an open science research culture in UK research institutions (e.g., through contributions to parliamentary inquiries [69]).

The Center for Open Science (COS), a nonprofit organization based in the United States, has also been pivotal in advancing open science practices and promoting transparency in research [70]. Many of the COS initiatives, such as the Open Science Framework (OSF), facilitate collaborative and transparent research workflows [71]. Through partnerships, education, and advocacy for open science principles, COS has significantly contributed to the global effort to transform research culture and improve research integrity [72].

To ensure the widespread adoption of transparent and reproducible research, we need a research culture that prioritizes training in open science practices. Training initiatives can be organized at various levels, from individual institutions to international collaborations. Nonprofit organizations (e.g., COS, ASAPbio [73,74]), academic institutions, and funding agencies (e.g., US NIH, Wellcome) provide open science training through initiatives such as curricula integration, professional development programs, funding support, and the provision of resources and workshops to promote open research practices and enhance research quality. These resources teach several topics, including open data, open access publishing, and how to create reproducible research workflows using open source tools like R and GitHub [53]. Emphasizing the importance of open science practices during early career development can be particularly valuable, as it fosters a culture of openness from the outset of a researcher’s career.

However, a general lack of adequate infrastructure and funding poses challenges for establishing and sustaining such initiatives. To overcome these challenges, institutions can support roles dedicated to improving research culture. For example, the University of Bristol in the UK employs an Associate Pro Vice-Chancellor for Research Culture. Making research culture and open science a key part of someone’s job description is likely to foster a better research ecosystem. Additional funding like the Enhancing Research Culture Fund from Research England provides grants to higher-education institutions to implement initiatives for positive research culture [75]. In Germany, the BIH QUEST Center for Responsible Research is a dedicated institutional initiative promoting transparent and reproducible research practices through education, services, tools, and meta-research, with a unique funding structure combining support from the Federal Ministry of Education and Research (BMBF) and the state of Berlin [76–78]. By providing resources and recognition, institutions can create an environment that actively encourages responsible and open research practices.

A modular and dynamic research record

In Open Science 2.0, researchers would regularly share individual components of their work (such as hypotheses, materials, protocols, data, code, manuscripts, and peer review) once that component is ready for external consumption, instead of at the end of the research cycle. A network of persistent digital object identifiers with citation pathways would link these various digital research outputs and allow other researchers to build upon them. Nondigital components of research (including reagents, researchers, and equipment) would also be given digital identifiers and linked to research outputs, in turn providing a record of their provenance (e.g., RRIDs, ORCIDs). Version control and forking (i.e., independent development of protocols or code based upon previous versions) would assure that relationships to previous items remain transparent while they are dynamically updated. This structure would spur a culture where comments on the work of others, including corrections and suggestions, become an integral part of the research record, instead of being scattered across myriad forums. Such feedback would arrive throughout the research lifecycle and encourage researchers to improve their output’s “record of versions” [79] rather than to defend a static “version of record.”

While this structure may seem fanciful to many researchers, it is already the basis for a thriving community centered around open source software and built upon platforms like GitHub. Within the research ecosystem, protocol repositories such as protocols.io, data repositories such as Figshare, Dryad, and The Dataverse Project, and platforms for sharing individual results such as microPublication apply similar concepts to particular steps of the research process. Nevertheless, these diverse research outputs are still not adequately linked to each other. Organizations such as Octopus and Research Equals provide a way to integrate these different outputs within a single platform, but their uptake remains limited [80].

As modular research outputs become more widely used, they would serve as the main pillar of the research record. Scientific articles would likely continue to exist, but as narrative descriptions of research, rather than the primary account of the research record. In this world, journals would need to make their value clear, as they would no longer be the primary venue for documenting the research record. They could emphasize their role as curators of science, selecting and summarizing the findings they deem most important [81], as evaluators of research through peer review (as exemplified by Peer Community In and eLife [82]), or reinvent their services in a multitude of ways [83].

Standardization and interoperability

For open science to foster collaboration, we can benefit from using agreed upon data structures, vocabularies, and metadata standards that allow both researchers and machines to easily integrate various open datasets and analyze them (i.e., they would be interoperable). Genomics and molecular biology provide strong examples of this standardization and associated interoperability. The creation of large databases such as GenBank and UniProt have led to gene and protein sequence data being deposited in a common format. This standardization fueled a revolution in bioinformatics, allowing large-scale analysis at the touch of a button [84–87]. In a particularly striking example, the Alpha Fold Protein Structure Database used AI to predict structures for over 200 million proteins, most of which were based on UniProt sequences [88]. This is one example of where AI can perform a task infinitely faster, and perhaps better, than any researcher could hope to. Automated tools would also conduct evidence synthesis in near real time and help scientists keep pace with an ever-growing scientific literature. To benefit from the capabilities of AI-powered research, however, requires that data are structured and transparently shared.

For many research fields, shared data still consist of custom-made spreadsheets, in which little attention is paid to standardization. This turns data synthesis into a painful process that can require hundreds of hours of human work in selecting articles on a given question, extracting data and analyzing it. In Open Science 2.0, we envision the level of structured transparency seen in the examples above as being common across disciplines.

Ongoing quality control

Several open science initiatives promote transparency with the hope that accountability will follow. However, if no person or software is checking the openly shared research outputs, or if openness comes only at the end of the research cycle, the effectiveness of quality control mechanisms remains limited. Historical examples (e.g., in manufacturing [89]) suggest that quality control is much more effective when conducted throughout each stage of a project.

Some initiatives already aim to move quality control earlier in the research process, such as Registered Reports [90]. But these initiatives are still based on what a researcher states they will do or did, rather than an audit of the actual research process. Embedding quality control systems within the routine of academic labs, as is commonplace in many industries, has proved a considerable challenge, and existing initiatives are still at an early stage [91,92]. Leveraging technology to make the research process more open, through the use of open lab notebooks, for example, can allow at least part of this quality control to be distributed and possibly automated. AI tools could warn researchers about missing information, protocol inconsistencies, references to retracted papers, or problematic RRIDs throughout the course of a project. They could also be leveraged during peer review to systematically check for issues that many expert reviewers regularly overlook [93–95] or be applied to entire corpuses of research [96,97]. In Open Science 2.0, we envision widespread transparency in standardized formats that support a mix of automated and manual quality control mechanisms that occur throughout each stage of the research cycle.

Reorganization of scientific labor

Achieving the level of openness, rigor, and interoperability present in Open Science 2.0 necessarily requires a reorganization of scientific labor to encourage task specialization across larger teams. These teams would include people with roles such as Data Manager, Systematic Reviewer, or Statistician, among others. Beyond teams within an institution, this kind of specialization can also be achieved through open science platforms that allow researchers to interact synergistically. Large-scale, distributed collaborations such as the Psychological Science Accelerator and ManyBabies are open to researchers across the globe who can contribute either with data collection or with other kinds of expertise but currently struggle to acquire sustained funding through standard government grants [65]. Regardless of whether teams are created within or across institutions, those involved in research would be rewarded for their specialization and not be expected to demonstrate proficiencies beyond their specialization. Assessments of research impact would also emphasize large-scale contributions, which would encourage institutions to hire individuals that will bring relevant expertise to existing teams, rather than focusing more narrowly on the potential of single principal investigators.