Preregistration—on the rise?

The preregistration of studies has been proposed to detect and counter these questionable research practices (e.g., see [15], and see [10] for an overview of other open science techniques). A preregistration is a research plan that is time-stamped, created before the data has been collected or examined, and most often submitted to a public registry, thus making planned study details available to others (possibly after an embargo period) [15, 16]. If the research plan changes afterwards, either a new version needs to be added or the deviations will be apparent when comparing the preregistration to the final manuscript. Thus, preregistration aims for a transparent presentation of what was planned at a certain time point and what changes may have been made to a study until its publication. Evidence from other scientific disciplines indicates that preregistration indeed works, i.e., it increases the transparency of the research process, and reduces questionable research practices and the rate of false positive findings (e.g., see [17, 18] for examples from clinical trials and epidemiology). Correspondingly, a recent study showed that preregistration, among other open science techniques, can drastically increase the replication rate: In their study, Protzko and colleagues [19] tested the prospective replicability of 16 novel empirical findings, using preregistration among other proposed current optimal practices. Here, 86% of effects could be replicated (p < .05) and effect sizes were 97% that of the original studies.

However, while preregistration is already well-established in other scientific disciplines and is mandated, for example, in medicine [20], it has been frequently demanded as a means to counter questionable research practices but is still not widely practiced in psychology.

On the one hand, in response to the replication crisis, many psychologists have committed themselves to the advancement and promotion of open science techniques such as preregistration (e.g., see [10, 13, 15, 21–23]). For example, Nosek et al. [21] describe preregistration as “hard, and worthwhile”, while Wagenmakers and Dutilh [23] posit “seven selfish reasons for preregistration”. Indeed, the number of preregistrations in psychology is increasing. For instance, the number of preregistrations on the Open Science Framework (OSF), a platform for sharing research materials, has been approximately doubling every year between 2012 and 2017 [22], and in a survey which was conducted in 2018, 44% of the sampled psychological researchers indicated having preregistered a hypothesis or analysis until 2017 [24].

Yet, looking at the fraction of published studies that were actually preregistered paints a different picture. In their recent study, Hardwicke et al. [25] found that only 3% of 188 examined articles from 2014 to 2017 which were randomly sampled from the literature included a preregistration statement, which contradicts the more positive outlook by [22, 24]. Stürmer et al. [26] also found that when early career researchers were asked about questionable research practices and open science, they deemed many open science practices necessary, yet toward preregistration they expressed more reluctance: Only about half of the participants found that preregistration was fairly necessary or very necessary, and even less indicated that they planned to consider preregistering their studies in the near future. Additionally, Logg and Dorison [27] showed that preregistration lags behind other open science practices such as open data and open material (as shown for articles in the journal “Organizational Behavior and Human Decision Processes”).

A number of reservations are mentioned frequently when discussing preregistration in psychology, including the fear that it leaves no flexibility during study administration and eliminates the possibility to conduct exploratory analyses (as presented by e.g., [15, 16]). Some people are concerned that this might stifle discovery [28]. Others worry that someone might take their preregistered and thus publicly available study idea and publish it before them (so-called scooping, see [29]). Additionally, the time costs and effort are often seen as obstacles regarding preregistration (e.g., see [16]). Besides these worries, some authors also express an overall critique regarding the concept of preregistration. Szollosi et al. [30–32] argue that preregistration is redundant when good theories are tested and does not itself improve theories. Others argue that preregistration cannot prevent some questionable research practices [33] or might not fit well with all types of research (see [16]). Moreover, some studies found problems with the current implementation of preregistration such as poor disclosure of deviations from preregistered plans in finished manuscripts [34–37]. Although most of the listed arguments against preregistration are counter-argued by supporters of preregistration (e.g., that exploratory analyses are still possible [15, 16, 29]) and findings from other scientific disciplines underline its benefits (e.g., see [17–19]), these reservations persist, and some researchers remain skeptical.

Aim of this survey

Previous surveys mainly inquired about preregistration in the more general context of open science [24, 26, 38, 39]. Meanwhile, comprehensive studies focusing on preregistration are lacking. We aimed to close this gap by exploring thoughts, motivations, and perceived obstacles of psychological researchers toward preregistration and how these are influenced by the time someone has worked in research or actual experience with preregistration, that is, whether someone preregistered a study in the past. For instance, we wanted to explore the data to find out whether the low rate of preregistrations is caused by fear of the unknown or based on negative experiences (and which ones), as well as whether the increase of preregistrations is driven by a few active supporters while others reject or are indifferent toward it. Thus, we aimed to shed light on the outlined discrepancy of public support for preregistration on the one hand, and a low fraction of preregistrations on the other, while also identifying possible roadblocks for preregistration in psychology. Mixed-methods were used, including both qualitative and quantitative approaches.

Additionally, we investigated two specific research questions: First, we examined which factors facilitate or prevent preregistration (research question 1). The theory of planned behavior [40, 41] is a prolific, influential (e.g., see [42]) and well-researched (e.g., see [43–48]) psychological theory that aims to predict social behavior and has been applied across various contexts (e.g., health). According to this theory, the intention to perform a behavior can be seen as a direct antecedent of the actual behavior. In this framework, the intention to preregister predicts preregistration, and how the intention is formed may be informative for effectively promoting this behavior. To our knowledge, this has not yet been studied in the context of preregistration or open science. As described by Ajzen and colleagues [40, 41], three aspects influence intentions which are defined as follows: 1) Attitudes toward the behavior which result from the ratio of perceived advantages to disadvantages of performing the behavior, 2) the subjective norm which represents the perceived social pressure to perform or not perform the behavior, and 3) the perceived behavioral control which focuses on the question if the subject has the resources and skills to perform the behavior or not (also see [43–48] for meta-analytical support of this model, and [49] for an overview). We measured attitudes toward preregistration as well as subjective norms and perceived behavioral control through items in an online questionnaire and investigated how they influence researchers’ intention to preregister their studies in the future. Based on the model’s postulations, we expected that more favorable attitudes and subjective norms as well as higher perceived behavioral control positively influence the intention to use preregistration. As the relative importance of attitudes, subjective norms and perceived behavioral control differs in dependence of considered behaviors, situations, and populations [40, 41], we tested which of these is the strongest predictor for the intention to use preregistration. We also included the perceived importance of preregistration as moderator for the strength of the influence of attitudes on intention, and the preregistration experience as a control variable. Such an extension of the model made it possible to compensate for potential non-attitudes (e.g., see [50]) and a potential sampling bias toward researchers that have already preregistered, and is explicitly allowed by the theory of planned behavior [40, 49].

Regarding the intention formation, we formulated the following hypotheses:

1. 1. The theory of planned behavior [40, 41] can be applied to the context of preregistration to significantly predict researchers’ intention to preregister their studies in the near future, using a moderated multiple regression model. We predicted that:
   1. 1.1 More beneficial attitudes are a positive predictor for the intention to preregister.
   2. 1.2 The perceived importance of preregistration moderates the effect of attitudes on intention positively.
   3. 1.3 The perceived importance of preregistration is a positive predictor for the intention to preregister.
   4. 1.4 Beneficial subjective norms are a positive predictor for the intention to preregister.
   5. 1.5 Higher perceived behavioral control is also expected to be a positive predictor.
   6. 1.6 These predictors combined can significantly predict researchers’ intention to preregister.

Second, we examined whether research experience predicts attitudes and the perceived intensity of motivations and obstacles (research question 2). Research experience was operationalized as the number of years someone indicates they have worked in psychological research. Early career researchers are oftentimes seen as the driving force of the open science movement (e.g., see [51]). We investigated if the research experience indeed has an influence on researchers’ responses about preregistration (a similar effect was reported by Abele-Brehm et al. in a comparison of academic groups regarding hopes and fears toward open science [38] and by Logg and Dorison when inspecting generational differences [27]).

Regarding this second research question, the following hypotheses were tested:

1. 2. We predicted that the research experience, that is, the amount of time someone has already worked in psychological research, has an influence on attitudes, motivation, and perceived obstacles regarding preregistration. Specifically, we conducted three multiple regressions (including preregistration experience as a control variable), and we posited the following non-directional hypotheses:
   1. 2.1 Research experience is a predictor for attitudes regarding preregistration.
   2. 2.2 Research experience is a predictor for the strength of motivation to preregister.
   3. 2.3 Research experience is a predictor for how strongly obstacles to preregister are perceived.

The present survey aimed to sample the general population of psychological researchers by recruiting participants whose articles appear on Web of Science, PubMed, PSYNDEX, and PsycInfo, as well as the subgroup of researchers who have preregistered before and who were identified through their preregistrations on OSF Registries.

Material

The online survey was created with the software SoSci Survey (version 3.2.29) [52] and was supplied via www.soscisurvey.de. It was presented in English. Items of the survey could be categorized in nine categories: 1) Sociodemographic questions, 2) items concerning the general usage of preregistration, 3) an attitude scale consisting of 24 items indicating the overall attitudes of participants regarding the concept of preregistration, 4) a subjective norm scale of eight items representing perceived social norms and pressure, 5) a perceived behavioral control scale of five items inquiring about researchers’ perceived control over the potential preregistration of their studies, 6) an intention scale of three items inquiring about researchers’ intention to use preregistration in the future, 7) items about motivations (a motivation scale including ten items measuring how strongly participants agree with potential motivations to preregister, plus additional more open items), 8) items about perceived obstacles (an obstacle scale featuring ten items measuring how strongly participants agree with potential obstacles to preregister, plus additional more open items), and 9) various open questions inquiring about suggestions for improving preregistration. Some items were adapted from other surveys that focused on related topics [26, 38] and were complemented with additional items generated in reference to other theoretical and empirical works on preregistration, open science [33, 53–55], and the theory of planned behavior [40, 41, 49, 56]. These sources were selected based on their theoretical relevance to the topic at hand (i.e., open science practices and theory of planned behavior). The identified items were either adapted (i.e., wording was changed slightly to make them easier to understand or make them fit the present topic, e.g., “I have more trust in research findings when the study has been preregistered.” instead of “… when the respective data are published.” [38]) or new items were generated using the concepts presented in the literature based on face validity. The items of the present survey as well as the original items including references are available in the supporting information (see S1 Table). For most of the original items, no validity measures are given in the literature. Only for the theory of planned behavior, predictive validity is considered in more detail [49]. However, since none of the survey items were used in their original form but were adapted or newly created based on the given concepts, no validity measures can be provided. Instead, only face validity is assumed. This validity evaluation is based on Flake and Fried [57].

Different item formats were included in the survey. All scale items (attitudes, subjective norm, perceived behavioral control, intention, motivations, and obstacles) were answered with a seven-point labeled answer scale as recommended by Ajzen [40] (1 = “Strongly disagree”, 2 = “Disagree”, 3 = “Slightly disagree”, 4 = “Neither agree nor disagree”, 5 = “Slightly agree”, 6 = “Agree”, 7 = “Strongly agree”). Scales were recoded from “1 to 7” to “-3 to +3” for data analyses yielding a middle category which has absolute meaning (i.e., 0 = neutral opinion, neither agreement nor disagreement). For the statistical analyses, the mean scores of the scales were used to measure how participants perceived 1) preregistration (attitude scale), 2) subjective norms regarding preregistration (subjective norm scale), 3) their own control about using preregistration or not (behavioral control scale), 4) their intention to use preregistration in the future (intention scale), and 5) their motivations (motivation scale) and 6) obstacles to preregister (obstacle scale). For each participant, the mean for each scale was calculated and used as the score.

Other items included a single or multiple choice response format, or the option for open text input. Whenever applicable, response options were displayed in randomized order to eliminate potential sequence effects.

Procedure

Participants received the link to the survey via personal email or social media call (see section sampling procedure). After the welcoming page, participation information was displayed, and informed consent needed to be provided to proceed. Furthermore, a captcha (arithmetic task) needed to be completed as a safeguard against bot responses. Then, participants completed the main body of the survey which successively focused on the different item categories (sociodemographic questions, general usage of preregistration, attitudes, subjective norms, perceived behavioral control, intention, motivations, perceived obstacles, and suggestions for improvement). Items which were relevant for the study procedure (e.g., informed consent, usability), as well as those important for the exclusions, quota sampling, and hypotheses tests were mandatory. Meanwhile, the items used for the descriptive reports were optional. Before any items related to preregistration were shown, a definition of preregistration was presented (see S1 and S2 Videos and S1 File) and correct understanding was checked, to ensure that all participants answered the items with the same concept in mind. At the end of the survey, participants had the option to participate in a lottery for 40 gift cards worth 50 € each and to sign up for receiving a preprint of the survey results by entering their email address, which was saved separately from the other data. Lastly, a debriefing followed that stated the aim of the survey and the research questions that were investigated. It took participants about 18 minutes (SD = 8 min, range = 44 min) on average to complete the survey (times were adjusted for interruptions by replacing the completion time of pages that took participants more than two hours or 3 x SD of a normal distribution, with the page median of the other participants). Screen recordings of the survey’s procedure are available in the supporting information (see S1 and S1 Videos). Additionally, a PDF of the questionnaire can be found in S1 File. After data collection, the winners were drawn, and the vouchers were awarded. The survey was approved by the ethics committee of Trier University, Germany (approval number: 07/2020). This form of consent was obtained in writing.

Pilot study

We conducted a pilot study to estimate the response rate and to test the recruitment method as well as the survey items. The pilot study featured mostly the same items as the main study but additional items, prompting participants for feedback about comprehensibility via open text input fields, were presented (“Did you have any problems answering the items of this page? Was anything unclear?”). Some items were adjusted based on the results.

We invited N = 200 participants to partake in the pilot study (100 were invited from OSF Registries, 50 from Web of Science, and 50 from PubMed) and sent a reminder one week after the initial invitation. Participations typically occurred shortly after each email contact. The pilot study was accessible for one month overall. In this time, 29 participants (17 PhD students, three postdocs, seven professors, and two members of other academic groups; the latter were screened out) started the survey of which 20 completed it (14 PhD students and six professors), yielding an overall response rate of 10%. Out of these 20, 18 (90%) participants had preregistration experience. No specific patterns in dropout behavior were found (e.g., dropping out at a certain position). No floor or ceiling effects were found. Reliability analyses for the measured scales were conducted, which showed an excellent reliability for the attitude (α = .93) scale, a good reliability for the motivation scale (α = .8), an adequate reliability for the obstacle (α = .79), and a moderate reliability for the subjective norm (α = .65) and perceived behavioral control scale (α = .66). One item of each scale correlated negatively with the remaining scale items (A11, M2, O8, SN2, and PBC3, see S1 Table). In the case of the PBC scale, Cronbach’s alpha was over .10 higher if the respective item was excluded. Nevertheless, the items remained in the survey and were checked again in the main study, since the pilot results were based on a very small sample and could only be interpreted with caution. And indeed, the reliability for the main study sample was adequate to excellent, the items that were unreliable in the pilot study exhibited normal patterns and were included in the main analyses based on the specified criteria (see section data analysis and preprocessing). Reliability analyses for the pilot study were slightly adjusted for the Registered Report compared to its Protocol because an error was discovered in the analysis script for the pilot recoding data.

After conducting the pilot study, the sampling strategy was revised by extending the recruitment of participants to four instead of two general databases for psychological articles and by inviting less participants via OSF Registries (10% instead of 50%). Moreover, some comments indicated ambiguous interpretations which led us to slightly modify the wording of respective items or to change categories (i.e., academic position was replaced by academic degree in the sociodemographic part of the survey). Lastly, a few items were added, e.g., two items assessing intention, following a manual about constructing questionnaires based on the theory of planned behavior [56].

Sampling procedure

Power analyses.

Data from psychological researchers at different career stages were collected. The optimal sample size of N = 296 was determined by using G*Power [58, 59] in combination with a thorough review of the existing literature as described in the following paragraphs. Each of the power analyses described below was specified to achieve a statistical power of 95% at a given significance threshold of 5% (α = β = .05). All power analyses are displayed in Fig 1 and are also included in the supporting information (see S2 Text).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the power analyses.

The detailed statistical models and corresponding power analyses are displayed for A) the moderated multiple regression that were computed for testing hypothesis 1 and B) the multiple regressions that were used to test hypothesis 2.

https://doi.org/10.1371/journal.pone.0281086.g001

To test which factors influence the intention to preregister (scale of three items, see hypothesis 1), a moderated multiple regression model was computed based on the rationale of the theory of planned behavior, which included six predictors: Attitudes (scale of 24 items), perceived importance of preregistration (one item in our questionnaire), attitudes x importance, subjective norm (scale of eight items), perceived behavioral control (scale of five items) [40, 41, 49], and preregistration experience (one item). The theory of planned behavior has been examined using meta-analytical approaches in various contexts (e.g., health behavior). The percentage of variance of intention that was explained by attitudes, subjective norm and perceived behavioral control combined, ranged between 30.4% < R² < 44.3% [43, 45–48]. We chose the lowest reported effect size (R² = 30.4%) as minimal effect size of interest. The power analysis for the overall regression model yielded an optimal sample size of N = 55 to be able to detect the determined effect size given the effect exists (see hypothesis 1.6). Additional power analyses were conducted to compute the optimal sample size to test the individual predictors. As comparable effect sizes, R² based on the averaged correlations of individual variables were searched for in the aforementioned meta-analyses, and the smallest ones were chosen for each power analysis. This resulted in an optimal sample size of N = 42 for testing attitudes (see hypothesis 1.1), N = 85 for testing subjective norms (see hypothesis 1.4), and N = 236 for testing perceived behavioral control (see hypothesis 1.5) as predictors for intention (with α = β = .05). Since perceived importance and its interaction with attitudes, as well as preregistration experience were not originally included in the model but added by us, no comparable effect sizes were available and thus, no power analyses were conducted for these variables.

To investigate whether research experience has an effect on attitudes (see hypothesis 2.1), and the perceived intensity of motivations (see hypothesis 2.2) and obstacles (see hypothesis 2.3) regarding preregistration, three regression models were computed, again including preregistration experience as a control variable. As comparable effect size, the smallest reported effect size for group differences by Abele-Brehm et al. [38] was used, and the corresponding F value was used to calculate R² for the power analyses. For these regression models, the alpha level was corrected using the Bonferroni-Holm method to account for multiple testing. Thus, the power analyses were calculated for the smallest alpha of 1.67% (5% divided by 3). These power analyses suggested an optimal sample size of N = 296 to test the influence of research experience on attitudes (see hypothesis 2.1), perceived intensity of motivations (see hypothesis 2.2), and perceived intensity of obstacles (see hypothesis 2.3) regarding preregistration.

As the optimal sample size for the regression models to test hypothesis 2 was the highest computed necessary N, it constituted the targeted sample size of N = 296.

Timeline data collection.

The collection of email addresses started on January 4, 2021. The first wave of invitations was sent on November 15, 2021. Overall, N = 1615 persons clicked on the survey link and N = 513 started the survey. The planned sample size was reached just before the end of the set time frame, (however, some data were excluded post-data collection, see section exclusion, missing data, and sample size). As preregistered, the survey was still accessible to participants that were already invited until the end of the defined timeframe, but recruitment was discontinued. Data collection was stopped based on the set criteria on December 21, 2021. The timeline of the data collection is displayed in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Survey timeline and corresponding actions.

The arrow illustrates the timeline while the boxes contain details about the sampling procedure.

https://doi.org/10.1371/journal.pone.0281086.g002

Data analysis and preprocessing

This Registered Report was written in Microsoft Word and R Markdown (version 2.7) [61, 62]. R (version 4.1.1) [63] and the R-packages afex (version 0.27–2) [64], Amelia (version 1.7.6) [65], car (version 3.0–8) [66, 67], corrplot (version 0.84) [68], lm.beta (version 1.5–1) [69], olsrr (version 0.5.3) [70], onewaytests (version 2.6) [71], psych (version 1.9.12.31) [72], stats (version 4.4.1) [63], tidyverse (version 1.3.0) [73], and writexl (version 1.3) [74] were used for the analyses. Scripts for preprocessing, assumption testing and all analyses are available in the supporting information (see S1–S4 Scripts). Additionally, all data (including meta-data about variables and values) are publicly accessible in the digital research repository PsychArchives (https://doi.org/10.23668/psycharchives.8400).

Data were processed in the following way: Responses from all scale items were recoded as described in the section material, and the polarity of negatively poled items was reversed. Additionally, multiple choice questions were recoded (originally: 1 = “not selected” and 2 = “selected”; new: 0 = “not selected” and 1 = “selected”), and single choice items were transformed into factors. Comments were inspected. Furthermore, data was screened for any inappropriate responses (i.e., advertising, offenses) which led to an overall exclusion (see section exclusion, missing data, and sample size).

As a quality check, the attitude, subjective norm, perceived behavioral control, intention, motivation, and obstacle items were inspected with respect to floor or ceiling effects which would then be excluded from the analyses (specifically, items for which ≥ 90% of participants indicated the lowest or highest category). Additionally, Cronbach’s alpha was calculated as an indicator for reliability for each scale. We planned to exclude items that reduced the Cronbach’s alpha of a scale by more than .10. Considering these three quality checks in the complete datasets, no survey items were excluded. Overall, reliability analyses of the survey items showed excellent reliability for the attitude (α = .94) and intention scale (α = .93), good reliability for the motivation (α = .8) and obstacle scale (α = .84), and adequate reliability for the subjective norm (α = .72) and perceived behavioral control scale (α = .72). Thus, all scale items were used to calculate mean scores for the attitude, subjective norm, perceived behavioral control, intention, motivation, and obstacle scale respectively for each participant. All preprocessing scripts are available in the supporting information (see S1 Script).

Descriptive reports

Several descriptive reports are provided to gain qualitative insight into the psychological research community’s experiences, opinions, and behavior with respect to preregistration, and display participants’ suggestions for improving the preregistration process. Since incomplete datasets were also used for these reports, parameters are based on all responses given to the respective item (N is provided in each case).

Open text inputs of both the open “other” options of selection items and the open text input items were analyzed to identify common themes. The following mixed-methods approach was used: Two coders qualitatively identified themes mentioned by the participants and subsequently categorized all responses accordingly in order to receive a frequency measure that was analyzed quantitatively. Specifically, open text inputs of each item were shuffled and the first 10% of these shuffled responses were used to establish initial categories of themes. Coders read the responses and added each theme as a column in a coding sheet. It was coded whether the respective theme appeared in the other responses (= 1) or not (= 0). If a coder encountered new relevant themes, they were added and coded later. Nonsense responses were excluded during coding. Ambiguities were discussed and solved in pairs. If no solution could be found, a third coder was consulted. After all responses had been coded, the sums for each column (= theme) were calculated to obtain the frequency of how often a theme was mentioned over all responses. Results from the open text input items are displayed in the text and stacked bar plots, and the results of the analysis of “other” options were added to the individual items’ response presentations. For brevity, the reports of the open text input items include only those themes that were mentioned in more than 5% of the respective comments, and for the “other” comments, themes which were indicated only by single participants are omitted. Wherever applicable, the results were compared between participants that had preregistered before or not. Both the coding sheets and the open text inputs are published alongside the other data. Furthermore, all analysis scripts are included in the supporting information (see S3 Script).

Perceived importance and intention to preregister.

First, we compared the perceived importance of preregistration (seven-point labeled answer scale item) and the intention to preregister in the future (scale of three items) between researchers with different academic degrees by conducting one-factorial ANOVAs with intention/importance as the dependent variable and degree as the independent variable. These analyses were conducted separately for participants with and without preregistration experience and corrected for multiple testing using the Bonferroni-Holm method. For the analysis of intention in participants without preregistration experience, a Brown-Forsythe test for equality of means was used since variances were not equal and group sizes differed. The results showed that intention and perceived importance were similar between academic groups for participants with preregistration experience (both p_cor = .952) but differed for participants that had not preregistered before (importance: F(3, 96) = 4.38, p_cor = .018; intention: F(3, 63.35) = 4.94, p_cor = .015), as displayed in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Importance and intention.

Mean values for A) the perceived importance of preregistration (one item) and B) participants’ intention to preregister (scale of three items, see S1 Table). Scales ranged from 1 = “Strongly disagree” to 7 = “Strongly agree” and were recoded to “-3 to +3” for data analyses. The error bars show + / − 1 SD.

https://doi.org/10.1371/journal.pone.0281086.g003

Comparison of worries and actual problems.

In addition to examining these overall benefits and drawbacks, we also directly compared the worries of researchers who had not yet preregistered with the problems encountered by participants when preregistering their study in the past (multiple choice items, see Table 2). Worries associated with preregistering were, above all, high time costs (61.7% of 94 responses), low flexibility (54.26%), and the inability to deal with deviations, that is, that changes might not be possible (46.81%) or would lead to decreased credibility (44.68%). However, while the time costs were also the most indicated problem encountered by researchers that had preregistered (45.26% of 190 responses), low flexibility (15.26%) and deviations (changes not possible: 13.68%; loss of credibility: 17.37%) were not considered as problematic. Instead, researchers were more insecure what aspects needed to be included in the preregistration (41.05%). Additional problems experienced by the sample, as indicated by the open text input, comprised problems concerning changes after the preregistration (6.32%), problems with preregistration templates (2.11%), or a lack of understanding among reviewers (1.58%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of worries and actual problems regarding preregistration.

https://doi.org/10.1371/journal.pone.0281086.t002

Proportion of participants that read preregistrations.

When asked how often they read the corresponding preregistration of a paper (single choice item), only about a third of participants indicated that they always (4.57% of 328 responses) or most of the time (24.09%) did so. The remaining participants either rarely (39.63%) or never (18.9%) read the corresponding preregistration to a paper, or even do not recall reading a paper that was preregistered (12.8%, see Fig 4). Specifically, while only few participants that had preregistered before could not recall reading a paper that was preregistered (5.02% of 219 responses), for participants without preregistration experience this was more common (28.44% of 109 responses).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. How often do participants read preregistrations.

Above each bar, the percentage (frequency) of participants who selected the respective answer is shown.

https://doi.org/10.1371/journal.pone.0281086.g004

Persons and institutions that influence the decision to preregister.

Additionally, participants were asked which other persons or institutions influence their decision to use or not to use preregistration (multiple choice item). For early career researchers (i.e., participants with a bachelor’s or master’s degree), their supervisor was the biggest influence, while for the more senior groups, peers/colleagues and co-authors were considered the most influential. Overall responses as well as differences between academic groups and participants with and without preregistration experience are displayed in Fig 5. Other important influences, as indicated in the “other” comments, comprised researchers’ own scientific and moral values (1.83%) or students (0.61%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Parties that influence participants’ decision for or against preregistration.

Results are displayed separately for participants with different degrees (A-D). Above each bar, the percentage (frequency) of participants who selected the respective answer compared to all participants that responded to the item is shown. With preregistration experience: A) n = 23, B) n = 70, C) n = 79, D) n = 48. Without preregistration experience: A) n = 24, B) n = 34, C) n = 34, D) n = 17. Multiple options could be selected.

https://doi.org/10.1371/journal.pone.0281086.g005

Suggestions to improve preregistration.

In the final section of the survey, participants were asked for suggestions to increase the motivation and reduce obstacles of preregistration, and to improve the preregistration process in various open text input items.

The main ideas to increase the motivation to preregister were better incentives (e.g., funding, advantages for publication, job applications or tenure; 16.79% of 131 responses), providing more education (14.5%), and making preregistration mandatory (14.5%). To reduce obstacles to preregister, education was also the most frequently mentioned suggestion (27.78% of 108 responses). Other ideas included taking off pressure, for example, by destigmatizing deviations (12.04%), and providing clear and easily accessible templates (11.11%). An overview of all suggestions to increase the motivation and decrease obstacles to preregister is given in Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6.

Suggestions to A) increase the motivation and B) decrease obstacles to preregister. Next to each bar, the percentage (frequency) of participants who indicated the respective theme is shown. A) With preregistration experience: n = 78, without preregistration experience: n = 53. B) With preregistration experience: n = 68; without preregistration experience: n = 40.

https://doi.org/10.1371/journal.pone.0281086.g006

Regarding suggestions to facilitate the preregistration process, participants were first asked about template improvements. Provided ideas included, above all, easy usability (18.18% of 55 responses), clear guidelines and standardization (16.36%), and good examples (12.73%). Furthermore, participants suggested a more automated process with many possibilities (e.g., including tables and figures; 10.91%). Specialized templates should be available, fitting different research situations (9.09%), and ideally with multiple levels of complexity (e.g., short templates where extra items can be added depending on the study, and “other” options; 5.45%) which allow for much flexibility (e.g., by also providing a “not yet planned” option; 5.45%). When asked directly whether they would prefer a more open or automated preregistration process (single choice item), participants expressed that they preferred a more restricted template that gives a lot of suggestions and reminds you of left-out information (53.19% of 188 responses) over a more open template where you are free to write what you want (34.57%). Furthermore, 6.38% of participants do not use templates at all and write their own text, 2.13% indicated that they preferred an open template with details or a mix of both answers, and 1.06% indicated that they had no preference or made this decision based on their expected data. Regarding the preregistration process (single choice item), participants also preferred a more automated, computer-assisted (57.84% of 185 responses) over a more open, self-administered process (37.3%). Again, some participants preferred a combination of both (1.62%) or said this depended on their project (1.08%).

Enhancing the usability was also often mentioned as a suggestion for improving preregistration repositories (47.62% of 42 responses), with user-friendliness and possible integration into a central network being the main points mentioned. Additionally, simple tracking of deviations (e.g., by enabling timestamped revisions) was emphasized (11.9%).

Regarding the review process, participants suggested making it as quick as possible (12.77% of 47 responses). Additionally, the preregistration should be checked against the final manuscript (10.64%), while deviations (10.64%) and null results (6.38%) should be more accepted. Lastly, participants highlighted the need for detailed and clear feedback during the reviewing and revisions process (6.38%).

To better integrate preregistrations into papers, participants suggested to make preregistration mandatory (23.08% of 39 responses), and to display the link to the preregistration more prominently in the paper (23.08%). Open science badges [75] that are displayed prominently in research articles and signal to readers which open science techniques are associated with this article (e.g., preregistration) were also mentioned (15.38%). These already exist today. In addition, participants requested that deviations should be clearly described (7.69%) using explicit guidelines (5.13%), and that in general it should be clearer what was preregistered and what was not (5.13%).

Lastly, ideas were requested for education on preregistration. Here, it was suggested as a priority that preregistration should be integrated into the university curriculum (36.54% of 52 responses) and that there should be mandatory courses and workshops (34.62%). Improved accessibility of education resources, example preregistrations and clear templates, and guidance on how to deal with deviations were specific suggestions to improve researchers’ knowledge about preregistration (each 5.77%).

Table 1 in S6 Text provides an overview of participants’ suggestions for improving preregistration templates, repositories and publication, review, integration into papers, and education, displayed separately for participants with and without preregistration experience.

Hypotheses tests

To find out how the intention to preregister is formed, and whether research experience has an impact on attitudes and the intensity of motivations and perceived obstacles regarding preregistration, we analyzed participants’ responses to the attitude, subjective norm, perceived behavioral control, intention, motivation, and obstacle scale (see S3 Script for all analyses). Only complete datasets (see section exclusion, missing data, and sample size) were considered for these analyses. Before the analyses, the scales were inspected (see section data analysis and preprocessing), and the mean, standard deviation, and distribution of responses were displayed per item in tables for both the general and the database sample (i.e., only including participants recruited via the databases) for easy inspection (see S7 Text). Next, means, standard deviations, and ranges were computed for each scale (see Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Overview of scales used for the hypotheses tests.

https://doi.org/10.1371/journal.pone.0281086.t003

A significance level of α = .05 was used for our hypotheses tests. Before conducting statistical analyses, assumptions were tested for each method (see S2 Script), all of which were fulfilled. If assumptions had been violated and tests had not been robust against these violations, alternative methods would have been used as described in the Registered Report Protocol (DOI: 10.1371/journal.pone.0253950). Furthermore, it was defined a priori how different result patterns would be interpreted, which was also described in the Registered Report Protocol.

For hypothesis 1, we predicted that attitudes, subjective norms, and perceived behavioral control influence researchers’ intention to preregister their studies in the near future. Perceived importance of the topic of preregistration was included as a moderator for attitudes, and previous preregistration experience served as a control variable. To test this hypothesis, a moderated multiple regression model was computed: The intention (i.e., mean score of three items) was included as dependent variable. As predictors, the attitude scale (i.e., mean score of 24 items inquiring about how positive preregistrations are perceived), perceived importance of preregistration (one item), and their product term, the subjective norm scale (i.e., mean score of eight items inquiring about how beneficial participants perceive the social norm of using preregistration), the perceived behavioral control scale (i.e., mean score of five items inquiring about perceived controllability of the behavior), and previous preregistration experience (one item: yes vs. no) were included. We expected that higher scores on the attitude scale (i.e., more positive attitudes, see hypothesis 1.1), higher scores on the subjective norm scale (i.e., higher perceived social pressure, see hypothesis 1.4), higher scores on the perceived behavioral control scale (i.e., higher perceived control, see hypothesis 1.5) are positive predictors for participants’ intention scores (i.e., higher intention to preregister their studies in the near future). Additionally, we expected that the perceived importance of preregistration is a positive predictor for intention (see hypothesis 1.3), and significantly moderates the strength of the influence of attitudes positively (see hypothesis 1.2). Furthermore, we expected that the overall model including all predictors can significantly predict researchers’ intention to preregister their studies in the future (see hypothesis 1.6). Because of these directional hypotheses, the regression weights were tested in a one-tailed fashion. Standardized regression weights were interpreted and compared. Preregistration experience was included into the model to control for a potential sampling bias. We had no strong predictions regarding the direction of this effect, since it might be the case that a) researchers who have used preregistration before are more likely to preregister again or b) that their preregistration experiences were negative, and they will less likely preregister again. Therefore, in contrast to the other predictors, this effect was tested in a two-tailed fashion.

As expected, researchers’ attitudes (hypothesis 1.1), t(282) = 10.07, p_one-sided < .001, β = .457, subjective norms (hypothesis 1.4), t(282) = 3.87, p_one-sided < .001, β = .148, and perceived behavioral control (hypothesis 1.5), t(282) = 4.65, p_one-sided < .001, β = .199, significantly predicted their intention to use preregistration positively. Furthermore, perceived importance of preregistration was a significant positive predictor for intention (hypothesis 1.3), t(282) = 7.16, p_one-sided < .001, β = .355. Overall, attitudes and perceived importance were the strongest predictors of intention. In contrast to our hypothesis, perceived importance did not moderate the effect of attitudes on intention positively (hypothesis 1.2), t(282) = -3.92, p_one-sided > .999, β = -.175. Instead, the interaction effect was negative. When tested two-sided (as specified a priori), preregistration experience did not significantly predict the intention to use preregistration, t(282) = 1.87, p_two-sided = .062, β = .071. As expected, the overall model significantly predicted the intention to preregister in the future (hypothesis 1.6), F(6, 282) = 137.6, p < .001, explaining 73.99% of the dependent variable’s variance (adjusted R²).

The second hypothesis predicted that the research experience (i.e., the years someone has worked in research) influences researchers’ attitudes (see hypothesis 2.1) as well as motivations (see hypothesis 2.2) and the perception of obstacles (see hypothesis 2.3). To investigate this, three multiple regression models were computed: Mean scores on the attitude scale (i.e., mean score of 24 items inquiring about how positive preregistrations are perceived), motivation scale (i.e., mean score of ten items measuring how strongly participants agree with potential motivations to preregister), and obstacle scale (i.e., mean score of ten items measuring how strongly participants agree with potential obstacles to preregister) served as dependent variables in the respective regression models, while research experience (i.e., the years someone has worked in research) and the preregistration experience (control variable) served as predictors. As our hypotheses regarding the impact of research experience on attitudes and the intensity of motivations and perceived obstacles were non-directional, the regression weights were analyzed with two-tailed tests. Since three regressions were conducted to test one set of hypotheses, a Bonferroni-Holm correction was used to adjust the alpha level.

As predicted, the research experience significantly influenced researchers’ attitudes (hypothesis 2.1), t(286) = -5.04, p_cor < .001, β = -.279, as well as their perception of motivations to preregister (hypothesis 2.2), t(286) = -5.03, p_cor < .001, β = -.279. However, it had no effect on the perception of obstacles (hypothesis 2.3), t(286) = 0.49, p_cor = .623, β = .027. The influence of research experience on attitudes and motivations was negative, implying that researchers who have only worked a small amount of time in research have more positive attitudes and feel more motivation to preregister than individuals who have worked in research longer. In contrast, obstacles were perceived the same regardless of the length of research work. Preregistration experience was a significant predictor for attitudes, t(286) = 4.38, p_cor < .001, β = .242, and for the perception of motivations, t(286) = 4.08, p_cor < .001, β = .226, and obstacles, t(286) = -6.26, p_cor < .001, β = -.348. Participants with preregistration experience had more positive attitudes and perceived motivations to preregister more strongly, while they agreed less with the presented obstacles. Research experience and preregistration experience combined explained 12.29% of the variance of attitudes, 11.59% of the variance of motivations, and 11.45% of the variance of obstacles (adjusted R², all p_cor < .001). For a clear overview, the results of all regression models are presented in Tables 1 to 4 in S8 Text.

Positive perception of preregistration

In our sample, around 62% of researchers reported having preregistered before. This fits with the positive trend described by Christensen et al. [24], according to which in 2017, 44% of the psychological researchers they surveyed had preregistered at least once (however, it could also indicate a sampling bias, see section limitations). In our survey, almost 50% of those with a bachelor’s degree had already preregistered, and in the more senior research groups the proportions were even higher. On average, preregistration was perceived positively, and most researchers planned to preregister their studies in the future. Specifically, 83.6% of participants had scores above zero on the intention scale, with zero indicating an undecided opinion and positive values indicating they were more likely to intend to preregister in the near future. It was even proposed by many participants that preregistration should be made mandatory.

The benefits of preregistration indicated by our sample were associated with improved planning and documentation, transparency, higher quality, and replicability of research, which is consistent with the arguments for preregistration found in the literature (e.g., see [15, 16, 29]). The overall positive outlook found in our survey aligns with recent studies by Logg et al. [27] and Sarafoglou et al. [76] which also showed mostly positive attitudes of researchers regarding preregistration when inspecting generational differences and the perceived impact of preregistration on the research workflow.

Perceived obstacles and how they might be overcome

Various obstacles were identified that currently discourage psychological researchers from using preregistration, many of which align with arguments made by critics of preregistration [15, 16, 28, 30–32]. For some of these, solutions already exist. Others can be used as a starting point for improvement, as outlined below.

Facilitating factors and influence of research experience

Besides these descriptive insights into the psychological research community, we investigated two specific research questions. First, we wanted to identify which factors facilitate or prevent the usage of preregistration. We expected that the predictors of the theory of planned behavior (attitudes, subjective norm, and perceived behavioral control) [40, 41] as well as the perceived importance of preregistration (and its interaction with attitudes) influence researchers’ intention to preregister. Our results show that, except for the expected interaction effect, all predictors indeed significantly influenced researchers’ intention to preregister their studies, with a huge proportion of variance of intention being explained (R² = 73.99%). Thus, we conclude that the theory of planned behavior is indeed well suited to be used to predict researchers’ intention to preregister, with attitudes and perceived importance being the strongest predictors. Therefore, to promote preregistration, it might be especially beneficial to highlight the relevance of preregistration for each individual researcher, for example, by emphasizing individual benefits like the usefulness for planning, which was the highest ranked benefit reported by participants in our survey.

Second, we examined whether research experience, i.e., the time someone has worked in research, influences attitudes and the perceived intensity of motivations and obstacles. Indeed, research experience was a significant predictor of both attitudes and the perceived intensity of motivations, while it did not significantly influence the perception of obstacles. More specifically, it was the case that psychologists newer to research had more positive attitudes toward preregistration and agreed more with statements expressing a high motivation, whereas obstacles were perceived independently of the research experience. These results could be interpreted as supporting the assumption that early career researchers are more invested in preregistration than more senior researchers. However, in our sample, 71.67% of participants with a habilitation or full professorship had already preregistered and for the most part also valued preregistration as important (especially those who had already preregistered, see section perceived importance and intention to preregister). Therefore, it can be argued that attitudes toward preregistration tended to be positive for this group as well, only more so for early career researchers.

Limitations

The main limitation of our study is a potential selection bias of individuals who are more positive towards preregistration or have preregistered before. We tried to counteract this by recruiting researchers who were identified through their articles in large databases. We also reported, wherever possible, the results of individuals with and without preregistration experience; excluded participants recruited through the OSF from reports intended to represent the overall psychology community; and included preregistration experience as a control variable in our analyses. Nevertheless, bias cannot be conclusively ruled out, and the proportion of participants with preregistration experience was very high in our sample (61.81%).

On a more general note, it should be borne in mind that that preregistration has a variety of benefits that help increase scientific rigor, but it is not a panacea (as argued by [87]). Therefore, preregistration should ideally be used together with complementary open science practices increasing transparency and confidence (see [87] for an overview). Even so, for comprehensive and long-term improvement of the credibility of scientific evidence, the academic system needs to be revised, for example, by incentivizing methodological quality instead of results.

Future research

The theoretical benefits of preregistration are obvious and have been argued for by numerous authors (e.g., [see 15, 16, 21, 23]). However, these should be tested more thoroughly empirically to ensure that preregistration is used and works in the intended way and to realize its full potential. An example for this is the current inadequate reporting of deviations [34–37], which might compromise the positive effect of preregistration on transparency. Our survey provided further support that researchers worry about negative consequences after the occurrence and reporting of deviations. We suggest developing a standardized reporting method for deviations that should then be empirically tested to see if it leads to deviations becoming more transparently disclosed.