Review team

The review team consists of a creativity research expert with PhD training in health research methods, a PhD in clinical epidemiology with experience designing and conducting experimental studies, a clinician-scientist with leading experience in health research in the field of orthopaedics, an MD candidate with training in research methods and prior experience conducting methodological studies of creativity, and an MSc candidate with training in research methods and prior experience with systematic reviews and methodological studies.

Study objectives

The primary objective is to determine the fragility of statistically significant test results derived from AUT testing. The secondary objective is to identify factors associated with fragility and explore the difference in fragility between randomized and non-randomized trials.

Selection criteria

Studies must meet the following eligibility criteria to be included: (1) two-group studies that report a statistically significant result (p < 0.05) on the AUT score; (2) published in a peer-reviewed journal; and (3) reporting details on the sample size, mean values, and spread statistics (e.g., standard deviation, confidence interval, range) for the significant outcome for both arms. The exclusion criteria will be: (1) letters, reviews, meta-analyses, protocols, and abstract-only publications; and (2) studies utilizing crossover designs and within-subjects designs. Any studies not published in English will be translated with Google Translate.

Search strategies

We will systematically search Medline, Embase, PsycInfo, and ERIC from inception to July 21, 2025. In each database, we will perform comprehensive searches for all studies that included different permutations of the keyword ‘AUT,’ such as ‘alternate uses,’ ‘alternative uses,’ and ‘Guilford’s uses,’ in order to capture all studies that referenced the Alternate Uses Task. The full search strategy for each database is included in S2 File. Additionally, we will conduct a manual search of the reference lists of all included studies and relevant review articles to identify any additional eligible studies that may not have been captured in the initial database search.

Screening

Two reviewers will independently screen the titles and abstracts of all records identified in the search. Any disagreements will be resolved through discussion, and if needed, a third reviewer will be consulted. Full texts will then be retrieved for studies deemed potentially eligible. The same dual-reviewer process will be followed during full-text screening. We will use Covidence (Melbourne, Victoria, Australia) as the platform for screening studies [16]. The results of our study screening procedure will be presented using a PRISMA flow chart.

Data extraction

We will develop a standardized data extraction template to extract the relevant data from eligible studies. Two reviewers will independently extract the data, with any discrepancies resolved by discussion to achieve consensus or adjudication by a third reviewer. Extracted data will include study characteristics (journal name, journal impact factor, country of conduct, year of publication, number of citations), type of intervention, sample size, sample demographics, p-value, and the mean and standard deviation for the AUT scores. If the standard deviation is not available, we will extract other available measures of spread, such as standard error or range.

Continuous fragility index

The continuous fragility index (CFI) for continuous outcomes estimates the number of data point shifts needed to render a result non-significant (p ≥ 0.05). To determine CFI, Caldwell and colleagues described an iterative algorithm that performs a Welch’s t-test on two datasets representing a continuous outcome variable [11]. The process begins by identifying the group with the higher mean, and the data point closest to the mean in the higher-scoring group is transferred to the lower-scoring group. Then, the Welch’s t-test is recalculated. This process of shifting data points and recalculating is repeated, one data point at a time, until the p-value is above 0.05. The CFI is defined as the number of data points that must change to cross this statistical threshold. However, this method requires complete individual participant data. Most published studies only report summary statistics, such as group means, standard deviations, and sample sizes. To address this limitation, Caldwell et al. developed a simulation-based approximation method that reconstructs candidate datasets assuming a normal distribution around the reported group statistics, allowing researchers to estimate the CFI using only aggregate data [11].

For this review, CFI values will be calculated using an online calculator based on the simulation methodology established by Calwell and colleagues. For each study, the required inputs will include the sample size, mean, and standard deviation for each of the two groups. If studies report spread statistics other than standard deviation, such as standard error or the range, we will estimate the standard deviation using appropriate conversion methods [17,18]. To minimize bias introduced by random data generation, each CFI calculation will be repeated across five simulation iterations. We will report and use the average CFI for all further analyses. A tolerance level of 0.01 will be set to ensure that simulated datasets closely match the original summary statistics reported in each study.

Risk of bias assessment

Two reviewers will independently assess the risk of bias for each included study using standardized tools appropriate for the study design. Discrepancies between reviewers will be resolved through discussion; if a consensus cannot be reached, we will consult an additional reviewer.

For randomized studies, we will use the Cochrane Risk of Bias 2 tool [19], which evaluates five domains of potential bias: (1) bias arising from the randomization process, (2) bias due to deviations from intended interventions, (3) bias due to missing outcome data, (4) bias in measurement of the outcome, and (5) bias in selection of the reported result. For each domain, studies will be rated as having a low risk of bias, some concerns, or a high risk of bias, with an overall risk of bias judgment derived according to Cochrane guidelines.

For non-randomized studies, we will use the Risk Of Bias In Non-randomized Studies – of Interventions (ROBINS-I) tool [20]. This tool evaluates seven domains: (1) bias due to confounding, (2) bias in selection of participants, (3) bias in classification of interventions, (4) bias due to deviations from intended interventions, (5) bias due to missing data, (6) bias in measurement of outcomes, and (7) bias in selection of the reported result. Each domain will be rated as low, moderate, serious, or critical risk of bias, or no information if details are insufficient for judgment. Each study will be classified as having a low, moderate, serious, or critical risk of bias, or as having no information if the details are insufficient for judgment.

We will present groupings of overall risk of bias in tables and prepare visualizations (e.g., traffic light plots) to facilitate a clear presentation of the overall risk of bias across studies. For the purposes of further analyses, we will consider a serious or critical risk of bias in non-randomized studies to be equivalent to a high risk of bias in randomized studies. Additionally, the moderate and some concerns categories for non-randomized and randomized trials, respectively, will be considered equivalent, allowing us to combine data from both study designs.

Statistical analysis

To meet our primary objective, we will report the CFI of included studies descriptively, presenting the median and interquartile range (IQR). We will also graphically show the distribution of CFIs using a histogram. Additionally, we will report the median CFIs with IQRs for the following sub-groups in a table: journal impact factor, year of publication, country of conduct, number of citations, type of intervention, and risk of bias classification.

For the secondary objective, we will perform multivariable linear regression to explore factors associated with statistical fragility in AUT studies. Specifically, we will consider journal impact factor, year of publication, sample size, country of conduct, the number of citations, and risk of bias classification. Based on recommended guidelines [21], we will only include the categorical variables (i.e., country of conduct and risk of bias classification) if the events per variable threshold of 10 is met. For the model, we will report beta regression coefficients along with the associated 95% confidence intervals. If needed, we will transform the CFI values to meet the assumptions of linear regression modelling [22]. For our final secondary objective, we will perform a Mann-Whitney U test to determine if there is a statistically significant difference in the CFI of randomized and non-randomized trials.

Study timeline

We plan to begin this systematic review in September 2025, when we will conduct all database searches. Title and abstract screening, followed by full-text screening and data extraction, will be completed by the end of October 2025. We plan to prepare the first draft of the manuscript by February 2026, with final submission to a peer-reviewed journal targeted for April 2026.