Research aims

The overall research aim is to gain an insight into the sample size practices of funded and non-funded RfPB applications. Specifically we aim to:

• Describe the methodological characteristics of a sample of external randomised pilot applications that were successful, as well as unsuccessful, at obtaining RfPB funding.
• Gain a better understanding of the sample size practices used by researchers who apply for RfPB funding.
• Develop an understanding of what differs in terms of sample size practices, experimental design and objectives, between external randomised pilot trials which obtain RfPB funding and those which do not.
• Gain an insight into committee feedback provided to researchers of funded and non-funded pilot trials.
• Provide guidance to assist researchers when planning their pilot trials

Research objectives

Our primary and secondary objectives are as follows:

Primary objective

1. Describe the planned sample size and sample size justification of funded and non-funded applications.

Secondary objectives

1. Describe the methodological characteristics of funded and non-funded applications focusing on:
   1. i. Planned primary and secondary objectives.
   2. ii. Experimental design.
2. Synthesize committee feedback related to sample size practices such as justifications, as well as the experimental design and objectives in funded and non-funded applications.
3. To explore whether there is an association between the objectives and the sample size justification.
4. To explore whether there is an association between the methodological characteristics and whether the application received funding. In particular we aim to answer the following questions:
   1. i. Are studies that state their feasibility related objective(s) successful or unsuccessful at obtaining funding?
   2. ii. Are studies that state their sample size justification(s) successful or unsuccessful at obtaining funding?
   3. iii. Is there an association between the sample size justification and whether the application received funding?
5. Describe how the planned sample size justifications change between stage one and stage two, if applicable.
6. Produce guidance based on the above findings that could be used to assist researchers when planning their pilot trials.

Study design

The protocol was developed in line with the PRISMA-P statement [14] and elaboration and explanation document [15] (S1 Checklist). The protocol is registered on Open Science Framework, a research project management tool [16]. The registration DOI is https://doi.org/10.17605/OSF.IO/PYKVG

Eligibility criteria

As the terms ‘pilot’ and ‘feasibility’, as well as the terms ‘trial’ and ‘study’ are often used synonymously within the PAFS literature, they will also be used synonymously in this current work. Therefore, although the focus of this current work is external randomised controlled pilot trials, if an eligible study, which is an external pilot trial by design, is described as a ‘feasibility study’ or ‘feasibility trial’ or ‘pilot trial’ or ‘pilot study’ then the study will be considered for inclusion.

Inclusion criteria

1. Described as a ‘pilot study’, ‘feasibility study’, ‘pilot trial’, ‘feasibility trial’.
2. Within the context of any area of health care research.
3. Randomised controlled trial.
4. External pilot design.
5. Primary research.
6. Applications with a funding decision received from RfPB between July 2017 to July 2019 inclusive.

The dates chosen were in line with a request from other collaborators who are not involved within this present research but were similarly requesting applications from RfPB for research purposes [17].

Exclusion criteria

1. Non pilot/feasibility experimental design.
2. Economic evaluation.
3. Internal pilot design.

RfPB only fund English organisations and do not fund pure audits or animal research. Therefore any non-English, health care audits, and non-human applications are by definition excluded.

Information sources

Two sources of data will be used, namely NIHR RfPB stage one and two pilot and feasibility study applications, and the corresponding outcome letters. The Secretary of State for Health and Social Care is acknowledged as the source of the data, in line with the Data Sharing Agreement between the Secretary of State for Health and Social Care of 39 Victoria Street, Westminster, London, SW1H 0EU (the Provider Institution) and Queen Mary University of London (the Recipient Institution).

Search strategy

RfPB had 918 stage one applications from Competition 31 (grant awarded in July 2017) to 37 (grant awarded in July 2019). RfPB searched the titles and plain English summaries for keywords ‘feasibility’, ‘feasible’ and ‘pilot’, and found 341 feasibility studies. They then searched the titles and plain English summaries for random* to identify those with a randomised design. Those that did not have random* in the title or plain English summary, they reviewed the research plan to identify whether the design was randomised or not. This gave 269 applications for a feasibility study with a randomised design. Of these 269 applications, 110 had been invited to stage two of the RfPB application process (41%).

The 269 applications corresponded to 236 lead applicants who were contacted by RfPB for their consent. Of these, 89 applicants consented, corresponding to 100 applications, of which 53 applications made it to stage two of the RfPB application process (53%). Chief investigator permission was sought from RfPB using the standard wording in S2 Appendix. JH and BM created the wording of consent requests to RfPB applicants in line with their legal team, and contacted researchers using the consent form to ask for their permission to share research funding applications with us (as well as a team at the University of Oxford doing a similar project). JH and BM accessed the applications between 12^th March 2021 and 30^th June 2021, and had access to identifying information as they collected and anonymised the applications. We will only be using the redacted research funding applications in this project.

We submitted a formal ethics application for review to Queen Mary, University of London ethics team. We were advised that the project did not need ethical approval and obtained a formal letter of objection for this (S3 File).

Application selection process

We will screen the 100 applications available to us to identify those satisfying all the eligibility criteria. The full text RfPB applications will be reviewed. Any applications that fulfil the inclusion criteria and do not meet any of the exclusion criteria will be eligible. If the eligibility of any applications are unclear after reviewing the full text RfPB application then SE, GL or CR will be consulted. At most 100 eligible RfPB pilot study stage one applications will therefore be included within the review, along with any of their respective stage two applications. A precision-based sample size calculation shows that 100 applications would enable us to estimate the proportion of applications reporting a sample size justification to within approximately 8%, assuming a 95% desired confidence level and an estimated proportion of 0.80.

Data extraction

We have created a bespoke data extraction sheet in Excel to record the data. The data extraction form was piloted by CC on 10% of eligible articles to determine if additional fields were required, or items in the codebook needed further clarification. Based on this, Table 1 shows the data that will be extracted from the RfPB applications, if available:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Data extraction items.

https://doi.org/10.1371/journal.pone.0343981.t001

CC will extract data from eligible applications, with SEd verifying 20% of the data extracted. If discrepancies, ‘pairs of statements that could not both be true’ are found, then these will be resolved by discussion, referring to SE, GL or CR if there is no consensus. If any information is unclear, a note of this will be made in the review.

We expect data collection to be completed by May 2026.

Sub study

A case study will be performed on any pilot trial applications identified related to a common medical area, asthma, as part of work funded by the Asthma UK Centre for Applied Research. Asthma is a common long-term lung condition that affects about 8 in every 100 people in the UK, with four people estimated to die every day in the UK from asthma attacks [18]. It is therefore an important health topic and there is a need to make sure that research related to asthma is completed to a high standard [19]. Based on sifting, we expect around 5 applications to be specifically related to asthma. Therefore, we will focus on descriptive summaries. In particular, for these applications, we will describe the planned sample size and sample size justification of the funded and non-funded applications, and summarise any sample size related committee feedback. This will provide the opportunity to explore in more depth sample size practices of RfPB applications related to asthma.

Analysis

A record of the number of articles assessed at each stage in the study selection process will be made and a PRISMA flow chart will be produced [20].

The primary outcomes will be the planned sample size and the planned sample size justification(s). Secondary outcomes will be the planned primary objective(s), the planned secondary objective(s), the experimental design, and the sample size related committee feedback contained within the outcome letters.

Quantitative analysis

Variables that appear to follow a continuous normal distribution will be summarised with means and standard deviations; otherwise, medians, interquartile ranges and ranges will be used. We will summarise categorical variables using frequencies and percentages. All summarised data will be presented in tables, and in particular for the objectives looking at associations, we will present two-way tables. We will not perform statistical tests when looking at associations as our study is not powered to do so. All analyses will be conducted using the statistical software Stata version 18.5 or higher [21].

Qualitative analysis

A pragmatic paradigm will be utilised for this research [22]. The outcome letters will be analysed by SEd using thematic analysis [23], based on the results of a piloting exercise of a random 10% sample of the outcome letters. Thematic analysis will broadly follow the following steps suggested by Braun and Clarke [23]: familiarisation of the data, generating codes, identifying themes or categories, reviewing themes or categories, defining and naming themes or categories, and identifying examples from the data. We will not monitor for data saturation or information power, as there is no possibility of modifying the analysis sample beyond the 100 applications that have been made available to us.

An inductive approach will be used whereby codes and categories will be identified, defined and labelled in response to the data as SEd reads the outcomes letters and uses her interpretation. An inductive approach involves identifying codes as we read through the data. An inductive approach is routinely used when exploring new phenomena [24], which is appropriate here as we explore previously unexamined committee feedback, and there is a lack of clarity as to whether there are stock phrases or structured feedback or open text in the committee feedback. Furthermore, the variety of different RfPB funding committees, their varying personnel and the range of years of the committee feedback letters means that an inductive approach is the most appropriate approach in the first instance. Reflective memos will be recorded, which will contain SEd’s and HMM’s personal impressions whist reading the committee feedback and a description of how and why key analytical decisions were made. The reflective memos will act as an audit trail. SEd and HMM will practice reflexivity throughout this research. HMM will independently double code the codes and categories from at least 10% of outcome letters. Any major discrepancies regarding the codes and categories will be discussed by SEd and HMM until consensus. If consensus cannot be reached SE, GL or CR will be consulted and an additional 10% will be double coded. Qualitative data will be managed and digitally coded within Microsoft Word and Excel. After the coding frame has been finalised, illustrative examples of each code will be identified from the outcome letters and will be reported. The results from this research will be reported in line with the Standards for Reporting Qualitative Research (SRQR) guidelines [25]. While conducting our analysis, we will present our research at institutional work in progress meetings for feedback to identify and address possible biases.

Findings from all the objectives, quantitative and qualitative, will be summarised through discussion with the project contributors to produce a reader-friendly guidance document to assist researchers when planning their external pilot trials.