Introduction

Comparing and synthesising data across studies is fundamental to answering important clinical questions about health and clinical care. The ability to compare and combine data is, however, dependent on the compatibility of data. The development and use of a core data set, such as a core outcome set (COS), is one established approach to improve data synthesis [1]. COS are being increasingly developed for use in comparative effectiveness trials but are also important to standardise reporting in other contexts, such as disease- or treatment-specific registries. As such, there is growing interest in core set development methodology to optimise the efficiency and effectiveness of the development process and Delphi surveys are commonly used to prioritise core set items [2,3].

The Delphi survey method uses a simple framework in which researchers have flexibility in the methods used to seek consensus on a core set [4]. A key feature of this process is the provision of feedback to participants in subsequent survey rounds, which enables other participants’ opinions to be considered before re-prioritisation of items. Whilst ‘describe how participants receive feedback’ is an item on a checklist to standardise COS development [3], there is no agreed format of how it should be presented. Previous studies have shown the importance of feedback presentation, since patients and healthcare professionals prioritise different types of outcomes [5] and providing feedback from individual stakeholder groups may improve consensus between groups and agreement in items prioritised [6]. But how stakeholders interpret and utilise feedback, is unknown. Most Delphi studies use a Likert scale to score items [4,7] and therefore feedback is commonly presented as a summary statistic such as a mean or median value, with or without an indicator of range (e.g., standard deviation, inter-quartile range) [4]. Several published studies have noted that rating scales, such as Likert, lead to less variation in scores across items as repetition of scores is allowed (compared to ranking) [8] and that health Delphi studies tend to prioritise more items due to their perceived importance [8]. Together, in a health Delphi, this results in many items being scored highly and does not allow differentiation between the items considered the most important; a problem when the purpose of Delphi surveys is to reduce the number of items prioritised in each round. Given the increasing interest in the development of core sets in health service research, it seems pertinent to identify ways to improve the efficiency of prioritisation and consensus.

The aim of this nested randomised controlled trial (RCT) was to explore whether the provision of summarised feedback in addition to usual feedback (enhanced instructions), promoted the prioritisation of items compared to those receiving usual feedback only (standard feedback). This was explored within a Delphi survey for the prioritisation of items in a core data set for an obesity surgery registry.

Methods

Delphi survey

Survey development.

A comprehensive list of potentially relevant items for the core sets was identified from earlier work, including a previous COS for bariatric surgery trials [5], a bariatric registry data dictionary project [10], systematic literature reviews and patient and public involvement group discussions. Items from each source were collated into a single long list where duplicates were removed and overlapping themes combined. The study team, comprising methodologists and health professionals, grouped the items into broader domains which reflected the different phases of data collection within bariatric registries. These items were included in a 97-item questionnaire divided into sections which would go on to form the four core data sets – 1. baseline information, 2. effectiveness outcomes, 3. surgical procedure information and 4. potential complications and side-effects of surgery. Free text boxes were included at the end of each section to enable stakeholders to propose new items. The electronic questionnaire was hosted on REDCap data capture software [11,12] and sent to all IFSO members via email.

Prioritisation of items.

In round 1 of the Delphi survey, all participants received identical surveys. They were asked to rate the importance of each item for inclusion in the final core data sets on a Likert scale of 1–9. A score of 7–9 indicated an item of critical importance, 4–6 an item important but not critical, and 1–3 an item of limited importance. Median scores for each item were calculated for all participants combined and for each stakeholder group (bariatric surgeons, dietitians, specialist nurses, psychologists, other). For round 2, in line with COMET guidelines [4], all data items were retained, regardless of their scoring in round 1. All responding participants received the same survey as in round 1 plus anonymised feedback for each item detailing their own score, their peer group’s median score, the other stakeholder groups (combined) median score and the overall median score. In addition to the feedback scores, patients were randomised to receive either ‘standard’ or ‘enhanced’ instructions to guide them in re-rating the items in round 2 (see ‘Randomisation’ below).

The percentage of participants rating each item 7–9 (critically important) after round 2 was calculated. In line with previous core sets [13,14] and COS guidance documents [3,13–16], consensus to prioritise an item was defined, a priori, as being achieved if at least 70% of responding participants scored the item 7–9.

Items were categorised into three groups to facilitate discussion and voting on the final core set at the consensus meeting. These were: (1) items rated 7–9 by ≥95% participants to be ratified ‘in’ if no objections; (2) items rated 7–9 by 70–95% of all participants to be discussed in small groups and voted on by the whole group; and (3) items rated 7–9 by <70% of all participants to be ratified ‘out’ if no objections. Full details of the consensus meeting discussions and voting criteria have been described previously [9]. The final core data sets comprised 12 items [9].

Instructions.

In the round 1 survey, all participants received identical information about core sets and instructions on how to complete the survey (Fig 1). For the round 2 survey, two types of instructions (standard and enhanced) were developed (Fig 2). Standard instructions emphasised the provision of feedback from peer groups and other healthcare professionals and offered simple guidance on how to use the feedback presented when re-rating the data items for round 2 (Fig 2a). This is similar to that used in previous Delphi surveys to develop COS [13,17]. The aim of the enhanced instructions was to provide participants with summarised data of the ‘Top 5’ data items for each core set in order to facilitate re-rating data items in round 2. The Top 5 items were the five items prioritised (scored 7–9) by the highest percentage of participants in round 1. As with standard instructions, the introductory information emphasised the provision of feedback from peer groups and other healthcare professional groups but in addition, highlighted that the Top 5 data items for each core set were also provided (Fig 2b). For each of the four core sets, the Top 5 data items for each of the four sections were displayed (Fig 2b) with instructions on how to re-rate items.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Information and instructions presented to all participants to guide them in the completion of the questions in the round 1 survey.

https://doi.org/10.1371/journal.pone.0348136.g001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Information about the feedback (results) provided and instructions for how to re-rate data items in round 2.

Participants randomised to either standard (a) or enhanced (b) instructions in the round 2 survey.

https://doi.org/10.1371/journal.pone.0348136.g002

Results

Response rates and stakeholder characteristics of randomisation groups

All members of IFSO were invited by email to take part in the survey. The survey link was live for 6 months (April-September 2021) to allow a reasonable time for as many people as possible to complete the round 1 and round 2 surveys. A reminder was sent to all IFSO members in July 2021. In round 1, 272 participants answered at least one question; 260/272 (96%) participants completed section 1, 198 (73%) completed section 2 and 192 (71%) completed the whole survey. The least number of questions answered was three, by five people. All 272 round 1 respondents were randomised to receive either standard (n = 131, 48%) or enhanced (n = 141, 52%) instructions in round 2. In round 2, 123 participants (57 (44%) receiving standard instructions and 66 (47%) receiving enhanced instructions), answered at least one question (response rate of 45%). The majority (122/123 (99%)) completed section 1, 119 (97%) completed section 2 and 114 (93%) completed the whole survey. The least number of questions answered was 12, by one person. There was no difference in round 2 response rates between the two instruction groups. Details of participants’ characteristics, who answered both survey rounds, are presented in Table 1. Most respondents were surgeons, with a higher percentage of surgeons responding in the standard instruction group than the enhanced group (60% vs 52%). Dietitians were also well represented although similarly across both groups (21% standard vs 24% enhanced). The distribution of specialities in this sub-study, was broadly similar to all participants that completed round 1 [9]. Experience was similar between the groups with 60% of standard instructions and 59% enhanced group having >10 years’ experience. Respondents represented 35 countries with around one quarter from the United Kingdom.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Details of participants completing both rounds 1 and 2 for each instruction group.

https://doi.org/10.1371/journal.pone.0348136.t001

Prioritisation of items

The number of items prioritised (scored 7–9 by ≥70%) by participants increased in both groups between rounds. The standard instructions group prioritised 58 (60%) items in round 1 and 70 (72%) in round 2. These additional 12 items were the result of thirteen further items being prioritised and one item that was prioritised in round 1, not being prioritised in round 2 (Table 2). The enhanced instructions group prioritised 53 items (55%) in round 1 and 66 (68%) in round 2–13 extra items. Differences in the items prioritised (scored 7–9 by ≥70% stakeholders) by at least one group in round 1, 2, or both are shown in Table 2. Items not prioritised by any group in either round are shown in Supplementary Table 1, and are also available in the original publication of the main study [9].

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Items prioritised (scored 7-9 by ≥70% stakeholders) by at least one group in at least one round.

https://doi.org/10.1371/journal.pone.0348136.t002

Agreement of items prioritised

In rounds 1 and 2 respectively, 86 (89%) and 93 (96%) items were concordant (prioritised or not prioritised by both groups) and 11 (11%) and 4 (4%) were discordant (prioritised or not prioritised by only one group) (Table 3).

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Agreement between groups in prioritised items.

https://doi.org/10.1371/journal.pone.0348136.t003

In round 1, agreement between groups in prioritised items was ‘substantial’ for core sets 1, 2 and 3b (κ = 0.61, 0.76 and 0.71, respectively) and perfect (κ = 1.0) for core set 3a (Table 3). The overall agreement for the four core sets was ‘substantial’ (κ = 0.77). In round 1, the standard instructions group prioritised 58 items and the enhanced instructions group 53, although there were 11 discordant items (three in core set 1, four in core set 2 and four in core set 3b) (Tables 2 and 3).

In round 2, agreement was found to be substantial for core set 1 (κ = 0.74), almost perfect for core sets 2 and 3a (κ = 0.93 and 0.82 respectively) and perfect for core set 3b (κ = 1.0). The overall agreement for the four core sets combined was κ = 0.90 indicating almost perfect agreement. In round 2, the standard instructions group prioritised 70 items and the enhanced instructions group 66 items. These 66 items were common to both instruction groups. The four discordant items prioritised by the standard instructions group were ‘details of the MDT’ (core set 1), ‘pre-surgery

weight loss’ (core set 1), ‘how well the pancreas produces insulin’ (core set 2) and ‘height of staples used’ (core set 3b) (Tables 2 and 3).

Overall, agreement between groups in prioritised items improved in round 2 for three of the four core sets. Overall agreement was higher in round 2 than round 1 (κ = 0.9 versus κ = 0.77) with only 4% discordant items, compared to 11% in round 1 (Table 3).

Differences in prioritisation of ‘Top 5’ items

The differences between groups in the percentage of participants prioritising round 1 ‘Top 5’ items are shown in Table 4. Differences between rounds 1 and 2 in the percentages of participants receiving standard instructions who prioritised the Top 5 items ranged from −2.8% (name of surgical procedure 96.6% to 93.8%) to +14.1% (hiatus hernia repair 82.8% to 96.9%). The mean change across the four core sets was + 2.3%. In the enhanced instructions group, differences ranged from −6.7% (hiatus hernia 92.0% to 85.3%) to +20.5% (other medical conditions not directly related to obesity 64.1% to 84.6%), with a mean change across the four core sets of +3.2%.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Differences between groups in the percentage of participants prioritising round 1 ‘Top 5’ items.

https://doi.org/10.1371/journal.pone.0348136.t004

Discussion

This study examined the effects of providing summarised feedback on the prioritisation of items during a Delphi survey to reach consensus on the items to be included in a core data set. It was an exploratory randomised design, within the confines of an already planned core set development study, and therefore not powered sufficiently, so findings should be interpreted with this in mind. Results indicated that, overall, enhancing participant instructions in round 2 by including information about which five items were rated the highest in round 1, does not further promote consensus. Rather, both instruction groups prioritised a larger number of items in round 2 than in round 1 and there was little difference in the percentage of respondents prioritising the ‘Top 5’ items in round 2. Additional survey rounds due to lack of prioritisation and subsequent loss of participants due to survey fatigue may diminish the rigor of the core sets. Core sets are a useful tool in standardising research outputs, and gaining consensus is at the heart of this process.

Growing interest in core set methodology suggests that core set developers are trying to establish the best methodology to facilitate prioritisation and consensus. In the last five years (2019-current) [7], 64 studies were registered with the COMET database as investigating ‘COS methods research’. These studies explore numerous aspects of core set development including the ordering of survey items (e.g., patient-reported before/after clinical outcomes) [20], Likert rating scales [21,22] and presentation of feedback to participants in subsequent round(s) [6]. However, currently, there is no definitive method for developing COSs. Guidance documents created to standardise the development and reporting of a COS [3,7,15,16] navigate users through the process but do not prescribe methods for accomplishing COS generation and researchers continue to take different approaches [4]. The COS-STAP checklist states that the definition of consensus and how patients will receive feedback during the consensus process should be described [3], but does not provide guidance about developing survey and/or prioritisation instructions. This study has identified research questions about the role of instructions to achieve consensus during core set development but also highlights other areas of the Delphi process that warrant further exploration.

We hypothesised that the use of enhanced instructions, which provided summarised feedback in the form of the top five items prioritised by participants in round 1 for each core set, would efficiently guide participants through a sea of ‘critically important’ health items to those voted ‘critically important’ by the greatest proportion of participants. We reasoned that participants would be in a more informed position to re-rate the top five items higher, if in agreement, or lower, to indicate disagreement, thereby promoting fewer items being prioritised and consensus reached on a minimal core set. Instead, however, we observed that a greater number of items were prioritised, making the finalisation of a minimal core set, more challenging. The absence of prioritisation in both groups suggests that the presentation of median scores with or without a ‘Top 5’ did not facilitate prioritisation further and the reasons for this need to be considered. It is plausible that, in an effort to improve consensus (by modifying their opinions more in line with others’), stakeholders overlooked the aim of prioritising only ‘critically important’ items, by scoring more items as ‘important’. This finding highlights complexities in the mechanisms by which instructions and feedback influence participants’ behaviour in Delphi surveys, suggesting the need for in-depth qualitative work to explore participants’ interpretations and understanding during core set development. Indeed, it may have been a valuable addition to this work to have interviewed participants following completion of the surveys, however it was outside the scope of the study. Such work may focus on improving participants’ understanding of the concepts of ‘prioritisation’ and ‘consensus’ and how instructions may optimally distinguish between the two while simultaneously promoting both.

Previous qualitative work has explored participants’ understanding of the purpose of a COS and the Delphi process [23,24]. Through online feedback surveys [24] and in-person interviews [23], it was evident that participants’ understanding was variable [23,24] and was affected by previous experience with COS development and/or the Delphi survey process [23]. It was concluded that participants would benefit from repeated guidance on the principles for COS development during voting [24] and that further guidance and support needed to be accessible and salient [23]. Whilst the COMET Initiative website provides resources for lay audiences to explain the concepts of COSs and Delphi consensus process [25], these lack the detail required to differentiate between prioritisation and consensus. Understanding the relationship between these two concepts is important in finalising a minimum core set in a timely manner. Biggane’s recommendation [23] of considering the most appropriate medium(s) to communicate the core set study is pertinent. Whilst written instructions have historically been the norm and are undoubtable the simplest and quickest form of communication, they are perhaps not the most engaging and may even be ambiguous to readers. Use of more visual resources [23] such as demonstration videos [26] may help in engaging and educating participants. A novel ‘live’ approach in COS development was recently published [27]; researchers ran a Delphi ‘hackathon’ where all participants simultaneously completed the online Delphi surveys [27]. This live event involved an introductory session explaining the methodology and purpose of the study and provided breakout rooms for any questions regarding any aspect of the process [27]. Subsequent rounds emphasised that the goal of the Delphi survey was to reach consensus and participants were able to ask questions via a chat function [27]. The ability for participants to access timely guidance and support is undoubtedly valuable, and something which is not possible with current Delphi survey methods. It is vitally important that those taking part in a Delphi survey understand the significance of the role of feedback since utilising feedback to reconsider opinions and responses in order to gain consensus is fundamental to this process. If participants are not using the provided feedback effectively, then the process is not fit for purpose. Further qualitative work to explore how to clarify and communicate the concepts of prioritisation and consensus to all stakeholder groups in Delphi instructions, whether written, verbal or visual, is needed.

In addition to improving communication, ways by which to expedite prioritisation could be explored. The James Lind Alliance (JLA) specialises in prioritising research questions to be answered in specific healthcare fields [28,29]. The methodology involves asking relevant stakeholders what are the most important questions to research, collating this information and then asking stakeholders to rank or choose their top 10 research questions to be taken to a workshop for further discussion³⁰ – a process very similar to core set development using Delphi techniques. As with all methodologies, there are advantages and disadvantages (as described in their guidebook [30]), but ultimately the outcome is a prioritised set of data items.

A strength of this study was that participants were not aware of their allocation to receive different instructions in the second survey round and responses were therefore free from performance bias. This study used a randomised design, in which participants who were assigned an identity number in the first survey round, were randomised by the REDCap software [11,12], to receive standard or enhanced instructions in the second survey round. Whilst randomisation ensures balanced groups, a potential limitation of using REDCap software to perform randomisation in this study was that all identity numbers were assigned to standard or enhanced feedback, regardless of whether participants answered questions or opened the survey numerous times. By including participants with multiple identity numbers and non-responders, the randomisation of participants to the two groups was not equal. This could have resulted in selection bias, however, in this study, baseline characteristics were similar between both groups and allocation did not affect response rates. Though desirable, not all nested COS methodology studies are randomised [21,31]. In addition, because this nested study was opportunistic in nature, it was not powered to be able to detect meaningful differences between the randomised groups and as such, results should be interpreted with caution and need further validation. A larger sample size would have increased the confidence of detecting a true effect of the different instruction types between the two randomisation groups and would have allowed more in-depth statistical analyses of the effects on individual stakeholder groups. When working within study limitations, efforts to engage and retain participants should be prioritised to increase the sample size and improve the strength and validity of the study. Response rates in round 2 were below half, lower than our previous core sets [13,17,32,33]. A participant was considered to have ‘responded’ if at least one core data item question had been answered. In round 1, this would likely have resulted in the inclusion of individuals curious about the survey content but then choosing not to proceed beyond the first few questions or first section (almost a quarter of participants did not continue to section 2). Ultimately, this would have led to inflated response rates in round 1 and exaggerated attrition rates in round 2. Comprehensive datasets strengthen the validity of study findings since missing data can introduce bias depending on the reason for omission. Omission, and therefore attrition, can be attributed to a number of factors including the length of the survey and the time elapsed between the first and final round [4], but in this instance could also be attributed to the inclusion of non-invested individuals. Ultimately, there is no guidance as to the number of completed questions required for inclusion as a ‘respondent’ or the length of the survey or the duration of the core set development process. But further work to explore completion inclusion thresholds and methods to promote completion and thereby reduce attrition would all be beneficial in promoting a comprehensive dataset. Finally, a key reflection in the writing up of this study is the recognition of the role that qualitative work would have played in progressing our understanding of how feedback was interpreted and applied.

Conclusion

Using a nested study approach within the development of a registry core data set to explore core set methodology, this study suggests that enhanced instructions emphasising the top items prioritised do not promote the prioritisation of items in a Delphi survey during core set development. Due to the complexities of the core set development process, it is likely that there are multiple factors at play. The absence of prioritisation in both groups perhaps suggests a lack of, or limited, understanding of the pathway to gaining consensus. Further work to explore how participants interpret and respond to feedback and instructions to simultaneously prioritise items and reach consensus with others is warranted. Educating and communicating these crucial elements will improve the efficiency and the value of the core set.

Supporting information