Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

How are trial outcomes prioritised by stakeholders from different regions? Analysis of an international Delphi survey to develop a core outcome set in gastric cancer surgery

  • Bilal Alkhaffaf ,

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Visualization, Writing – original draft, Writing – review & editing

    Affiliations Department of Oesophago-Gastric Surgery, Salford Royal Hospital, Salford Royal NHS Foundation Trust, Manchester, United Kingdom, Division of Cancer Sciences, School of Medical Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, United Kingdom

  • Aleksandra Metryka,

    Roles Data curation, Formal analysis, Project administration, Writing – review & editing

    Affiliation Paediatric ENT Department, Royal Manchester Children’s Hospital, Manchester University NHS Foundation Trust, Manchester, United Kingdom

  • Jane M. Blazeby,

    Roles Conceptualization, Funding acquisition, Methodology, Supervision, Writing – review & editing

    Affiliation Centre for Surgical Research and Bristol NIHR Biomedical Research Centre, University of Bristol, Bristol, United Kingdom

  • Anne-Marie Glenny,

    Roles Funding acquisition, Supervision, Writing – review & editing

    Affiliation Division of Dentistry, School of Medical Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, United Kingdom

  • Paula R. Williamson ,

    Roles Funding acquisition, Methodology, Supervision, Writing – review & editing

    ‡ PRW and IAB are joint senior authors on this work.

    Affiliation MRC North West Hub for Trials Methodology Research, University of Liverpool and a Member of Liverpool Health Partners, Liverpool, United Kingdom

  • Iain A. Bruce ,

    Roles Funding acquisition, Supervision, Writing – original draft, Writing – review & editing

    ‡ PRW and IAB are joint senior authors on this work.

    Affiliations Paediatric ENT Department, Royal Manchester Children’s Hospital, Manchester University NHS Foundation Trust, Manchester, United Kingdom, Division of Infection, Immunity and Respiratory Medicine, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, United Kingdom

  • on behalf of the GASTROS International Working Group

    Membership of the GASTROS working group is listed in the Acknowledgments.



International stakeholder participation is important in the development of core outcome sets (COS). Stakeholders from varying regions may value health outcomes differently. Here, we explore how region, health income and participant characteristics influence prioritisation of outcomes during development of a COS for gastric cancer surgery trials (the GASTROS study).


952 participants from 55 countries participating in a Delphi survey during COS development were eligible for inclusion. Recruits were grouped according to region (East or West), country income classification (high and low-to-middle income) and other characteristics (e.g. patients; age, sex, time since surgery, mode of treatment, surgical approach and healthcare professionals; clinical experience). Groups were compared with respect to how they categorised 56 outcomes identified as potentially important to include in the final COS (‘consensus in’, ‘consensus out’, ‘no consensus’). Outcomes categorised as ‘consensus in’ or ‘consensus out’ by all 3 stakeholder groups would be automatically included in or excluded from the COS respectively.


In total, 13 outcomes were categorised ‘consensus in’ (disease-free survival, disease-specific survival, surgery-related death, recurrence of cancer, completeness of tumour removal, overall quality of life, nutritional effects, all-cause complications, intraoperative complications, anaesthetic complications, anastomotic complications, multiple organ failure, and bleeding), 13 ‘consensus out’ and 31 ‘no consensus’. There was little variation in prioritisation of outcomes by stakeholders from Eastern or Western countries and high or low-to-middle income countries. There was little variation in outcome prioritisation within either health professional or patient groups.


Our study suggests that there is little variation in opinion within stakeholder groups when participant region and other characteristics are considered. This finding may help COS developers when designing their Delphi surveys and recruitment strategies. Further work across other clinical fields is needed before broad recommendations can be made.

1. Introduction

A core outcome set (COS) is an agreed minimum group of critically important outcomes which should be reported by all trials within a research field [1]. The GASTROS study ( aims to develop a COS in the field of gastric cancer surgery to promote uniform reporting of important outcomes and facilitate evidence synthesis [2]. This is necessary as there is significant variation and heterogeneity in this field with respect to reporting and measurement of outcomes [3]. Furthermore, the outcomes chosen by researchers to report in surgical trials for gastric cancer often do not reflect the priorities held by patients [4]. For this reason, the GASTROS study has sought consensus between patients and healthcare professionals with respect to outcome selection.

Delphi surveys and consensus meetings are commonly used methodologies in the development of COS [1, 5]. Delphi surveys ask participants deemed by the study group to hold an important perspective (key stakeholders) to prioritise outcomes and achieve consensus. The completed Delphi survey often informs and influences discussions during a subsequent consensus meeting, with the aim of resolving uncertainties regarding prioritisation and ratifying the final composition of the COS. Clear recruitment strategies for Delphi surveys are an important consideration. If recruitment does not result in representative stakeholder groups, there is a risk that the results of the Delphi may not be valid [6]. This is particularly important in international COS where significant regional and cultural differences may influence the results ahead of a consensus meeting and, ultimately, the final COS.

Ensuring stakeholder groups are representative can be a challenging task. There is a need to consider many factors including the incidence of the disease, treatment protocols, international variation in healthcare systems and values and socio-economic issues. In the case of curative surgery for gastric cancer it is known that practice varies worldwide (e.g. how surgery is carried out and the extent of resection) and typically surgeons value different outcomes to patients [4]. For example, due to screening programmes, cancers are generally earlier in the Far East where patients tend to be younger with fewer co-morbidities. There is therefore a need to explore these issues to understand how key stakeholders are selected for survey participation. In the GASTROS study 952 participants were recruited to a Delphi survey (268 were patients, 445 surgeons and 239 nurses) from 55 countries. It was therefore possible to explore how stakeholder characteristics influenced outcome prioritisation.

This study had two main objectives:

  1. To describe the characteristics of Delphi participants and explore their possible influence on the prioritisation of outcomes within stakeholder groups.
  2. To explore how stakeholders from different regions prioritised outcomes.

2. Methods

This was an analysis of registration data supplied by Delphi survey (S1 File) participants as part of the GASTROS study. Both rounds of the survey took place between March and October 2019. Details of the scope, objectives and methodology of the study have been previously described [24]. In summary, participants were asked to score outcomes in terms of importance. The results of the Delphi survey informed discussions in a consensus meeting where final recommendations were made regarding which outcomes to include in the COS.

2.1. Stakeholder selection and baseline information

The GASTROS study sought to involve key stakeholders–patients, surgeons, and oncology nurses—to identify a COS for surgical trials in gastric cancer. Our guiding principle has been to promote the ‘patient voice’ as they are the beneficiaries of trials in this field and have all-important ‘lived experience’. The patient voice has previously been shown to be under-represented in COS development [7]. Surgeons provide a clinical perspective and the experience of treating large volumes of patients. Oncology nurses were invited to participate given their central roles as care-givers, patient advocates and core members of the clinical team.

Recruitment was achieved by promoting the study at surgical and nursing congresses, social media and through patient groups and charities. The study website ( allowed stakeholders to register their interest ahead of the Delphi survey. Local recruitment of patient healthcare professionals by members of the international working group was also undertaken. Participation in the Delphi survey was open to all interested stakeholders who fulfilled the following criteria:

  • Surgeons who had completed their training and routinely treat gastric cancer.
  • Oncology nurses with a recognised proportion of their role involved in the care and follow-up of gastric cancer patients.
  • Patients who have undergone surgical resection for gastric cancer with the intention of cure.

There is no sample size requirement for Delphi surveys. To be able to demonstrate the enrolment of a broad and representative range of stakeholders, participants were asked to provide the information listed below:


  • Age
  • Sex
  • Surgical approach (laparoscopic or open)
  • Type of gastrectomy (total or partial)
  • Modality of treatment (surgery alone or a combination of surgery and chemotherapy or radiotherapy)
  • Time since surgery


  • Experience (number of gastrectomies undertaken)


  • Experience (years of service)

These datapoints were developed based on information that was likely to be readily known to participants and the expert opinion of the GASTROS study management group (SMG) with respect to important factors that may influence outcomes or perspectives. In the context of patients, different health outcomes, such as complications and survival, may impact their lived experience and ultimately how outcomes are prioritised. Similarly, as clinical experience changes with time, there may be a greater exposure to and therefore appreciation of the impact or importance of longer-term consequences of surgery.

Additionally, all participants were asked to provide their country of residence so that regional differences could be considered. Participants were categorised into ‘Eastern’ or ‘Western’ countries (Fig 1) and ‘high-income’ or ‘low- to medium-income countries’ as defined by the Organisation for Economic Co-operation and Development’s Development Assistance Committee [8]. Eastern countries were defined as those within East Asia, South East Asia, and Eastern Russia, and included China, Japan, South Korea, Thailand, Vietnam, Malaysia, and Singapore [9]. Western countries were defined as those from Western Europe, North America, Australia, and New Zealand [10]. Contrasting between the ‘East’ and ‘West’ is of particular importance to gastric cancer given the differences in incidence, pathology, treatment and outcome. It was hypothesised that these differences in approach and survival may influence how stakeholders in these regions prioritise different health outcomes which could be examined further in this study [11, 12]. Similarly, health priorities may be influenced by resource availability as categorised by country income.

Fig 1. Countries from which participants were recruited.

Eastern countries were defined as those within East Asia, Southeast Asia, and Eastern Russia, and included China, Japan, South Korea, Thailand, Vietnam, Malaysia, and Singapore [9]. Western countries were defined as those from Western Europe, North America, Australia, and New Zealand [10].

2.2 Scoring of outcomes in the Delphi survey and categorisation of outcomes

A list of 56 outcomes identified from previous trials and patient interviews [3, 4] were presented to survey participants who were asked to rate each outcome on a scale of importance (1–3: not important, 4–6: important, 7–9: critically important). Outcomes were organised according to five core areas (mortality/survival; clinical/physiological outcomes; life impact; resource use; adverse events) based on a taxonomy developed for COS development [13]. Patients, surgeons, and nurses group ratings were considered separately to ensure that each group had an equal voice. Participants had the opportunity to suggest further outcomes that they believed had not been presented in round 1. One additional new outcome suggested by participants in round 1 was identified and after consideration by the SMG was presented to participants for scoring in round 2. Therefore, a total of 57 outcomes were presented in round 2 where, for each outcome, participants were shown the scores from each stakeholder group, and given the opportunity to change their rating if they wished.

After two rounds of rating, outcomes were categorised as follows:

  • To be included in the COS (‘consensus in’)
  • To be excluded from the COS (‘consensus out’)
  • ‘No consensus’ reached i.e. no decision reached as to whether the outcome should be included in of excluded from the COS.

Criteria for categorising outcomes was set a priori by the SMG and based on established COS methodology [1]. If an outcome was rated 7–9 (critically important) by 70% or more of a stakeholder group and 1–3 (not important) by no more than 15% of the group, then the consensus amongst that group was that the outcome should be included in the COS. If an outcome was rated 7–9 (critically important) by less than 50% of the group, the consensus amongst that group was for the outcome to be excluded from the COS. Unanimous agreement amongst all three stakeholder groups was required for inclusion in, or exclusion from, the COS. Any other combination resulted in the outcome being placed in the ‘no consensus’ category and was discussed at a pre-planned consensus meeting to finalise the COS.

2.3 Data analysis and interpretation

In round 1, participants completing 50% or more of the Delphi survey were included in the round 1 analysis and invited to participate in round 2. Likewise, participants completing 50% or more of the survey in round 2 were included in the round 2 analysis. For the purpose of this present analysis, participants were placed into ‘sub-groups’ according to the registration data they submitted (e.g. patient treatment type, surgeon experience etc) to examine the differences in outcome scoring. The following analyses were performed after 2 rounds of ratings:

  1. The proportion of participants scoring each outcome as ‘critically important’ (score 7–9). This analysis approach was chosen as these figures were presented in the consensus meeting discussing results from the Delphi survey.
  2. The consensus opinion of each sub-group with respect to whether the outcome should be ‘included’ in the COS, ‘excluded’ from the COS or whether ‘no consensus’ could be reached. These categorisations were compared against the overall ‘in’, ‘out’ and ‘no consensus’ categorisations by each stakeholder group (patients, surgeons and nurses) which was presented to the consensus meeting participants.

Participants not providing demographic data during registration were excluded from the sub-group analyses. When exploring differences in prioritisation, particular focus was placed on outcomes that were categorised as ‘consensus in’ by one sub-group and ‘consensus out’ by another.

To examine the possible influence of attrition bias between rounds, the characteristics of stakeholders participating in both rounds were compared to those who only completed round 1. A descriptive analysis was undertaken, and the Chi squared test applied (using SPSS—IBM Corp. Released 2019. IBM SPSS Statistics for Windows, Version 26.0. Armonk, NY: IBM Corp) to examine for statistically significant differences at the 0.05 level.

2.4 Ethical approval

The study was given ethical approval by the North West—Greater Manchester East Research Ethics Committee (18/NW/0347) and governance approvals by Manchester University Hospitals NHS Foundation Trust. All participants were provided with a participant information booklet. Informed written consent was obtained during the online registration process for participants in the Delphi survey.

3. Results

3.1 Overview

The characteristics of participants included in the analysis and attrition rates are summarised in Table 1. After 2 rounds of voting, agreement was reached amongst all three stakeholder groups to include 13 outcomes (disease-free survival, disease-specific survival, surgery-related death, recurrence of cancer, completeness of tumour removal, overall quality of life, nutritional effects, all-cause complications, intraoperative complications, anaesthetic complications, anastomotic complications, multiple organ failure, and bleeding) into the COS. A further 13 outcomes were excluded from the COS (endocrine complications, fatigue, surgical stress response, post-operative psychosis, insomnia, impact on sexual function, ability to eat socially, ability to interact socially, impact on physical appearance, impact on spirituality or faith, wound size, cost and destination on discharge), leaving 31 ‘no consensus’ outcomes for discussion at the consensus meeting.

Table 1. Demographic characteristics of participants included in analysis of round 1 and 2 scores.

3.2 Prioritisation of outcomes within stakeholder groups (subgroup analysis)

Baseline characteristics reported by stakeholders during the registration process were examined to understand whether these influenced how outcomes were prioritised.

3.2.1 Patient prioritisation of outcomes.

A summary of outcomes categorised for ‘inclusion’ into the COS by at least one patient sub-group is presented in Table 2. Thirty outcomes were categorised for inclusion in the COS by at least one subgroup. Four outcomes were simultaneously categorised both for ‘inclusion’ and ‘exclusion’ by different subgroups. None of the outcomes categorised for inclusion by all stakeholder groups were voted ‘consensus out’ by any patient sub-group. Seven outcomes were categorised for inclusion in the COS by all patient subgroups.

Table 2. Outcomes categorised for inclusion in the COS by at least one subgroup of patients.

3.2.2 Surgeon prioritisation of outcomes.

Table 3 summarises and compares outcomes categorised for inclusion into the COS by at least one surgeon sub-group. Twenty-one outcomes were categorised for inclusion by at least one subgroup. No outcomes were simultaneously categorised both for ‘inclusion’ and ‘exclusion’ by different subgroups. Twelve outcomes were categorised by all surgeon subgroups for inclusion.

Table 3. Outcomes categorised for inclusion in the COS by at least one subgroup of surgeons.

3.2.3 Nurse prioritisation of outcomes.

Table 4 summarises and compares the outcomes categorised for inclusion by at least one nurse sub-group. Twenty-two outcomes were categorised for inclusion by at least one subgroup. Five outcomes were simultaneously categorised both for ‘inclusion’ and ‘exclusion’ by different subgroups. None of the outcomes categorised for automatic inclusion by all stakeholder groups were voted ‘consensus out’ by any nurse sub-group. Ten outcomes were categorised by all nurses’ subgroups for inclusion.

Table 4. Outcomes categorised for inclusion in the COS by at least one subgroup of nurses.

3.3 Impact of regional variation on prioritisation of outcomes

Table 5 details the final categorisation of outcomes in the Delphi survey as agreed by all stakeholder groups. This is compared to outcome categorisation lists based on the region (East versus West) and country income differences (HIC versus LMIC). Consensus agreement to include 8 and exclude 7 outcomes was reached across all regional sub-groups. No outcomes were simultaneously categorised as ‘consensus in’ and ‘consensus out’ across different regional sub-groups.

Table 5. Regional differences in consensus on outcomes voted for inclusion or exclusion from the COS by at least 1 subgroup.

4. Discussion

The GASTROS study ( is the first to bring together healthcare professionals and patients with the purpose of identifying outcomes to include in a COS for surgical trials in gastric cancer. The multi-language survey recruited a broad spectrum of stakeholders with different personal and professional experiences from over 50 countries across 6 continents. We aimed to examine whether certain stakeholder characteristics influenced how outcomes were prioritised and whether there were regional influences also. Our analysis from nearly 1000 survey participants suggested that little variation within the stakeholder groups exists. Similarly, when all stakeholders were categorised according to region or country income, significant differences were not identified. These are important findings which should serve to reassure researchers and patients that the resulting COS has sought and considered international opinion which is widely representative. Furthermore, these findings suggest that priorities within stakeholder groups and across regions are more aligned than may have been previously thought.

4.1 Planning recruitment to Delphi surveys

Few studies have previously examined factors which influence how stakeholders prioritise outcomes in the field of COS development. The BRAVO study explored this in the field of breast cancer reconstruction and found that priorities varied within patient and healthcare professional groups [6]. This led them to recommend careful participant selection for Delphi surveys by COS developers. These same differences, however, were not identified in our study. The BRAVO study’s healthcare professional stakeholder group was more heterogenous than the groups in this study (breast surgeons, plastic surgeons, nurses and psychologists grouped together) and so these differences may be expected. Furthermore, reconstructive breast surgery is a complex area which covers many different types of procedures. This may also account for the significant variation in outcome prioritisation by patients which was not mirrored in the GASTROS study. In comparison, gastric cancer surgery generally comprises of either a partial or total gastrectomy both of which can result in similar short and long-term problems which may explain why priorities were more aligned within stakeholder groups. Similarly, a COS study in the field of bariatric surgery identified significant variation in outcome prioritisation amongst healthcare professionals [14]. Again, healthcare professionals in this study were heterogenous, which supports our strategy to separate surgeons and nurses into different stakeholder groups.

Achieving the ‘correct balance’ of representative stakeholders is an important consideration during the design phase. For example, knowledge of the patient demographic and which types of interventions are prevalent within that group, will enable researchers to recruit an appropriate number of stakeholders with those characteristics. As this is the first study to specifically examine regional variations amongst stakeholders in COS development, it is unknown whether these findings necessary mirror those from other COS studies. Further examination of previously undertaken Delphi surveys is required, and future surveys will need to ensure that similar baseline characteristics are recorded. This is relatively straightforward information to capture and can be supplied quickly and easily by survey participants during the registration process.

With respect to the GASTROS study, the importance of seeking international agreement on core outcomes was identified at the conception stage and subsequently influenced the design of the prioritisation exercise. Our strategy for addressing the significant challenges associated with international involvement included 1) an international working group with regional collaborators, 2) translating surveys and 3) seeking the support of relevant patient and professional groups. Transparent reporting of methodological approaches adopted during COS development are of paramount importance. Ultimately, a COS will only achieve its stated goals if researchers use it. And whilst there are likely several factors which influence the uptake of COS, ensuring researchers have the confidence that the COS is relevant to them and has been developed through a methodologically robust process are likely to be important factors which influence uptake and dissemination [15].

There are challenges in deciding how to sample participants for a Delphi study. Epidemiological studies, registries and audits provide descriptive regional or national information [1618]. However, in the case of gastric cancer, these resources are not always complete or available. Consequently, the study team widened the promotion and enrolment into the Delphi to capture as many patients as possible. In our study, we demonstrated that there was not significant variation in outcome prioritisation within stakeholder sub-groups with respect to the characteristics that we examined. Consequently, whilst over 1000 participants were enrolled, it may not have been necessary to recruit such large numbers. This will likely guide our recruitment strategy during future planned stages of work when reviewing the COS and identifying outcome measurement instruments. Our experience may also help guide other COS developers as they consider the number of participants to recruit to their Delphi surveys. However, given some of our findings differed from those in the field of breast surgery reconstruction and bariatric surgery, more work is needed before broad recommendations can be made.

4.2 Variations within stakeholder groups

When regional variations across the three stakeholder groups were compared, the greatest differences in prioritisation were observed amongst nurses. For example, in four outcomes (pain, ability to undertake physical exercise, impact on mental health, need for additional intervention) different subgroups of nurses categorised them as ‘consensus in’ and ‘consensus out’. These outcomes seemed less important in LMIC and HIC settings within the nurse group. Understanding the reason for this is likely to be complex. It may be argued that this is simply because nurses are reflecting the importance that patients from these cultures or regions place on these outcomes as similar trends were seen amongst patients. Limited resource in LMIC settings which may affect follow-up may also play a role in understanding how important longer-term problems are in these regions. Further exploration using qualitative research methods may help understand these differences further.

In examining the differences between patient sub-groups, one would expect to see some differences given the number of characteristics that were examined. Despite this, only two outcomes (urinary complications and conversion to open surgery) were simultaneously categorised as ‘consensus in’ and ‘consensus out’ by different sub-groups. This finding suggests that despite the many possible influences on patient experience following gastric cancer surgery, there is not a significant variation in how health related outcomes are prioritised in this group. Surgeons had the greatest concordance with respect to outcome prioritisation. Overall, the observed differences in outcome prioritisation were small within each stakeholder group reassuring researchers using the COS that it is based on the views of a representative cohort of patients and healthcare professionals.

4.3 Impact of regional variations on outcomes automatically included in COS

The aim of a COS is to identify outcomes which are critically important across all stakeholder groups participating in the process. In the case of the GASTROS study, an outcome would only be automatically included in the COS if patients, surgeons, and nurses each categorise it ‘consensus in’. Ultimately, it is not possible to confidently assess how regional differences may have affected the final categorisation of outcomes which informed the consensus meeting. Participants in round 2 were shown the scores of all stakeholder groups from round 1 before being asked to change their score if they wish. To assess regional differences, Western participants, for example, in round 2 would have needed to see only Western stakeholder group scores from round 1. Furthermore, there are a number of other confounding factors which influence why participants change scores between rounds (see below) further making an analysis of regional impacts challenging.

Despite this, some assessments could be made. No outcomes categorised for automatic inclusion by all three stakeholder groups were categorised for automatic exclusion by a regional sub-group. And no outcomes categorised for automatic exclusion from the COS by all three stakeholder groups were categorised for automatic inclusion by a regional sub-group. This suggests that the regional differences in approach to management or patient outcome may not significantly influence how stakeholders prioritise outcomes

There were two outcomes (gastrointestinal functional problems and ability to complete treatment pathway) which were categorised for automatic inclusion by stakeholders from the West and HIC that were not included in the final list presented to the consensus meeting. Furthermore, some outcomes (surgery-related death, nutritional outcomes, bleeding, overall quality of life, anaesthetic complications) did not reach consensus for automatic inclusion by regional sub-groups yet were automatically included when the overall views of stakeholders were considered. This may bring some to the conclusion that different COS should be developed for different regions as some researchers may be collecting outcomes that were not deemed critically important in their region. However, researchers should be cognisant of the fact that their trials are internationally relevant and vitally important to the larger picture where evidence synthesis is concerned. From a different perspective, some researchers may feel aggrieved if outcomes which are critically important in their region are not eventually included in the COS. It is important to emphasise that COS are minimum reporting guidelines and that researchers are encouraged to report additional outcomes that they believe are important.

4.4 Dissemination of results

From the study’s inception, the management team understood the importance of regularly reporting findings to encourage participation from all stakeholder groups. Regular reporting also aimed to increase the study’s exposure and highlight its importance to minimising research waste in future trial design. Finally, uptake of the COS requires researchers and funders to have knowledge of its existence. A clear dissemination policy was set out a priori and included scientific publications, presentations at medical and nursing congresses as well as lay summaries delivered to patient participants through patient groups and social media. The success of this policy to this point has been reflected by nearly 1000 participants being recruited to the Delphi survey. Continued efforts will be required to ensure that the COS is used, and researchers and patients benefit from it.

4.5 Strengths and limitations

Strengths of this study include that it is novel and that was able to recruit a large number of participants from many countries. However, there are some limitations which should be acknowledged. This was an analysis which was not powered to make definitive conclusions about relationships between sub-groups and how outcomes were rated. Therefore, the results should be viewed in this context. Furthermore, the sub-groups examined in this paper were chosen by members of the study team based on their extensive experience in the field of gastric cancer and their understanding of factors which may impact on stakeholder experience, perceptions and subsequently how outcomes may be prioritised. It is possible that other unexplored characteristics impact on how stakeholders prioritise outcomes. In addition, this study did not explore how different characteristics interact with one another to impact on outcome prioritisation (e.g. years since surgery and type of gastrectomy). Doing so would create results which would remove the focus from regional differences and would be difficult to interpret. Furthermore, there were significantly fewer patients from Eastern countries enrolled compared to their Western counterparts. This may have influenced how outcomes were categorised ahead of the consensus meeting. However, due to the interplay of other factors described above, reaching a definite conclusion about the degree of this possible limitation is difficult. This is an area that may benefit from further exploration using qualitative research methods.

Delphi surveys are an established method of reaching consensus in the design of COS [1]. They give participants the opportunity to reflect on their ratings from previous rounds before giving a final score. Only after this opportunity should all scores be analysed, and outcomes categorised ahead of the consensus meeting. During the process of rating outcomes in round 2 of the survey, participants are shown the results from each separate stakeholder group in round 1. The topic of why participants change their scores between rounds is an interesting one which has been examined elsewhere [19]. Through our previous analysis we identified that the reasons for changing scores provided by stakeholders were varied, including having the time to reflect on the question being asked, changing their minds on the importance, impact or usefulness of the outcome in question, and changes in personal experience of the outcome. In fact, the influence of other stakeholder ratings as a reason for significantly changing a score in round 2 was cited by only a minority of healthcare professionals and patients. Another factor which may influence scores between rounds is attrition. Our strategy to keep this as low as possible, alongside other approaches to facilitate international participation in Delphi surveys for COS is a topic which will be described separately. Whilst overall attrition was 30%, the group this affected the most were nurses with nearly 45% attrition. However, the characteristics of those completing both rounds were not significantly different to those only completing round 1. Likewise, a statistically significant difference was identified in the characteristics of surgeons completing both rounds who were predominantly Western and from HIC compared to the balance of surgeons completing round 1. It could be argued therefore that retaining a greater number of Eastern and LMIC surgeons may have led to slightly different survey results. However, whilst statistically significant, this difference is unlikely to be clinically significant given that the number of surgeons not participating in round 2 was relatively small.

One may consider that, given the multimodal nature of treatment for gastric cancer, a COS would be more relevant if it incorporated all therapies (e.g. chemotherapy, radiotherapy, and endotherapy) and that the views of oncologists should also be sought. However, at the time that GASTROS was conceived, there were 24 ongoing surgical trials planning to recruit 11 000 patients for whom non-surgical-related outcomes would not be applicable or relevant. Other considerations such as the resource and time required to recruit participants from other stakeholder groups were also important, hence why a pragmatic decision was made to limit stakeholder groups to those chosen in this study.

Our methodology for COS development was based on an established approach as described by the COMET handbook [1] and COS developers. This aims to seek consensus based on identifying a long list of potentially important outcomes from a systematic review and patient interviews, followed by a Delphi survey, concluded by a consensus meeting. Whilst this approach is well-studied and has been adopted by many, our experience indicates that the process, particularly when seeking broad international opinion, can be both time and resource intensive. Some groups have already begun to explore whether COS development can be streamlined [20]. It should therefore be acknowledged that COS methodology is a developing field for which a single ‘optimal’ approach does not necessarily exist. Examination of differing methodological adaptations should therefore form an important part of future COS development studies and appropriate funding should be made available to support this.

4.6 Conclusion

The GASTROS Delphi survey recruited a broad spectrum of international stakeholders of patients, nurses and surgeons to produce a list of outcomes which should be included or excluded from a COS and others which required further discussion at a consensus meeting. Consensus across these groups was achieved to include 13 outcomes into the COS which will be discussed further at a final consensus meeting (disease-free survival, disease-specific survival, surgery-related death, recurrence of cancer, completeness of tumour removal, overall quality of life, nutritional effects, all-cause complications, intraoperative complications, anaesthetic complications, anastomotic complications, multiple organ failure, and bleeding). Whilst some regional differences were highlighted, there was little variation within stakeholder groups and between regions with respect to how outcomes were prioritised. This may reassure COS users that the adopted methodology was robust and that the views captured during its development were representative. COS developers should carefully consider the characteristics of Delphi survey participants when planning their recruitment strategy. These variables should be explored further to examine the generalisability of this study’s findings.

Supporting information

S1 File. Full version of the Delphi survey which was translated into target language.



The authors would like to highlight the role undertaken by Dr Aleksandra Metryka, Senior Clinical Trials Coordinator, who facilitated the running of this study. In addition, the authors would like to thank the following associations and groups for their support in facilitating recruitment to the GASTROS study Delphi survey:

GASTROS International Working Group Collaborators (To be cited as collaborators in PUBMED)

  • Ademola Adeyeye
  • Paulo Matos Costa
  • Ismael Diez del Val
  • Suzanne Gisbertz
  • Ali Guner
  • Simon Law
  • Hyuk-Joon Lee
  • Ziyu Li
  • Koji Nakada
  • Daniel Reim
  • Gian Luca Baiochhi
  • William Allum
  • Asif Chaudhry
  • Ewen Griffiths
  • Shuangxi Li
  • Yu-long He
  • Zekuan Xu
  • Yingwei Xue
  • Han Liang
  • Guoxin Li
  • Enhao Zhao
  • Philipp Neumann
  • Linda O’Neill
  • Emer Guinan
  • Daniela Zanotti
  • Giovanni de Manzoni
  • Eliza R.C. Hagens
  • Mark I. van Berge Henegouwen
  • Patrícia Lages
  • Susana Onofre
  • Rafael Mauricio Restrepo Nuñez
  • Gabriel Salcedo Cabañas
  • Maria Posada Gonzalez
  • Cristina Marin Campos
  • Bahar Candas
  • Bahadır Emre Baki
  • Muhammed Selim Bodur
  • Reyyan Yildirim
  • Arif Burak Cekic


  1. 1. Williamson P. R. et al., “The COMET Handbook: version 1.0,” Trials, vol. 18, no. S3, p. 280, Jun. 2017. pmid:28681707
  2. 2. Alkhaffaf B., Glenny A.-M., Blazeby J. M., Williamson P., and Bruce I. A., “Standardising the Reporting of Outcomes in Gastric Cancer Surgery Trials: Protocol for the Development of a Core Outcome Set and Accompanying Outcome Measurement Instrument Set (The GASTROS Study).,” Trials, vol. 18, no. 1, p. 370, Dec. 2017. pmid:28793921
  3. 3. Alkhaffaf B., Blazeby J. M., Williamson P. R., Bruce I. A., and Glenny A.-M., “Reporting of outcomes in gastric cancer surgery trials: a systematic review,” BMJ Open, vol. 8, no. 10, p. e021796, Oct. 2018. pmid:30337308
  4. 4. Alkhaffaf B., Blazeby J. M., Bruce I. A., and Morris R. L., “Patient priorities in relation to surgery for gastric cancer: qualitative interviews with gastric cancer surgery patients to inform the development of a core outcome set,” BMJ Open, vol. 10, no. 2, p. e034782, Feb. 2020. pmid:32051319
  5. 5. Williamson P. R. et al., “Developing core outcome sets for clinical trials: issues to consider,” Trials, vol. 13, no. 1, p. 132, Dec. 2012. pmid:22867278
  6. 6. Potter S., Brookes S. T., Holcombe C., Ward J. A., and Blazeby J. M., “Exploring methods the for selection and integration of stakeholder views in the development of core outcome sets: A case study in reconstructive breast surgery,” Trials, vol. 17, no. 1, pp. 1–11, Sep. 2016. pmid:27664072
  7. 7. Gargon E. et al., “Choosing Important Health Outcomes for Comparative Effectiveness Research: A Systematic Review,” PLoS One, vol. 9, no. 6, p. e99111, Jun. 2014. pmid:24932522
  8. 8. “World Bank Country and Lending Groups–World Bank Data Help Desk.” [Online]. Available: [Accessed: 05-May-2020].
  9. 9. “Far East—Wikipedia.” [Online]. Available: [Accessed: 28-Jun-2020].
  10. 10. “Western world—Wikipedia.” [Online]. Available: [Accessed: 28-Jun-2020].
  11. 11. Yamamoto M., Rashid O. M., and Wong J., “Surgical management of gastric cancer: The East vs. West perspective,” Journal of Gastrointestinal Oncology, vol. 6, no. 1. Pioneer Bioscience Publishing, pp. 79–88, 2015. pmid:25642341
  12. 12. Chan W. L. et al., “Gastric Cancer–From Aetiology to Management: Differences Between the East and the West,” Clin. Oncol., vol. 31, no. 8, pp. 570–577, Aug. 2019. pmid:31178345
  13. 13. Dodd S., Clarke M., Becker L., Mavergames C., Fish R., and Williamson P. R., “A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery,” J. Clin. Epidemiol., vol. 96, pp. 84–92, Apr. 2018. pmid:29288712
  14. 14. Coulman K. D. et al., “A Comparison of Health Professionals’ and Patients’ Views of the Importance of Outcomes of Bariatric Surgery,” Obes. Surg., vol. 26, no. 11, pp. 2738–2746, Nov. 2016. pmid:27138600
  15. 15. Kirkham J. J., Bracken M., Hind L., Pennington K., Clarke M., and Williamson P. R., “Industry funding was associated with increased use of core outcome sets,” J. Clin. Epidemiol., vol. 115, pp. 90–97, Nov. 2019. pmid:31326541
  16. 16. Bang Y.-J. et al., “Registry of gastric cancer treatment evaluation (REGATE): I baseline disease characteristics,” Asia. Pac. J. Clin. Oncol., vol. 10, no. 1, pp. 38–52, Mar. 2014. pmid:23937356
  17. 17. Messager M. et al., “Variations among 5 European countries for curative treatment of resectable oesophageal and gastric cancer: A survey from the EURECCA Upper GI Group (EUropean REgistration of Cancer CAre),” Eur. J. Surg. Oncol., vol. 42, no. 1, pp. 116–122, Jan. 2016. pmid:26461256
  18. 18. Michalowski J., Salvador A., and Napper R., “Commissioned by Healthcare Quality Improvement Partnership National Oesophago-Gastric Cancer Audit 2018 An audit of the care received by people with Oesophago-Gastric Cancer in England and Wales 2018 Annual Report,” 2018.
  19. 19. Fish R., MacLennan S., Alkhaffaf B., and Williamson P. R., “‘Vicarious thinking’ was a key driver of score change in Delphi surveys for COS development and is facilitated by feedback of results,” J. Clin. Epidemiol., vol. 128, pp. 118–129, Dec. 2020. pmid:33011214
  20. 20. Remus A., Smith V., and Wuytack F., “Methodology in core outcome set (COS) development: the impact of patient interviews and using a 5-point versus a 9-point Delphi rating scale on core outcome selection in a COS development study,” BMC Med. Res. Methodol. 2021 211, vol. 21, no. 1, pp. 1–15, Jan. 2021. pmid:33397292