Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

The Importance of Integration of Stakeholder Views in Core Outcome Set Development: Otitis Media with Effusion in Children with Cleft Palate

  • Nicola L. Harman ,

    Affiliation MRC North West Hub for Trials Methodology Research, Department of Biostatistics, University of Liverpool, Liverpool, United Kingdom

  • Iain A. Bruce,

    Affiliation Central Manchester University Hospitals NHS Foundation Trust, Royal Manchester Children’s Hospital, Manchester, United Kingdom

  • Jamie J. Kirkham,

    Affiliation MRC North West Hub for Trials Methodology Research, Department of Biostatistics, University of Liverpool, Liverpool, United Kingdom

  • Stephanie Tierney,

    Affiliation Royal College of Nursing Research Institute, Warwick Medical School, University of Warwick, Warwick, United Kingdom

  • Peter Callery,

    Affiliation School of Nursing, Midwifery and Social Work, Jean McFarlane Building, University of Manchester, Manchester, United Kingdom

  • Kevin O'Brien,

    Affiliation The Healing Foundation Cleft and Craniofacial Clinical Research Centre, School of Dentistry, University of Manchester, Manchester, United Kingdom

  • Alex M. D. Bennett ,

    Contributed equally to this work with: Alex M. D. Bennett, Raouf Chorbachi, Per N. Hall, Anne Harding-Bell, Victoria H. Parfect, Nichola Rumsey, Debbie Sell, Ravi Sharma

    Affiliation NHS Lothian, ENT Department, Lauriston Place, Edinburgh, United Kingdom

  • Raouf Chorbachi ,

    Contributed equally to this work with: Alex M. D. Bennett, Raouf Chorbachi, Per N. Hall, Anne Harding-Bell, Victoria H. Parfect, Nichola Rumsey, Debbie Sell, Ravi Sharma

    Affiliation North Thames Cleft Service and the Department of Audiological Medicine/ Audiology/Cochlear implants. Great Ormond Street Hospital for Children, London, United Kingdom

  • Per N. Hall ,

    Contributed equally to this work with: Alex M. D. Bennett, Raouf Chorbachi, Per N. Hall, Anne Harding-Bell, Victoria H. Parfect, Nichola Rumsey, Debbie Sell, Ravi Sharma

    Affiliation Cleft Net East, Cambridge University Hospital NHS Trust, Hills Road, Cambridge, United Kingdom

  • Anne Harding-Bell ,

    Contributed equally to this work with: Alex M. D. Bennett, Raouf Chorbachi, Per N. Hall, Anne Harding-Bell, Victoria H. Parfect, Nichola Rumsey, Debbie Sell, Ravi Sharma

    Affiliation Cleft Net East, Cambridge University Hospital NHS Trust, Hills Road, Cambridge, United Kingdom

  • Victoria H. Parfect ,

    Contributed equally to this work with: Alex M. D. Bennett, Raouf Chorbachi, Per N. Hall, Anne Harding-Bell, Victoria H. Parfect, Nichola Rumsey, Debbie Sell, Ravi Sharma

    Affiliation Cleft Net East, Cambridge University Hospital NHS Trust, Hills Road, Cambridge, United Kingdom

  • Nichola Rumsey ,

    Contributed equally to this work with: Alex M. D. Bennett, Raouf Chorbachi, Per N. Hall, Anne Harding-Bell, Victoria H. Parfect, Nichola Rumsey, Debbie Sell, Ravi Sharma

    Affiliation Centre for Appearance Research, Department of Health & Social Sciences, University of the West of England, Bristol, United Kingdom

  • Debbie Sell ,

    Contributed equally to this work with: Alex M. D. Bennett, Raouf Chorbachi, Per N. Hall, Anne Harding-Bell, Victoria H. Parfect, Nichola Rumsey, Debbie Sell, Ravi Sharma

    Affiliation North Thames Cleft Service, Speech and Language Therapy, Great Ormond Street Hospital for Children, London, United Kingdom

  • Ravi Sharma ,

    Contributed equally to this work with: Alex M. D. Bennett, Raouf Chorbachi, Per N. Hall, Anne Harding-Bell, Victoria H. Parfect, Nichola Rumsey, Debbie Sell, Ravi Sharma

    Affiliation North West, Isle of Man and North Wales Cleft Lip and Palate Network, Alder Hey Children’s NHS Foundation Trust, Liverpool, United Kingdom

  • Paula R. Williamson

    Affiliation MRC North West Hub for Trials Methodology Research, Department of Biostatistics, University of Liverpool, Liverpool, United Kingdom



Approximately 75% of children with cleft palate (CP) have Otitis Media with Effusion (OME) histories. Evidence for the effective management of OME in these children is lacking. The inconsistency in outcome measurement in previous studies has led to a call for the development of a Core Outcome Set (COS). Despite the increase in the number of published COS, involvement of patients in the COS development process, and methods to integrate the views of patients and health professionals, to date have been limited.

Methods and Findings

A list of outcomes measured in previous research was identified through reviewing the literature. Opinion on the importance of each of these outcomes was then sought from key stakeholders: Ear, Nose and Throat (ENT) surgeons, audiologists, cleft surgeons, speech and language therapists, specialist cleft nurses, psychologists, parents and children. The opinion of health professionals was sought in a three round Delphi survey where participants were asked to score each outcome using a bespoke online system. Parents and children were also asked to score outcomes in a survey and provided an in-depth insight into having OME through semi-structured interviews. The results of the Delphi survey, interviews and parent/patient survey were brought together in a final consensus meeting with representation from all stakeholders. A final set of eleven outcomes reached the definition of “consensus in” to form the recommended COS: hearing; chronic otitis media (COM); OME; receptive language skills; speech development; psycho social development; acute otitis media (AOM); cholesteatoma; side effects of treatment; listening skills; otalgia.


We have produced a recommendation about the outcomes that should be measured, as a minimum, in studies of the management of OME in children with CP. The development process included input from key stakeholders and used novel methodology to integrate the opinion of healthcare professionals, parents and children.


Cleft lip and palate has an incidence of around 1 in 700 individuals making it one of the most common congenital malformations worldwide [1]. In children with cleft palate (CP) there is a tendency towards Eustachian tube dysfunction, which can contribute to the development and persistence of negative middle ear pressure and the accumulation of mucoid or serous fluid within the middle ear space (Otitis Media with Effusion (OME), glue ear)[2,3]. The tendency to develop OME is greater, and persists for longer, in children with CP. Consequently, approximately 75% of children with CP will have a history of non-trivial OME [1,4].

The consequences of persistent OME can include increased tendency to develop ear infections (acute otitis media, AOM), long-term middle ear problems (chronic otitis media, COM) and hearing loss, which can have a negative impact on speech and language development, communication, behavior and educational attainment. There are several approaches to the management of OME in children with clefts and they include watchful waiting, the provision of hearing aids and the insertion of ventilation tubes. However, the evidence underpinning these strategies is not clear, particularly for children with CP [5].

The MOMENT study (Management of Otitis Media with Effusion in childreN with cleft palaTe) was a feasibility study designed in response to a commissioned call from the National Institute of Health Research, Health Technology Assessment Programme to answer the question “What is the most appropriate way to manage otitis media with effusion in children with cleft palate?”. There is currently no Core Outcome Set (COS) for clinical trials of the management of OME in children with CP [6]. Therefore, one objective of the study was the development of a COS relevant to the treatment of OME in children with CP.

A Core Outcome Set represent the minimum that should be measured and reported in effectiveness trials in a particular condition [7]. The use of a minimum set of core outcomes aims to increase consistency of reporting in clinical trials. This has been demonstrated for trials in rheumatological conditions with an increase in the consistency of outcome reporting following the publication of a COS [8]. A systematic review directed at the early routine insertion of ventilation tubes for the management of OME in children with CP identified a variety of primary and secondary outcomes together with inconsistency in the method of measurement [9]. The use of an agreed set of core outcomes, measured and reported in all randomized controlled trials (RCTs) of treatments for OME in children with CP, could overcome well documented issues of heterogeneity and outcome reporting bias (ORB) [1012], whilst at the same time increasing the potential for meta-analyses of key outcomes in this area.

Specific objectives of the COS development in the MOMENT study were: to identify outcomes that had been previously reported in studies of the treatment of OME; to prioritise outcomes from the perspective of health professionals; to prioritise outcomes from the perspective of patients who can express their views, and parents; and to integrate the opinions of patients, parents and health professionals into a combined COS.

Limitations of previous methods for COS development

Less than a fifth of previous COS studies have involved public representatives, and the majority of those included only a handful of patients [11]. Despite evidence that patients may hold different views from health professionals, and a recommendation to include patients in the process [9], it is unclear whether different stakeholder group views were transparent to those involved [11]. This current study implemented a novel design for COS development, including a new method proposed by children and young people to elicit opinion from children. We investigated the influence of the method of stakeholder feedback on subsequent opinion. We report the results from this work here.


The study protocol for this work, including search strategy and inclusion criteria for the systematic review, has been previously published [13]. Methods used for the systematic review, health professional Delphi survey, semi-structured interviews and final study consensus meeting are described briefly below. An online survey of parents and children is described more fully. An overview of the COS development process is provided in Fig 1.

Systematic Review and generation of the list of outcomes

A list of outcomes previously reported in studies of the treatment of OME was generated by updating a 2009 systematic review [9] using the same search strategy. The review of papers was completed by two authors independently (NLH and IAB) and a list of all outcomes measured in identified papers generated (detailed information is given in the protocol) [13]. This list of outcomes was further refined to standardize the name given to each of the outcomes. All outcomes and domains were discussed with members of the Study Advisory Group (SAG) prior to being finalised. Due to the varied health professional groups likely to be completing the Delphi an ‘outcome tip’, further describing the outcome, was also written for each outcome and reviewed by the SAG. The SAG comprised of Speech and Language Therapists (n = 2), Cleft Surgeons (n = 1), ENT Surgeons (n = 2), Audiologists (n = 2) and Clinical Psychologists (n = 1). A patient representative was approached during the study and accepted membership of the SAG but then, due to unforeseen personal circumstances, needed to withdraw from membership. prior to attending an SAG meeting. The SAG and also the Study Steering Committee (SSC, comprising a trial methodologist, patient representative, health economist and a paediatric otorhinolaryngologist) were also given the opportunity to add outcomes to the list that they considered important.

Health Professional Delphi Survey

The opinion of health professionals was sought through a three round Delphi survey delivered using a bespoke online system[13]. Health professionals were eligible to participate in the Delphi survey if they were affiliated to a UK Cleft Centre and were a cleft surgeon, ENT surgeon, audiologist, cleft nurse specialist, clinical psychologist or speech and language therapist. Potentially eligible health professionals were identified through contact with clinical leads at each of the 15 UK centres who provided a list of current members of their cleft team and their clinical role (S3 Table).

Prior to completion of round 1 it was agreed that for the feedback of results in round 2 to be presented by stakeholder groups, approximately 10 participants per stakeholder group would be required for the presentation of results to be meaningful. In the second round results were presented by health professional stakeholder group, with an individual seeing only the aggregated results from their particular group together with a reminder of their own round 1 score. In the third and final round the results of all stakeholder groups, including parents and children, presented separately, were shown to each participant together with a reminder of their round 2 score.

In each round of the Delphi survey, health professionals were asked to score a list of outcomes using the Grading of Recommendations, Assessment, Development and Evaluations scale of 1 to 9, with 1 to 3 labelled ‘not important’, 4 to 6 labelled ‘important but not critical’ and 7 to 9 labelled ‘critical’ [14] (S1 Fig). Consensus on outcomes for inclusion in the COS was determined using a pre-defined definition of consensus (S1 Table).

Opinions of patients and parents

The views of parents of children with a CP aged 0–11 years, and children with a CP aged 6–11 years, were explored in semi-structured interviews conducted by one of the authors (ST), who does not have a clinical background and was not known to parents or children prior to the interview. A purposive sample was recruited to provide maximum variation in terms of a child’s age and gender and type of treatment experienced for OME. Participants were recruited from two cleft centres in the UK with contrasting approaches to audiology care, one a centralized service, the other distributed across a hub and spoke” model. Interviews were audio-recorded, transcribed verbatim, and then Framework analysis [14] was used to manage and interpret data. Discussion around outcomes took place throughout interviews. However, there was a specific section of the topic guide used within interviews that focused on capturing data on this issue. Results of interviews relating to experiences of OME are reported elsewhere [15,16].

Semi-structured interviews with parents and children gave in-depth information on outcomes of importance for these groups. Interviews did not include a discussion of the outcomes list generated through systematically reviewing the literature because we wanted participants to express their opinions, in their own words, of what they felt were important results or indicators of successful management of OME. In order to give parents and children the same opportunity to score outcomes, an online survey, similar to that completed by health professionals, was developed. This involved review and re-wording, using a plain language description, of each outcome scored by health care professionals. Each re-worded outcome was tested for readability using the NIACE SMOG calculator [17]. Understanding was explored with the Cleft Lip and Palate Association (CLAPA) children and young person’s council (CYPC) and a local CLAPA ‘Happy Faces’ group. The same outcome wording was used for all participants with the exception of minor changes such as “your/your child’s” to ensure appropriate context.

Parents and children aged 7–16 years were asked to consider the appropriate question, described in Fig 2, and to score each of the outcomes. The labels of the 1–9 scale were modified for parents whilst children, under the recommendation of the CYPC, scored each outcome using a traffic light system where the scores 1–3 were represented by a red box labelled as “not that important”, scores 4–6 as an amber box labelled as “important” and scores 7–9 as a green box labelled “really important” (S1 Fig). Both parents and children were provided with a free text box to add anything else that they considered relevant. The definition of consensus (S1 Table) was applied to the results from both parents and children.

Fig 2. Initial question asked prior to scoring outcomes for parents, adults and children with cleft palate.

The question asked of health professionals is included for comparison.

Participants of the survey for parents and children were independent of those who completed a semi-structured interview and were identified using the CLAPA mailing list and social media pages with a potential reach of 4,710 and 9,564 respectively. An individual email was sent to all those on the CLAPA mailing list together with a reminder in their e-newsletter. A link was posted on the Facebook page which included the researcher’s name and photograph (NLH) and a link to the online survey. There is likely to be substantial overlap with membership of multiple Facebook pages and groups however, it was not possible to assess this.

Final consensus meeting

The consensus meeting brought together all sources of information. The results from the Delphi survey of health professionals and the survey of patients and parents were presented along with the opinions from 43 parents and 37 children who took part in a semi structured interviews. An invitation to attend was sent to: health professionals who had completed all rounds of the online Delphi survey and expressed an interest in attending future meetings; all parents who had completed an online survey and expressed an interest and provided contact details to be informed about future meetings; parents who had taken part in a semi-structured interview whose contact details were still valid; CLAPA members in the North West who subscribed to the CLAPA mailing list. The format of the consensus meeting comprised a short study overview, a summary of results from the semi-structured interviews and a review of each outcome on the scored list in turn, including presentation of how each stakeholder group had scored the outcome, and the number of stakeholder groups who achieved consensus. Discussion of each outcome was followed by anonymous electronic scoring by those at the consensus meeting. A written report of the final meeting was circulated to participants for comment.

Ethics statement

Ethical approval was received from the National Research Ethics Service North West – Greater Manchester East Research Ethics Committee (Reference 11/NW/0586) for the completion of semi-structured interviews and invitation of interviewees to the final consensus meeting. Written consent was sought for participation in semi-structured interviews, with written proxy consent sought from parents/guardians for their child’s participation. Written assent was also sought from children aged 6 years and older. Attendance at the consensus meeting was considered to be implied consent for participation with no written consent provided, this process was approved by the Research Ethics Committee. Advice was sought from the National Research Ethics Service who did not consider that ethical approval was required for an online survey of parental and child opinion. However, full information about initial the study was given in the initial pages of the online survey and survey completion considered to imply consent.


Systematic review

The search retrieved 85 potentially eligible studies with an additional 42 identified from other sources, after screening titles and abstracts, all but nine studies were deemed to be irrelevant (S2 Fig). After further analysis of the full texts one further study was excluded as it was undertaken to determine the frequency that children with CP pass their new born hearing test [18]. Two non-English papers were identified [19,20], for these the abstract and the review by Ponduri et al were used to assess eligibility and extract outcomes. A total of 49 studies were included: eight studies of children with CP [3,2127]; seventeen studies [19,20,2842] identified in the previous review [9] and 24 studies [4366] identified from six Cochrane systematic reviews relating to OME [6772].

Generation of an outcome list

Each outcome measured was listed by study. Only individual outcomes were included, for example, where an outcome was measured using different methods this was counted as one outcome but the methods of measurement noted (S2 Table).

The number of outcomes measured in an individual study varied with a median of 6 outcomes (range 1–14 outcomes) per paper. Outcomes related to resource use were considered to be outside the scope of the COS. Consequently, the outcomes, “necessity to visit doctor” and “level of speech therapy support required” were not considered in the list of outcomes. The final list of outcomes used in round 1 of the Health Professionals Delphi comprised 45 individual outcomes (43 identified from the systematic review and two added by the Study Steering Committee) grouped under 14 domains (Table 1). The list of outcomes scored by parents and children included the combination of some outcomes, for example those that related to specific clinical observations, so that a total of 36 outcomes were scored (Table 1).

Table 1. Outcomes scored by health professionals, parents and children.

Identification of outcomes of importance to parents and children with CP

Semi-structured interviews were completed with 43 parents of 37 children, and 22 children, according to the sampling matrix described in the trial protocol [13]. Interviews with parents lasted, on average, 40 minutes, whilst those with children took, on average, 20 minutes. Parent and child responses to specific questions about outcomes were cross checked against the outcomes list generated from the systematic review of the literature and, as the semi-structured interviews were in parallel to the first round of the health professionals Delphi, were also checked against any free text responses provided by health professionals in round 1. Two investigators (IAB and NLH) mapped each outcome from the interviews against the list of outcomes after round 1. No new outcomes were identified.

Online survey of parents and children

Two hundred and ninety three people accessed the online survey and 253 answered the initial question regarding eligibility. Of the 235 eligible only 51 (22%) completed the survey. Responses were received from 35 parents, eight adults and eight children. Of the eight children, four were in the 7–10 years age group and four aged 11–16 years.

The results were reviewed against the definition of consensus agreed prior to the start of the study [13]. Using this definition, parents and children had reached “consensus in” for 35 and 11 outcomes respectively (Table 2).

Table 2. Summary of all groups reaching consensus for individual outcomes scored in the health professional Delphi survey and online survey for parents and children.

Identification of Outcomes of Importance to Health Professionals

Round 1 –Health Professionals.

The overall response rate per round is given in S4 Table. The number of outcomes reaching consensus within each stakeholder group in each round is shown in Table 3.

The 18 free text responses provided by health professionals in round 1 were reviewed by the SMG and SAG. Eight responses represented a comment and two relating to the use of hearing aids were considered outside the scope of the study. The remaining eight described potential outcomes of which two (listening skills and psychosocial wellbeing) were not already represented and therefore were taken forward to round 2. Thus in round two health professionals scored 47 outcomes.

Round 2 –Health Professionals.

Of the 104 participants who completed round 1 only 99 were eligible to participate in round 2: three clinical geneticists were excluded from further rounds as they were not involved in the management of OME and two participants were on maternity leave at the time of round 2 and therefore would not be able to participate further. A total of 85 responses were received in round 2 (86% of those completing round 1 and eligible for round 2). Participants were shown their own score from round 1 alongside the percentage of participants giving each score from their own stakeholder group. They were informed that they could change their score or keep it the same as their score in round 1. The median percentage of scores changed between round 1 and 2 was 18% (range 0–100%). One participant changed 100% of their score in round 1 whilst six participants (7%) made no changes to their scores.

Round 3- Health Professionals.

After round 2, four participants left the cleft service and so were no longer eligible to participate in round 3. A total of 73 responses were received (90% of those completing round 2 and eligible to complete round 3). All sites were represented in the responses to round 3 with a variable representation of sites and health professional stakeholder groups within site (S4 Table). In round 3, 49 outcomes were scored. One additional outcome “hyperacusis” (sensitivity to loud noises) was identified from free text responses to the parent/child survey. A typing error in the entry of outcomes onto the online system in round 2 had led to “psychosocial wellbeing” being listed as “psychological wellbeing” which is considered to be a different outcome. Therefore in round 3 this was clarified and participants asked to score “psychosocial wellbeing” as well as re-score “psychological wellbeing”. This time, participants were shown their own score in round 2, together with the scores for each of the stakeholder groups including parents and children. The median percentage of scores changed between round 2 and 3 was 21% (range 0–83%). Six participants (8%) made no changes to their scores and no participants changed all scores.

Consensus Matrix.

The scores in round 3 were compared against the definition of consensus to determine which stakeholder groups had reached the definition of “consensus in”. After round 3 all eight stakeholder groups (health professionals plus parents and children) had reached “consensus in” for one outcome “hearing”. Results for all outcomes are given in Table 3.

Attrition bias between rounds.

To identify whether attrition in round 2 would introduce bias, the average score across outcomes from round 1 was calculated for each participant and then compared for those completing both rounds (n = 85) versus those completing round 1 only (n = 14). Likewise in round 3, scores were compared for those completing both rounds 2 and 3 (n = 73) versus those completing round 2 only (n = 8). The results of those who did not complete round 2 or round 3 did not represent extreme views suggesting that bias had not been introduced through attrition between rounds (S3 and S4 Figs).

Variability in outcomes achieving consensus between rounds.

Consensus was reached on additional outcomes within all health professional groups when shown results from either their own stakeholder group, or all groups plus those from parents and children with CP, or both (Table 3). This suggests the Delphi, as opposed to a one-off survey, was a useful exercise.

Consensus Meeting

Twenty five participants attended the consensus meeting of whom 14 were eligible to vote. All stakeholder groups with the exception of clinical psychologists were represented (S6 Table).

Outcomes were discussed in the order of the number of stakeholder groups achieving consensus. Each outcome was categorized based on the following: 1 Discussed and voted (19 outcomes); 2 Discussed and agreed to combine with another outcome and to be considered as part of the “how” an outcome is measured (14 outcomes); 3 Discussed and agreed that further discussion with parents was needed (seven outcomes); 4 Agreed not to discuss further or vote—not in the COS (nine outcomes). A full breakdown of outcomes discussed at the consensus meeting is provided in S5 Table.

The evidence from the health professional Delphi, the parent and child survey and discussion at the consensus meeting followed by voting, were integrated and a COS proposed. This was then further discussed and approved at a follow up meeting of the SAG.

The consensus meeting followed by discussion with the SAG identified 11 outcomes that required further discussion with parents. For example, at the consensus meeting the outcome “listening skills” reached consensus for inclusion in the COS. This outcome was added by health care professionals as part of their Delphi and was not scored by parents or children. Consequently “listening skills” was considered to require further discussion with parents. Outcomes identified at the consensus meeting as requiring additional input, were discussed with parents at a follow up meeting held as a parallel workshop at the CLAPA annual conference, October 2014. Nine parents and one cleft surgeon took part in the workshop. The session included discussion and, if needed, further explanation of each outcome. Each outcome was scored anonymously using an electronic scoring system for immediate feedback. The scores from the workshop were combined with the scores from the consensus meeting and the definition of consensus applied. Following voting, “listening skills” was confirmed for inclusion in the COS. Two additional outcomes, “cholesteatoma” and “otalgia”, also reached consensus in. All outcomes meeting the definition of consensus and included in the recommended COS are shown in Table 4.


This work has produced a consensus recommendation about what outcomes should be measured in studies related to the management of OME in children with CP and was undertaken as a component of the MOMENT study which investigated the feasibility of conducting an RCT in this area (Health Technology Assessment reference 09/167/02). The recommended COS includes: the presence of specific otological conditions (COM, OME and AOM); the potential physical or functional sequelae of these specific otological conditions (hearing loss, cholesteatoma, listening skills, psychosocial development, hearing, receptive language skills and speech development); and the potential negative consequences of treatment (side effects of treatment). The outcome “side effects of treatment” included in the COS will be dependent on the interventions/treatments that are being compared in a particular study

A recent systematic review of studies describing COS development demonstrated variability in stakeholder involvement with only 18% of studies including public representatives in the process [6]. The opinions of parents and children about the treatment of OME for children with CP are essential because this group will experience both the benefits and adverse effects of treatments, and be involved in decision-making about treatment. Importantly, the development of the COS in the MOMENT study has considered the opinion of patients and parents to ensure that outcomes regarded as most important, and included in the COS, are relevant to this stakeholder group.

We have demonstrated that clinical outcomes can be translated into plain language and that both parents and children are able to score these outcomes in an online survey. In this particular study, children were more discerning than their parents when considering which outcomes are most important to them with 11 outcomes reaching the definition of “consensus in”.

The use of an online survey of parents and children has allowed a broader range of outcomes to be considered than if interviews alone were used. Time constraints of the present study meant that only a one off survey was possible. However, future COS development should consider multiple rounds completed by patients/parents stakeholders in which the responses from health professionals can also be taken into account. Certainly the multiple rounds completed by health professionals resulted in changes being made to scores indicating that the responses of peers, parents and children and other health professional groups had an impact on the perceived importance of outcomes.

Strengths and limitations

We have used an efficient online system to deliver a multiple round Delphi to health care professionals that allowed automated collating of scores and feedback in each round together with automated email alerts to promote completion. We have shown that individuals do reflect on, and are influenced by, other groups' opinions about the importance of outcomes. This online system was also successfully modified for use by parents and children in a one off survey.

Both clinical and patient stakeholders were engaged, with the response rate of health professionals similar to that reported in other Delphi surveys [7375]. Notably the attrition rate was low with those taking part in round 1 likely to complete all rounds; as a result, no attrition bias was introduced. Whilst clinical stakeholder representation was good the number of parents and children completing the online survey and attending the face to face consensus meeting was lower than expected. In the present study it was not possible to ascertain whether the length of the survey for parents and children, the method of delivery or indeed the importance of the research question, due to perceived impact of OME, contributed to the low response rate. It is possible that those individuals completing the online survey are not representative of the wider group in terms of their views about important outcomes. Thirty seven parents were interviewed and these considered fewer outcomes to be critically important compared to parents who completed the online survey. However, with the exception of “externalising behaviour”, mentioned by one parent but not as the most important outcome, the opinions of those parents who were interviewed were comparable with the opinions expressed by parents completing the online survey. Research is needed on how best to engage with patients and/or their parents to facilitate patient involvement with the different stages of the COS development.

Future Work

The consensus meeting and follow up meeting with parents has resulted in a COS which recommends what to measure. However, if future research measures these outcomes in different ways it will still be difficult to compare studies. The next steps will involve consideration of how each of these outcomes should be defined and measured. For each outcome, definitions and measurement instruments will need to be reviewed, whether a validated tool already exists and what methods have been used to measure this outcome in previous studies, as described in the systematic review (S2 Table).

For each of the outcomes included in the recommended COS this will include: consideration of methods of assessing hearing that might be influenced by the intervention, for example, differing methods depending on ventilation tube or hearing aid use; agreeing a definition of COM and methods of measurement; determining which aspects of speech development should be measured and identifying whether methods of measurement are already available; reviewing methods for assessment of receptive language, psychosocial development, AOM, listening skills, cholesteatoma and otalgia; establishing the most appropriate way to measure side effects of treatment; consideration of the impact of patient age group on the chosen method of assessment. With the exception of “listening skills” all outcomes have been measured in one or more of the studies identified in the systematic review.

Guidelines for the selection of outcome measurement instruments to be included in a COS are being developed by the Core Outcome Measurement Instrument Selection (COMIS) project [76] and will be consulted when available. Furthermore, the UK Cleft Audit means that for some outcomes there are potentially methods of measurement that have already been agreed by health professionals providing cleft care in the UK [77,78].

Our study recommends a core outcome set but also acknowledges that further work is needed to identify agreed methods of measurement for each of the outcomes as this is beyond the scope of the current study. As part of the methods development process consideration will be given to the age at which outcome assessments are appropriate and this will inform the length of follow up needed in a given trial.

The length of follow up will not be mandated and a similar approach will be adopted as by the OMERACT group where, if appropriate, outcomes will only be considered core should they be appropriate to the age group of participants and duration of follow up. For example, the OMERACT core outcome set includes one outcome which is only relevant should the duration of follow up be greater than 52 weeks[79]. The MOMENT study has involved multiple key stakeholder groups from the UK to ensure that a COS is suitable and well accepted in future research. However, to promote good uptake of the COS into future studies international consensus is needed. Cleft organisations exist in both Europe (The European Cleft Organisation) and the United States (American Cleft Palate-Craniofacial Association) and we plan to work with COMET [80]to pursue engagement of international health professionals through their membership.

It should also be noted that a COS is not static and should be revised or updated as new information becomes available. Should future trials using the COS recommended in this paper identify difficulties in its application then review of the included outcomes would be warranted.

Whilst OME affects around 75% of children with CP, it is also a common condition for children without cleft, with almost a fifth of children aged 1–5 years affected [1]. The COS described in the current study includes outcomes that have been identified from previous studies in both cleft and non-cleft populations suggesting that they may also be of relevance to studies of OME in children without CP. Further work in this area and engagement with stakeholders is warranted.

Supporting Information

S1 Fig. Scoring categories for health care professionals, parents, children and young people.


S3 Fig. Average scores in round 1 across all outcomes by stakeholder group.

Shaded bars represent those who provided scores in round 1 only, open bars represent those scoring in both rounds 1 and 2.


S4 Fig. Average scores in round 2 across all outcomes by stakeholder group.

Shaded bars represent those who provided scores in round 2 only, open bars represent those scoring in both rounds 2 and 3.


S2 Table. Characteristics of included papers including outcomes used.


S3 Table. Breakdown of participants invited and completing all three rounds of the Delphi.


S4 Table. Breakdown of response rate in each round by health professional group.


S5 Table. Summary of outcomes discussed at consensus meeting.


S6 Table. Stakeholder Representation at consensus meeting.



We acknowledge both the time and the efforts from:

Study Steering Committee members Mike Clarke (Professor/Director of MRC Methodology Hub), Professor Anne Schilder (NIHR Research Professor Director), Professor Andrea Manca (Professor of Health Economics, Centre for Health Economics) and Rosanna Preston (Chief Executive of the Cleft Lip and Palate Association).

Alison Hooper (Paediatric Audiologist, University Hospitals Bristol NHS Foundation Trust) and Rona Slator (Consultant Plastic Surgeon, Birmingham Children's Hospital NHS Foundation Trust) for their expertise provided at the CLAPA conference workshop.

The Cleft Lip and Palate Association including the Children and Young Person’s Council, Adult Voices and Happy Faces groups for their support and input into the development and delivery of the Delphi survey for parents and children.

We would like to thank the children and parents who took part in an interview and children parents and health care professionals who attended the consensus meeting or completed the survey/Delphi.

Duncan Appelbe and Melanie Harper- Jones (Clinical Trials research Centre, University of Liverpool) for their contribution to the design of the online system for the Delphi survey of health professionals and online survey of parents and children with cleft palate and Patricia Barnby for her contribution to the design of the online system for the Delphi survey of Health Professionals. Joyce Russell, Jeanette Mooney and Haydn Bellardie for their support and contribution to the qualitative protocol. Tri Tat for comments on an earlier version of the Delphi survey for health professional’s protocol. Claire Bennett for administrative support of the study. Mohammed Owaise Sharif for his contribution to the literature search and identification of eligible papers.

Author Contributions

Conceived and designed the experiments: PRW IAB NLH KO. Performed the experiments: NLH IAB PRW ST PC. Analyzed the data: NLH IAB PRW JJK ST PC AMDB RC PNH AHB VHP NR DS RS. Wrote the paper: NH PRW IAB. Review and approval of the manuscript: JJK ST PC KO AMDB RC PNH AHB VHP NR DS RS.


  1. 1. Flynn T, Möller C, Jönsson R, Lohmander A (2009) The high prevalence of otitis media with effusion in children with cleft lip and palate as compared to children without clefts. International Journal of Pediatric Otorhinolaryngology 73: 1441–1446. pmid:19709760
  2. 2. Maheshwar AA, Milling MAP, Kumar M, Clayton MI, Thomas A (2002) Use of hearing aids in the management of children with cleft palate. International Journal of Pediatric Otorhinolaryngology 66: 55–62. pmid:12363423
  3. 3. Civelek B, Celebioglu S, Sagit M, Akin I (2007) Ventilation tubes in secretory otitis media associated with cleft palate: A retrospective analysis. Turkish Journal of Medical Sciences 37: 223–226.
  4. 4. Goldman JL, Martinez SA, Ganzel TM (1993) Eustachian tube dysfunction and its sequelae in patients with cleft palate. South Med J 86: 1236–1237. pmid:8235774
  5. 5. NICE (2008) Surgical management of children with otitis media with effusion (OME). Clinical guidelines, CG60. National Institute for Health and Clinical Excellence.
  6. 6. Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. (2014) Choosing Important Health Outcomes for Comparative Effectiveness Research: A Systematic Review. PLoS ONE 9: e99111. pmid:24932522
  7. 7. Williamson P, Altman D, Blazeby J, Clarke M, Devane D, Gargon E, et al. (2012) Developing core outcome sets for clinical trials: issues to consider. Trials 13: 132. pmid:22867278
  8. 8. Kirkham JJ, Boers M, Tugwell P, Clarke M, Williamson PR (2013) Outcome measures in rheumatoid arthritis randomised trials over the last 50 years. Trials 14: 324. pmid:24103529
  9. 9. Ponduri S, Bradley R, Ellis PE, Brookes ST, Sandy JR, Ness AR (2009) The Management of Otitis Media With Early Routine Insertion of Grommets in Children With Cleft Palate—A Systematic Review. The Cleft Palate-Craniofacial Journal 46: 30–38. pmid:19115800
  10. 10. Sinha I, Jones L, Smyth RL, Williamson PR (2008) A Systematic Review of Studies That Aim to Determine Which Outcomes to Measure in Clinical Trials in Children. PLoS Med 5: e96. pmid:18447577
  11. 11. Clarke M (2007) Standardising outcomes for clinical trials and systematic reviews. Trials 8: 39. pmid:18039365
  12. 12. Cooney R, Warren B, Altman D, Abreu M, Travis S (2007) Outcome measurement in clinical trials for ulcerative colitis: towards standardisation. Trials 8: 17. pmid:17592647
  13. 13. Harman N, Bruce I, Callery P, Tierney S, Sharif MO, O'Brien K, et al. (2013) MOMENT—Management of Otitis Media with Effusion in Cleft Palate: protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey. Trials 14: 70. pmid:23497540
  14. 14. Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. (2011) GRADE guidelines: 2. Framing the question and deciding on important outcomes. Journal of Clinical Epidemiology 64: 395–400. pmid:21194891
  15. 15. Tierney S, O’Brien K, Harman NL, Madden C, Sharma RK, Callery P (2013) Risks and benefits of ventilation tubes and hearing aids from the perspective of parents of children with cleft palate. International Journal of Pediatric Otorhinolaryngology 77: 1742–1748. pmid:24007893
  16. 16. Tierney S, O'Brien K, Harman NL, Sharma RK, Madden C, Callery P (2013) Otitis Media With Effusion: Experiences of Children With Cleft Palate and Their Parents. The Cleft Palate-Craniofacial Journal.
  17. 17. NIACE (2014) Available:
  18. 18. Chen JL, Messner AH, Curtin G (2008) Newborn Hearing Screening in Infants With Cleft Palates. Otology & Neurotology 29: 812–815 810.1097/MAO.1090b1013e318180a318184e318180.
  19. 19. Liu L, Sun Y-g, Ma L, Zhao W, Wu R (2004) Effect of ventilation tube insertion on otitis media with effusion in cleft palate children. Zhonghua er bi yan hou ke za zhi 39: 216–218. pmid:15283281
  20. 20. Zheng Q, Xu H, He Y (2003) Effects of tympanotomy and pressure equilibrium tube insertion during palatoplasty on prognoses of otitis media with effusion. Hua xi kou qiang yi xue za zhi = Huaxi kouqiang yixue zazhi = West China journal of stomatology 21: 28–30.
  21. 21. Curtin G, Messner AH, Chang KW (2009) Otorrhea in infants with tympanostomy tubes before and after surgical repair of a cleft palate. Arch Otolaryngol Head Neck Surg 135: 748–751. pmid:19687392
  22. 22. Hornigold R, Morley A, Glore RJ, Boorman J, Sergeant R (2008) The long-term effect of unilateral t-tube insertion in patients undergoing cleft palate repair: 20-year follow-up of a randomised controlled trial. Clinical Otolaryngology 33: 265–268. pmid:18559037
  23. 23. Merrick GD, Kunjur J, Watts R, Markus AF (2007) The effect of early insertion of grommets on the development of speech in children with cleft palates. British Journal of Oral & Maxillofacial Surgery 45: 527–533.
  24. 24. Phua YS, Salkeld LJ, de Chalain TM (2009) Middle ear disease in children with cleft palate: protocols for management. Int J Pediatr Otorhinolaryngol 73: 307–313. pmid:19091429
  25. 25. Reiter R, Haase S, Brosch S (2009) Repaired Cleft Palate and Ventilation Tubes and Their Associations With Cholesteatoma in Children and Adults. Cleft Palate-Craniofacial Journal 46: 598–602. pmid:19929085
  26. 26. Szabo C, Langevin K, Schoem S, Mabry K (2010) Treatment of persistent middle ear effusion in cleft palate patients. International Journal of Pediatric Otorhinolaryngology 74: 874–877. pmid:20537733
  27. 27. Tanpowpong K, Saisukul I, Kittimanont H, Rattanasiri S (2007) Outcome of myringotomy with ventilation tube for otitis media with effusion in Thai children: Ramathibodi experiences. J Med Assoc Thai 90: 1866–1871. pmid:17957932
  28. 28. Paradise JL, Bluestone CD (1974) Early treatment of the universal otitis media of infants with cleft palate. Pediatrics 53: 48–54. pmid:4809194
  29. 29. Moller P (1981) Hearing, Middle-Ear Pressure And Otopathology In A Cleft-Palate Population. Acta Oto-Laryngologica 92: 521–528. pmid:7198369
  30. 30. Frable MA, Brandon GT, Theogaraj SD (1985) Velar Closure And Ear Tubings As A Primary Procedure In The Repair Of Cleft Palates. Laryngoscope 95: 1044–1046. pmid:4033325
  31. 31. Smith TL, DiRuggiero DC, Jones KR (1994) Recovery of eustachian tube function and hearing outcome in patients with cleft palate. Otolaryngol Head Neck Surg 111: 423–429. pmid:7936674
  32. 32. Sheahan P, Blayney AW, Sheahan JN, Earley MJ (2002) Sequelae of otitis media with effusion among children with cleft lip and/or cleft palate. Clinical Otolaryngology 27: 494–500. pmid:12472518
  33. 33. Freeland AP, Evans DM (1981) Middle-Ear Disease In The Cleft-Palate Infant—Its Effect On Speech And Language-Development. British Journal of Plastic Surgery 34: 142–143. pmid:7236969
  34. 34. Hubbard TW, Paradise JL, McWilliams BJ, Elster BA, Taylor FH (1985) Consequences of unremitting middle-ear disease in early life. Otologic, audiologic, and developmental findings in children with cleft palate. N Engl J Med 312: 1529–1534. pmid:4039792
  35. 35. Gordon ASD, Jeanlouis F, Morton RP (1988) Late Ear Sequelae In Cleft-Palate Patients. International Journal of Pediatric Otorhinolaryngology 15: 149–156. pmid:3397234
  36. 36. Robson AK, Blanshard JD, Jones K, Albery EH, Smith IM, Maw AR (1992) A Conservative Approach to the Management of Otitis Media with Effusion in Cleft Palate Children Journal of Laryngology and Otology 106: 788–792.
  37. 37. Shaw R, Richardson D, McMahon S (2003) Conservative management of otitis media in cleft palate. Journal of Cranio-Maxillofacial Surgery 31: 316–320. pmid:14563333
  38. 38. Hormann K, Roehrs M (1991) Middle-ear findings in young cleft lip and palate children. Comparison of two treatment clinics. Deutsche Zeitschrift fur Mund-, Kiefer- und Gesichts-Chirurgie 15: 149–152. pmid:1816937
  39. 39. Broen PA, Moller KT, Carlstrom J, Doyle SS, Devers M, Keenan KM (1996) Comparison of the hearing histories of children with and without cleft palate. Cleft Palate-Craniofacial Journal 33: 127–133. pmid:8695620
  40. 40. Greig AV, Papesch ME, Rowsell AR (1999) Parental perceptions of grommet insertion in children with cleft palate. J Laryngol Otol 113: 1068–1071. pmid:10767917
  41. 41. Møller P (1982) Seletive use of ventilating tubes in the treatment of secretory otitis media and retractions of the ear drum. Acta Oto-Laryngologica 93: 158–160.
  42. 42. Potsic WP, Cohen M, Randall P, Winchester R (1979) Retrospective Study Of Hearing Impairment In 3 Groups Of Cleft-Palate Patients. Cleft Palate Journal 16: 56–58. pmid:282030
  43. 43. Black NA, Sanderson CFB, Freeland AP, Vessey MP (1990) A Randomized Controlled Trial Of Surgery For Glue Ear. British Medical Journal 300: 1551–1556. pmid:2196954
  44. 44. Casselbrant ML, Mandel EM, Rockette HE, Kurs-Lasky M, Fall PA, Bluestone CD (2009) Adenoidectomy for otitis media with effusion in 2-3-year-old children. International Journal of Pediatric Otorhinolaryngology 73: 1718–1724. pmid:19819563
  45. 45. Dempster JH, Browning GG, Gatehouse SG (1993) A Randomized Study Of The Surgical-Management Of Children With Persistent Otitis-Media With Effusion Associated With A Hearing Impairment. Journal of Laryngology and Otology 107: 284–289. pmid:8320510
  46. 46. Fiellaunikolajsen M (1980) Adenoidectomy For Middle-Ear Disorders—A Randomized Controlled Trial. Clinical Otolaryngology 5: 424–424.
  47. 47. Gates GA, Avery CA, Cooper JC Jr, Prihoda TJ (1989) Chronic secretory otitis media: effects of surgical management. Ann Otol Rhinol Laryngol Suppl 138: 2–32. pmid:2492178
  48. 48. Roydhouse N (1980) Adenoidectomy for Otitis Media with Mucoid Effusion. Annals of Otology Rhinology and Laryngology 89: 312–315.
  49. 49. Rynneldagoo B, Ahlbom A, Schiratzki H (1978) Effects of Adenoidectomy—Controlled 2-Year Follow Up. Annals of Otology Rhinology and Laryngology 87: 272–278.
  50. 50. Zielhuis GA, Rach GH, Vandenbroek P (1989) Screening for Otitis Media with Effusion in Preschool Children Lancet 1: 311–314. pmid:2563465
  51. 51. Hammaren-Malmi S, Saxen H, Tarkkanen J, Mattila PS (2005) Adenoidectomy does not significantly reduce the incidence of otitis media in conjunction with the insertion of tympanostomy tubes in children who are younger than 4 years: A randomized trial. Pediatrics 116: 185–189. pmid:15995051
  52. 52. Johnston LC, Feldman HM, Paradise JL, Bernard BS, Colborn DK, Casselbrant ML, et al. (2004) Tympanic membrane abnormalities and hearing levels at the ages of 5 and 6 years in relation to persistent otitis media and tympanostomy tube insertion in the first 3 years of life: A prospective study incorporating a randomized clinical trial. Pediatrics 114: E58–E67. pmid:15231974
  53. 53. Koivunen P, Uhari M, Luotonen J, Kristo A, Raski R, Pokka T, et al. (2004) Adenoidectomy versus chemoprophylaxis and placebo for recurrent acute otitis media in children aged under 2 years: randomised controlled trial. British Medical Journal 328: 487–490. pmid:14769785
  54. 54. Mandel EM, Casselbrant ML, Rockette HE, Fireman P, Kurs-Lasky M, Bluestone CD (2002) Systemic steroid for chronic otitis media with effusion in children. Pediatrics 110: 1071–1080. pmid:12456902
  55. 55. Mattila PS, Joki-Erkkila VP, Kilpi T, Jokinen J, Herva E, Puhakka H (2003) Prevention of otitis media by adenoidectomy in children younger than 2 years. Archives of Otolaryngology-Head & Neck Surgery 129: 163–168.
  56. 56. Maw R (1999) Early surgery compared with watchful waiting for glue ear and effect on language development in preschool children (vol 353, pg 960, 1999). Lancet 354: 1392–1392. pmid:10533897
  57. 57. Nguyen LHP, Manoukian JJ, Yoskovitch A, Al-Sebeih KH (2004) Adenoidectomy: Selection criteria for surgical cases of otitis media. Laryngoscope 114: 863–866. pmid:15126745
  58. 58. Paradise JL, Bluestone CD, Colborn DK, Bernard BS, Smith CG, Rockette HE, et al. (1999) Adenoidectomy and adenotonsillectomy for recurrent acute otitis media—Parallel randomized clinical trials in children not previously treated with tympanostomy tubes. Jama-Journal of the American Medical Association 282: 945–953.
  59. 59. Paradise JL, Campbell TF, Dollaghan CA, Feldman HM, Bernard BS, Colborn DK, et al. (2005) Developmental outcomes after early or delayed insertion of tympanostomy tubes. New England Journal of Medicine 353: 576–586. pmid:16093466
  60. 60. Paradise JL, Dollaghan CA, Campbell TF, Feldman HM, Bernard BS, Colborn DK, et al. (2003) Otitis media and tympanostomy tube insertion during the first three years of life: Developmental outcomes at the age of four years. Pediatrics 112: 265–277. pmid:12897272
  61. 61. Paradise JL, Feldman HM, Campbell TF, Dollaghan CA, Colborn DK, Bernard BS, et al. (2001) Effect of early or delayed insertion of tympanostomy tubes for persistent otitis media on developmental outcomes at the age of three years. New England Journal of Medicine 344: 1179–1187. pmid:11309632
  62. 62. Paradise JL, Feldman HM, Campbell TF, Dollaghan CA, Rockette HE, Pitcairn DL, et al. (2007) Tympanostomy tubes and developmental outcomes at 9 to 11 years of age. New England Journal of Medicine 356: 248–261. pmid:17229952
  63. 63. Rach GH, Zielhuis GA, Vanbaarle PW, Vandenbroek P (1991) The effect of treatment with ventialting tubes on language development in preschool children with otitis media with effusion. Clinical Otolaryngology 16: 128–132. pmid:2070526
  64. 64. Rovers MM, Straatman H, Ingels K, van der Wilt GJ, van den Broek P, Zielhuis GA (2000) The effect of ventilation tubes on language development in infants with otitis media with effusion: A randomized trial. Pediatrics 106: art. no.-e42.
  65. 65. Maw AR (1983) Chronic otitis-media with effusion (glue ear) and adenotonsillectomy—prospective randomized controlled-study. British Medical Journal 287: 1586–1588. pmid:6416513
  66. 66. Paradise JL, Bluestone CD, Rogers KD, Taylor FH, Colborn DK, Bachman RZ, et al. (1990) Efficacy of adenoidectomy for recurrent otitis media in children previously treated with tympanostomy-tube placement. Results of parallel randomized and nonrandomized trials. Jama 263: 2066–2073. pmid:2181158
  67. 67. Simpson SA, Thomas CL, van der Linden MK, Macmillan H, van der Wouden JC, Butler C (2007) Identification of children in the first four years of life for early treatment for otitis media with effusion. Cochrane Database Syst Rev: Cd004163.
  68. 68. Thomas CL, Simpson S, Butler CC, van der Voort JH (2006) Oral or topical nasal steroids for hearing loss associated with otitis media with effusion in children. Cochrane Database Syst Rev: Cd001935.
  69. 69. van den Aardweg MT, Schilder AG, Herkert E, Boonacker CW, Rovers MM (2010) Adenoidectomy for otitis media in children. Cochrane Database Syst Rev: Cd007810.
  70. 70. Griffin GH, Flynn C, Bailey RE, Schultz JK (2006) Antihistamines and/or decongestants for otitis media with effusion (OME) in children. Cochrane Database Syst Rev: Cd003423.
  71. 71. Browning GG, Rovers MM, Williamson I, Lous J, Burton MJ (2010) Grommets (ventilation tubes) for hearing loss associated with otitis media with effusion in children. Cochrane Database Syst Rev: Cd001801.
  72. 72. Perera R, Haynes J, Glasziou P, Heneghan CJ (2006) Autoinflation for hearing loss associated with otitis media with effusion. Cochrane Database Syst Rev: Cd006285.
  73. 73. Devane D, Begley CM, Clarke M, Horey D, O’Boyle C (2007) Evaluating maternity care: a core set of outcome measures. Birth 34: 164–172. pmid:17542821
  74. 74. Khanna D, Lovell DJ, Giannini E, Clements PJ, Merkel PA, Seibold JR, et al. (2008) Development of a provisional core set of response measures for clinical trials of systemic sclerosis. Annals of the Rheumatic Diseases 67: 703–709. pmid:17893248
  75. 75. Mease PJ, Arnold LM, Crofford LJ, Williams DA, Russell IJ, Humphrey L, et al. (2008) Identifying the clinical domains of fibromyalgia: contributions from clinician and patient Delphi exercises. Arthritis Rheum 59: 952–960. pmid:18576290
  76. 76. Prinsen CA, Vohra S, Rose M, King-Jones S, Ishaque S, Bhaloo Z, et al. (2014) Core Outcome Measures in Effectiveness Trials (COMET) initiative: protocol for an international Delphi study to achieve consensus on how to select outcome measurement instruments for outcomes included in a 'core outcome set'. Trials 15: 247. pmid:24962012
  77. 77. Sell D, John A, Harding-Bell A, Sweeney T, Hegarty F, Freeman J (2009) Cleft audit protocol for speech (CAPS-A): a comprehensive training package for speech analysis. Int J Lang Commun Disord 44: 529–548. pmid:18821108
  78. 78. John A, Sell D, Sweeney T, Harding-Bell A, Williams A (2006) The cleft audit protocol for speech-augmented: A validated and reliable measure for auditing cleft speech. Cleft Palate Craniofac J 43: 272–288. pmid:16681400
  79. 79. Boers M, Tugwell P, Felson DT, van Riel PL, Kirwan JR, Edmonds JP, et al. (1994) World Health Organization and International League of Associations for Rheumatology core endpoints for symptom modifying antirheumatic drugs in rheumatoid arthritis clinical trials. J Rheumatol Suppl 41: 86–89. pmid:7799394
  80. 80. COMETinitative. Available: