https://orcid.org/0000-0001-9203-9903
Figures
Abstract
Improvement of patients’ oral health-related quality of life (OHRQoL) is the main goal of oral health care professionals. However, OHRQoL is not a homogenous construct and how to assess it is challenging because of the large number of currently available instruments. Investigating available instruments and what they have in common would be necessary for consolidation and standardization of these instruments into a smaller set of tools. If the OHRQoL dimensions including Oral Function, Orofacial Pain, Orofacial Appearance, and Psychosocial Impact are the fundamental building blocks of the dental patient’s oral health experience, then these dimensions should be measured by generic multi-item dPROMs. In this study, a panel of 11 international dentists use the Delphi consensus process to determine how well 20 of these instruments measured the four OHRQoL dimensions. All 20 dPROMs questionnaires assessed at least one OHRQoL dimension while all four OHRQoL dimensions were measured by at least one dPROM instrument, i.e., the four OHRQoL dimensions were essential components of the patient’s oral health experience. This shows that the currently available generic multi-item dPROMs have a lot in common, in that they share Oral Function, Orofacial Pain, Orofacial Appearance, and Psychosocial Impact as targeted dimensions. Based on these commonalities, it is plausible and desirable to move towards a single four-dimensional metric to assess oral health impact in all clinical, community-based, and research settings. This step is necessary to advance evidence-based dentistry and value-based oral health care.
Citation: Chanthavisouk P, John MT, Paulson D, Pattanaik S (2022) Commonalities among dental patient-reported outcomes (dPROs)—A Delphi consensus study. PLoS ONE 17(6): e0268750. https://doi.org/10.1371/journal.pone.0268750
Editor: Kelvin I. Afrashtehfar, University of Bern: Universitat Bern, SWITZERLAND
Received: January 21, 2022; Accepted: May 6, 2022; Published: June 22, 2022
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: Data has been uploaded to the University of Minnesota public data repository Link: https://doi.org/10.13020/7s0c-wv21.
Funding: The research reported in this publication was supported by the National Institute of Dental and Craniofacial Research of the National Institutes of Health, USA, under the Award Numbers R01DE022331 and R01DE028059.
Competing interests: The authors have declared that no competing interests exist.
Background
Dental patient-reported outcomes (dPROs) represent what is important to dental patients [1,2]. Therefore, dental patient-reported outcome measures (dPROMs) determine which dental treatments are the most effective in addressing dental patients’ oral health problems. Two systematic reviews identified 155 multi-item dPROMs that capture oral disease impact [3,4]. These dPROMs provide a multitude of opportunities to measure oral disease impact on the quality of life, but they also present challenges to dentists and researchers such as which instrument to select and how to interpret instrument scores. The dPROMs essentially measure one or more of the four dimensions of OHRQoL Oral Function, Orofacial Pain, Orofacial Appearance, and Psychosocial Impact [5]. A search for commonalities among the currently available dPROMs could simplify and consolidate the complex array of existing dPROMs into a smaller and better set of instruments to measure the impact of oral disease. This would also facilitate the determination of treatment effectiveness in dental interventions comparable to the results of research studies. Consequently, evidence-based dental practice and value-based oral health care would greatly benefit from a standardization of patient-reported outcomes assessment.
A Delphi technique is such a structured process to derive a consensus about a topic such as the relationship between dPROMs and OHRQoL dimensions. The conceptual advantages of a Delphi process are accompanied by technical advantages of involving many dental experts with international representation and reasonable burden using electronic communication. This study implemented the Delphi process using a panel of international dental experts and aimed to investigate the commonalities among 20 generic multi-item dental patient-reported outcome measures (dPROMs).
Methods and material
In the present study, 20 questionnaires that measure oral disease impact were assigned to the four OHRQoL dimensions Oral Function, Orofacial Pain, Orofacial Appearance, or Psychosocial Impact by dental experts using a Delphi process after undergoing a reliability assessment and reaching consensus. The oral disease-generic dPROMs were identified as instruments that measured oral disease impact across of a broad range of patients [3]. The 20 questionnaires contained 36 unique dPROs. These dPROs were measured by 53 dPROMs [3].
Dental experts
The dental experts comprises of a group of 11 international dentists, known to one of the authors (MTJ). The selection criteria for dental experts were that they had to be a dentist, had to have practiced dentistry in the past year, and were fluent in the English language. Each dental expert received an invitation by email with detailed study instructions. The dental experts were requested to familiarize themselves with the contents of 20 dPROMs. They were provided the abstract of the published dPROM questionnaires by each author(s), a brief description of what the dPROM intended to measure, the reference of the dPROM, as well as the dPROM items. All dental experts who were approached to participate in the study had accepted the invitation, providing a response rate of 100%.
All dentists were full-time dentists, six were women and five men. Of those dental experts, six were from Europe and the rest from the United States. Five of the dentists were Doctorates of Dental Surgery (DDS) and a Doctor of Philosophy (PhD). The dental experts have had between five to 25 years of experience in the dental field. The major criterion for the dental experts were that they had to be a dentist. Dental experts should have had some understanding and experience about patients’ functional, pain-related, aesthetical, and psychosocial problems in regards to oral health [6].
The Delphi process
Calibration of dental experts.
The definitions of the four dimensions of OHRQoL: Oral Function, Orofacial Pain, Orofacial Appearance and Psychosocial Impact were provided to all dental experts. They were asked to familiarize themselves with their meaning and content, even if they had an intuitive understanding of what these dimensions were. These dimensions were identified from 10,778 49-item Oral Health Impact Profile (OHIP-49) data of prosthodontic patients and general population subjects covering an age range of 40 years or more in 35 studies conducted in six countries [7–10]. The OHIP-49 data formed an aggregate secondary data set used for the factor analyses in the Dimension of Oral Health-Related Quality of Life Project [7].
Assignment of PROM to OHRQoL dimensions using the Delphi technique.
The data was collected using an electronic survey in the English language. The survey was generated using Qualtrics software. Dental experts received an electronic link to the survey and provided their anonymous responses online by choosing a 7-digit number that was not to be shared with other dental experts or the study organizer. This number was used to link participants’ responses to the corresponding dental expert in both round one and round two. While the number identified a particular dental expert, completion of all ratings was anonymous.
The dental experts were requested to assess the 20 questionnaires one by one. The dental experts were asked to indicate with the four scores as to what extent the specific questionnaire assesses each of the four dimensions. The scores could range from 0 to 10 points, where 0 points meant the questionnaire does not measure an OHRQoL dimension at all and 10 points meant the questionnaire measures the dimension perfectly.
In subsequent rounds of the Delphi process, dental experts did not have access to their own previous ratings but were shown summarized group scores from other dental experts. For each questionnaire, rating medians and interquartile for all four dimensions were presented. The dental experts were asked to provide ratings again, but this time based on the knowledge of what the group’s assessment was. The dental experts were informed that group ratings can but do not have to influence their own new assessment. The Delphi process was stopped when a consensus was reached, which has been defined in the following section labelled as “Consensus—Validity assessment of panel expert ratings.”
Reliability assessment of dental experts’ ratings.
To find the commonalities among dPROs using dental experts’ opinions, the expert ratings needed to be stable over time, i.e., an expert should have the same opinion when asked again. Therefore, a test-retest assessment was performed by asking dental experts twice within two weeks to assign questionnaires to dimensions without knowledge of previous results. Ratings from the first round of the Delphi process were used as “test” results. Before dental experts were given feedback on the group’s assessment and ask to continue with the second round of the Delphi process, another survey round (“retest”) was performed to determine test-retest reliability.
Out of 11 dental experts, only seven provided retest data. We calculated an intraclass correlation coefficient (ICC) according to Fleiss [11], using a one-way ANOVA for each expert and for each dimension, resulting in 28 ICCs from 7 (number of raters) times 4 (number of dimensions). Box plots were used to present central tendency and spread of ICCs overall and per OHRQoL dimension. The median ICCs were interpreted according to Fleiss [11]. An ICC <0.40 is considered “poor,” 0.40–0.75 represents “fair-to-good,” and >0.75 is seen as “excellent” reliability. The software that was used to interpret this data is called, Stata.
The internal consistency of the panel expert ratings was also determined. Cronbach’s alpha, the overall and by dimension, was computed for the dental expert responses when they were asked which dimension a dPROM belonged to. A Cronbach’s alpha value of 0.90 or higher was adopted as the threshold for sufficient panel expert ratings’ internal consistency [12].
Consensus–validity assessment of panel expert ratings.
A Delphi study does not lead automatically to a consensus; it simply provides an opportunity to reach a consensus [13]. Only if dental experts agree on the topic, the result can be considered a consensus [13]. Therefore, we examined the “dimensionality” of dental expert ratings. Dimensionality refers to the number and nature of the attributes reflected in the dental experts’ responses. If dental expert responses could be considered unidimensional, i.e., they would only measure one attribute, it could be assumed that the dental experts have reached a consensus on the topic of commonalities among multi-item dPROMs that measure disease impact. In the case of dental experts’ response ratings being multi-dimensional, i.e., several attributes would be measured, it could be assumed that the dental experts would not have a homogenous opinion about the topic, indicating that a consensus was not reached.
To determine the dimensionality of dental expert ratings, a parallel analysis was performed suggested by Horn [14]. For 9 dental experts, this technique calculated the 9 dental expert’s ratings x 9 dental expert ratings correlation matrix from the 20 PRO questionnaires x 9 dental experts dataset. This technique created a scree plot [15]. The eigenvalues of the sample correlation matrix were plotted against their position from largest to smallest (1, 2, …, 9). The eigenvalue points related to a straight line. The parallel analysis also created a simulated dataset with 180 observations randomly sampled from 9 independent normal variates. It calculated a 9 x 9 correlation matrix for the simulated data and extracted the 9 eigenvalues, ordering them from largest to smallest. This step was repeated k times (in our case, k = 1000). From 1000 eigenvalues at positions 1, 2, …, 9, medians were calculated. The 9 eigenvalue medians were related to a dashed line and overlaid on the Scree plot. Finally, the intersection of the solid (actual data) and the dashed lines (simulated data) was the cutoff for determining the number of dimensions present in the data. Panel expert ratings would be considered unidimensional when only one eigenvalue would be above the dashed line of the simulated data.
Assignment of questionnaires to OHRQoL dimensions.
Having reached a consensus, median ratings for the group of raters were calculated for each questionnaire and for all four dimensions. Consequently, each questionnaire received four median ratings. For example, the “Psychological Impact of Dental Aesthetics Questionnaire” [16] received 0, 0, 9, and 10 points for median ratings for the dimensions Oral Function, Orofacial Pain, Orofacial Appearance, and Psychosocial Impact, respectively.
From this example, it becomes clear that situations such as 0, 0, 0, and 10 points, indicating a perfect fit for only one dimension, could have rarely occurred. It is expected that dPRO questionnaires measure several dimensions. Therefore, in addition to considering only median ratings of equal to or larger than 5 on the 0 to 10 numeric rating scale as, the highest median rating (in previous example, the 10 points) and any other median ratings within 2 points of the highest median rating (in the previous example, the 9 points) were considered indicators of the dimension. Therefore, the example of the questionnaire “Psychological Impact of Dental Aesthetics Questionnaire” measured Psychosocial Impact (score of 10) and Orofacial Appearance (score of 9) but did not measure the Oral Function (score of 0), or the Orofacial Pain (score of 0).
Two hypotheses about the presence of commonalities among dPROMs
To interpret commonalities among the 20 questionnaires, two hypotheses were generated:
- All 20 dPRO questionnaires assess at least one OHRQoL dimension.
- All four OHRQoL dimensions are measured by at least one questionnaire.
Confirmation of these two hypotheses would be considered evidence for the commonalities among generic multi-item dPROMs that measure oral disease impact.
Results
Reliability of dental expert ratings
Overall, temporal stability of the questionnaire to OHRQoL dimension assignment was “excellent” according to Fleiss’ guidelines [11]. This was indicated by a median reliability coefficient of 0.86 for all dimensions and all dental experts combined. For three of the four dimensions, the median reliability coefficient exceeded the 0.75 threshold, considered “excellent” reliability, indicative of temporal stability. For the fourth dimension Orofacial Pain the coefficient was 0.74 (Fig 1).
The dispersion of the reliability coefficients varied across the four dimensions. More variable responses were observed for the dimension Psychosocial Impact compared to the other dimensions Oral Function, Orofacial Pain, and Orofacial Appearance. One ICC estimate (0.20) was an outlier compared to 27 estimates that ranged from 0.56 to 0.99. A Cronbach’s alpha of 0.95 was calculated for all the dental experts’ ratings, and an estimate of 0.90 for all the dimensions. These findings indicated internal consistency sufficient for “clinical application.” [12].
Validity of dental expert ratings
For all dimensions combined and for each of the four dimensions separately, a dominant general factor of uni-dimensionality among the underlying dental expert ratings was present (Fig 2). The differences among dimensions were small. The four plots showed only one eigenvalue exceeded the dashed line of the random eigenvalues, thus illustrating the uni-dimensionality of each dimension. The validity assessment findings indicated that the dental experts reached a consensus on the topic.
Questionnaires to OHRQoL dimensions assignment
- Hypothesis 1—All 20 dPRO questionnaires assess at least one OHRQoL dimension
Seven of the 20 dROMs measured a single dimension, seven measured two dimensions, five measure three dimensions, and one instrument measured all four dimensions, confirming the hypothesis, found in Table 1.
”x” indicates a median expert rating ≥5 points and ≤2 points within the highest dimension rating. Empty cells indicate median expert ratings >2 points lower than the highest dimension rating.
- Hypothesis 2—All four OHRQoL dimensions are measured by at least one questionnaire.
Oral Function, Orofacial Pain, Orofacial Appearance, and Psychosocial Impact were measured by 14, 9, 4, and 13 questionnaires, respectively, confirming the hypothesis.
Confirmation of these two hypotheses was considered evidence for the existence of commonalities among the 20 questionnaires for adult dental patients, found in Table 1.
Discussion
An international panel of dental experts provided reliable and valid ratings so that a consensus could be reached in a Delphi study regarding how well 20 dental patient-reported outcome (dPRO) questionnaires fit the four dimensions of oral health-related quality of life (OHRQoL). Each questionnaire assessed at least one OHRQoL dimension, and each of the four dimensions was at least assessed by one questionnaire. Based on these results, it was concluded that multi-item dPROMs have the four OHRQoL dimensions Oral Function, Orofacial Pain, Orofacial Appearance, and Psychosocial Impact in common.
Comparison with literature
The commonalities among multi-item dPROMs assessing disease impact are fundamental for the meaning and scoring of questionnaires used to measure perceived oral health because they represent the major attributes measured by these instruments. While dimensionality for individual dPROMs has been studied, e.g., the Oral Esthetics Scale (OES) [36] the dimensionality of larger groups of dPROMs have not been investigated. Therefore, a direct comparison with similar studies is currently not possible.
However, previous studies which assigned names of dPROMs to the four OHRQoL dimensions are informative for interpretation of the current findings. A previous exploratory study, which investigated multi-item generic dPROMs, generated the research hypothesis for the present study [3]. In that study, the names for 53 generic dPROMs could all be assigned to the four OHRQoL dimensions. A second study which used the same methodology of relating dPROM names to the four dimensions through use of a different set of questionnaires yielded identical results [4]. In this study, all specific dPROMs were collected, i.e., instruments designed to measure impact for a specific oral disease and all (N = 102) disease-specific dPROMs could also be linked to the OHRQoL dimensions. A study investigating PROMs for pediatric dental patients came to similar conclusions [37]. In this systematic review, authors concluded the twelve pediatric dPROMs assessed included items that could be mapped to the four OHRQoL dimensions [37].
From these studies it can be concluded that, although oral disease-specific and generic dPROMs may target a different set of oral diseases in different dental patient populations like adult and pediatric, there is no difference in what they have in common as measurement targets—the four OHRQoL dimensions.
Because dPROMs capture what matters to patients, their common measurement targets should be related to the reasons why patients seek treatment for their oral health problems. An international study conducted in all six World Health Organization health regions corroborated this [37]. This study found that Oral Function, Orofacial Pain, Orofacial Appearance, and Psychosocial Impact capture dental patients’ perceived impact of oral health problems worldwide, regardless of whether the patient currently suffers from oral diseases or intends to prevent them in the future [3]. While these findings applied to all dental patients, a subsequent analysis restricted to pediatric dental patients confirmed these results [37].
Although OHRQoL is only one among many concepts in the larger group of dPROs, it is the most important concept [1]. Therefore, comparing our dimensionality findings for dPROMs with those of OHRQoL instrument could provide deeper insight into dPRO dimensions. Globally, the OHIP is the most widely applied OHRQoL instrument [5]. As the most extensive OHRQoL instrument and being a special case of a dPROM, therefore it should measure the same four dimensions. Indeed, OHIP’s four dimensions Oral Function, Orofacial Pain, Orofacial Appearance, and Psychosocial Impact were studied with several methodological approaches and all agreed on OHIP’s four dimensions [38].
Strengths and limitations
Methodological pros and cons of this study are centered around the basic components of the Delphi study—the expert panel, the investigated dPROMs, and methods used to achieve and demonstrate consensus.
In this present study, a small convenience sample of dental experts was selected—an approach typically adopted in Delphi studies. The participating dental experts may not represent all dentists worldwide.
Although specific dPROMs that deal with the specific aspects of oral diseases also exist, this study focused on the generic dPROMs because, conceptually, they are applicable to all oral diseases. As we studied dPROMs for adults, results may only apply to this age group. Nonetheless, the results may be applicable to older children because, the dPROMs for these children measure the same major attributes as determine by adult dPROMs.
To assess whether consensus on the commonalities among the dPROMs and OHRQoL dimensions was achieved—a necessary condition to have interpretable results—we used a method that is often applied to investigate the dimensionality of PROMs [39]. This study found strong evidence that dental experts’ ratings on the commonalities of dPROMs were unidimensional, indicating that a consensus among dental experts was indeed achieved and that the Delphi study results are interpretable.
Conclusion
Oral Function, Orofacial Pain, Orofacial Appearance, and Psychosocial Impact were solidly identified as the commonalities among generic dPROMs, supporting them as the building blocks of the dental patient’s oral health experience. These findings pave the way towards a four-dimensional metric to assess oral disease impact that is essential for pragmatic implementation oral disease outcome measurement to achieve patient-centered and evidence-based clinical decision-making.
References
- 1. Paulson DR, Pattanaik S, Chanthavisouk P, John MT. Including the patient’s oral health perspective in evidence-based decision-making. The Journal of Health Monitoring [published online ahead of print]. 2021 Jul 9. pmid:34244813
- 2. Afrashtehfar KI, Assery MKA, Bryant SR. Patient satisfaction in medicine and dentistry. Int J Dent. 2020;2020. pmid:33456467
- 3. Mittal H, John MT, Sekulić S, Theis-Mahon N, Rener-Sitar K. Patient-reported outcome measures for adult dental patients: a systematic review. J Evid Based Dent Pract. 2019;19(1):53–70. pmid:30926102
- 4. Rener-Sitar K, John MT, Truong V, Tambe S, Theis-Mahon N. Nonmalignant oral disease—specific dental patient-reported outcome measures for adult patients: a systematic review. J Evid Based Dent Pract. 2021;21(1):101529. pmid:34051957
- 5. John MT, Hujoel P, Miglioretti DL, LeResche L, Koepsell TD, Micheelis W. Dimensions of oral-health-related quality of life. J Dent Res. 2004 Dec;83(12):956–60. pmid:15557405
- 6. John MT, Sekulić S, Bekes K, Al-Harthy MH, Michelotti A, Reissmann DR, et al. Why patients visit dentists—A study in all World Health Organization regions. J Evid Based Dent Pract. 2020;20(3):1–12.
- 7. John MT, Reißmann DR, Dent M, Feuerstahler L, Waller N, Baba K. Factor analyses of the oral health impact profile—Overview and studied population. J Prosthodont Res. 2014;57(3):1–9.
- 8. John MT, Reissmann DR, Feuerstahler L, Waller N, Baba K, Larsson P, et al. Exploratory factor analysis of the oral health impact profile. J Oral Rehabil. 2014;41(9):635–43. pmid:24909881
- 9. John MT, Feuerstahler L, Waller N, Baba K, Larsson P, Čelebić A, et al. Confirmatory factor analysis of the oral health impact profile. J Oral Rehabil. 2014;41(9):644–52. pmid:24909797
- 10. John MT, Rener-Sitar K, Baba K, Celebic A, Larsson P, Szabo G, et al. Patterns of impaired oral health-related quality of life dimensions. J Oral Rehabil. 2016 Jul;43(7):519–27. pmid:27027734
- 11.
Fleiss JL. The Design and Analysis of Clinical Experiments. New York: John Wiley & Sons; 1986.
- 12. Bland JM, Altman DG. Statistics notes: Cronbach’s alpha. BMJ. 1997 Feb;314(7080):572. pmid:9055718
- 13. Cella D, Yount S, Rothrock N, Gershon R, Cook K, Reeve B, et al. The patient-reported outcomes measurement information system (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007 May;45(5 Suppl 1):S3–S11. pmid:17443116.
- 14. Horn JL. A rationale and test for the number of factors in factor analysis. Psychometrika. 1965;30(2):179–85. pmid:14306381
- 15. Cattell RB. The scree test for the number of factors. Multivariate Behav Res. 1966;1(2):245–76. pmid:26828106
- 16. Klages U, Claus N, Wehrbein H, Zentner A. Development of a questionnaire for assessment of the psychosocial impact of dental aesthetics in young adults. Eur J Orthod. 2006 Apr;28(2):103–11. pmid:16257989
- 17. Gooch BF, Dolan TA, Bourque LB. Correlates of self-reported dental health status upon enrollment in the Rand Health Insurance Experiment. J Dent Educ. 1989 Nov;53(11):629–37. pmid:2808877
- 18. Atchison KA, Dolan TA. Development of the Geriatric Oral Health Assessment Index. J Dent Educ. 1990 Nov;54(11):680–7. pmid:2229624
- 19. Dworkin SF, LeResche L. Research diagnostic criteria for temporomandibular disorders: review, criteria, examinations and specifications, critique. J Craniomandib Disord. 1992;6(4):301–55. pmid:1298767
- 20. Strauss RP, Hunt RJ. Understanding the value of teeth to older adults: influences on the quality of life. J Am Dent Assoc. 1993 Jan;124(1):105–10. pmid:8445136
- 21. Stegenga B, de Bont LG, de Leeuw R, Boering G. Assessment of mandibular function impairment associated with temporomandibular joint osteoarthrosis and internal derangement. J Orofac Pain. 1993;7(2):183–95. pmid:8358365
- 22. Slade GD, Spencer AJ. Development and evaluation of the Oral Health Impact profile. Community Dent Heal. 1994;11:3–11. pmid:8193981
- 23. Locker D, Miller Y. Evaluation of subjective oral health status indicators. J Public Health Dent. 1994;54(3):167–76. pmid:7932353
- 24. Leao A, Sheiham A. The development of a socio-dental measure of dental impacts on daily living. Community Dent Health. 1996 Mar;13(1):22–6. pmid:8634892
- 25.
Cornell JE, Saunders MJ, Paunovich ED, Frisch MB. Oral health quality of life inventory (OH-QoL). In: Slade Gary D, editor. Measuring oral health and quality of life. 1997. p. 136–49.
- 26.
Adulyanon S, Sheiham A, Slade GD. Oral Impacts on Daily Performances. In: Slade GD, editor. Measuring oral health and quality of life. Chapel Hill: University of North Carolina, Department of Dental Ecology; 1997. p. 151–60. (Measuring Oral Health and Quality of Life).
- 27. McGrath C, Bedi R. An evaluation of a new measure of oral health related quality of life—OHQoL-UK(W). Community Dent Health. 2001 Sep;18(3):138–43. pmid:11580088
- 28. Aggarwal VR, Lunt M, Zakrzewska JM, Macfarlane GJ, Macfarlane TV. Development and validation of the Manchester orofacial pain disability scale. Community Dent Oral Epidemiol. 2005 Apr;33(2):141–9. pmid:15725177
- 29. Ohrbach R, Larsson P, List T. The jaw functional limitation scale: development, reliability, and validity of 8-item and 20-item versions. J Orofac Pain. 2008;22(3):219–30. pmid:18780535
- 30. Baba K, John MT, Inukai M, Aridome K, Igarahsi Y. Validating an alternate version of the chewing function questionnaire in partially dentate patients. BMC Oral Health. 2009 Mar;9(1):9. pmid:19291293
- 31. Nixdorf DR, John MT, Wall MM, Fricton JR, Schiffman EL. Psychometric properties of the modified Symptom Severity Index (SSI). J Oral Rehabil. 2010 Jan;37(1):11–20. pmid:19889036
- 32. Larsson P, John MT, Nilner K, Bondemark L, List T. Development of an orofacial esthetic scale in prosthodontic patients. Int J Prosthodont. 2010;23(3):249–56. pmid:20552092
- 33. Lee JY, Chen HI, Urban C, Hojat A, Church E, Xie SX, et al. Development of and psychometric testing for the brief pain inventory-facial in patients with facial pain syndromes. J Neurosurg. 2010 Sep;113(3):516–23. pmid:20151778
- 34. Peršić S, Palac A, Bunjevac T, Celebić A. Development of a new chewing function questionnaire for assessment of a self-perceived chewing function. Community Dent Oral Epidemiol. 2013 Dec;41(6):565–73. pmid:23551089
- 35. La Touche R, Pardo-Montero J, Gil-Martínez A, Paris-Alemany A, Angulo-Díaz-Parreño S, Suárez-Falcón JC, et al. Craniofacial pain and disability inventory (CF-PDI): development and psychometric validation of a new questionnaire. Pain Physician. 2014;17(1):95–108. pmid:24452650
- 36. Reissmann DR, John MT, Enstad CJ, Lenton PA, Sierwald I. Measuring patients’ orofacial appearance: validity and reliability of the English-language orofacial esthetic scale. J Am Dent Assoc [Internet]. 2019;150(4):278–86. Available from: pmid:30922458
- 37. Bekes K, John MT, Rener-Sitar K, Al-Harthy MH, Michelotti A, Reissmann DR, et al. Pediatric patients’ reasons for visiting dentists in all WHO regions. Health Qual Life Outcomes. 2021;19(1):1–12. pmid:34120623
- 38. John MT. Foundations of oral health-related quality of life. J Oral Rehabil. 2020;(October 2019):355–9. pmid:32658317
- 39. Fink-Hafner D, Dagen T, Doušak M, Novak M, Hafner-Fink M. Delphi Method: Strengths and Weaknesses. Metod Zv [Internet]. 2019;16(2):1–19. Available from: https://www.researchgate.net/publication/337570516.