http://orcid.org/0000-0001-5254-0314
Figures
Abstract
Objective
This study aimed to administer a Delphi panel survey and provide evidence for the development of a psychological intervention protocol for use after disasters in South Korea.
Method
A three-round Delphi survey was conducted. In all rounds, respondents answered open- or closed-ended questions regarding their views on i) the concept of disaster, ii) evaluation, iii) intervention, and iv) considerations in a disaster. Data from Round 1 were subjected to content analysis. In Round 2, items with content validity ratios (CVRs) greater than 0.49 were included, and in Round 3, items with a CVR≥0.38 were accepted.
Results
The response rates for the Delphi survey were high: 83% (n = 15, Round 1), 80% (n = 16, Round 2), and 86% (n = 24, Round 3). The data collected during this survey showed a need for a support system for children; for preventive strategies, including disaster readiness plans; for the protection of children's safety; and for the development of post-disaster psychosocial care.
Conclusions
The panel experts reached a consensus regarding the steps they considered critical in post-disaster evaluation and intervention. The findings suggest a unified model for advancing the development of the Korean version of an intervention protocol for children and adolescents exposed to traumatic events.
Citation: Lee M-S, Hwang J-W, Lee C-S, Kim J-Y, Lee J-H, Kim E, et al. (2018) Development of post-disaster psychosocial evaluation and intervention for children: Results of a South Korean delphi panel survey. PLoS ONE 13(3): e0195235. https://doi.org/10.1371/journal.pone.0195235
Editor: Jeremy C. Kane, Johns Hopkins University Bloomberg School of Public Health, UNITED STATES
Received: June 3, 2017; Accepted: February 26, 2018; Published: March 29, 2018
Copyright: © 2018 Lee et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This study was supported by a grant from the Korean Mental Health Technology R&D Project, Ministry of Health & Welfare, Republic of Korea (HM15C1058). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Natural and man-made disasters are common worldwide. Various disasters have occurred in South Korea in the twenty-first century, such as typhoons, floods, and subway fires. Among them is the sinking of the Sewol ferry, which occurred on the morning of April 16, 2014. The ferry capsized while carrying 476 people, mostly secondary school students from Danwon High School. In total, 304 passengers and crew members died in the disaster. The Sewol ferry disaster severely shocked Korean society and resulted in widespread social and political reactions in South Korea.
Traumatic symptoms in children and adolescents are expressed in a variety of forms depending on their developmental stage. Children can develop PTSD (Post-Traumatic Stress Disorder) and other mental health problems following traumatic events.[1] Moreover, a significant minority of children who are particularly vulnerable have ongoing difficulties.[2] Compared with studies of adult samples, studies of youth outcomes after a disaster generally report higher estimates for the prevalence of mental health disorders.[3] Therefore, to help children and adolescents, it is very important to evaluate and intervene in situations of psychological trauma.
In South Korea, before April 16, 2014, there were no efforts to prepare the population for coping with disaster. Systematic psychological intervention guides for disaster situations have never been provided.
We searched through guidelines such as the WHO guidelines[4], the Mental Health Gap Action Programme (mhGAP) Humanitarian Intervention Guide[5] and recommendations by the Inter-Agency Standing Committee (IASC)[6]. However, the use of available practical guidelines for disaster and trauma patients might be limited due to cultural differences in medical situations and clinical environments. Therefore, protocols that can more aptly respond to culturally specific situations and issues in South Korea are required.[7] The country has suffered from a lack of crisis intervention approaches to follow after disasters. For these reasons, confusion arose when the sinking of MV Sewol occurred on April 16, 2014. Therefore, we seek to study and suggest practical directions for establishing guidelines in South Korea.
In this regard, a Delphi study for disaster care is necessary. The Delphi methodology is a widely used group survey technique typically conducted in three consecutive rounds to evaluate consensus among experts in a field. The quality of the panel of experts and their opinions on the given topic is considered a strength of the Delphi technique.[8] The approach has the advantage of obtaining expert opinion with a guarantee of anonymity, thus avoiding potential distortion caused by peer pressure in group situations such as focus group analysis.[9] Above all, this technique is most effective when there is a lack of information or only inadequate information on a particular issue.
In this context, it is particularly important to monitor the psychosocial care guidelines for children after a disaster. However, to our knowledge, no researchers have examined expert opinion via a Delphi study in post-disaster situations in South Korea.
This survey details the design of a Delphi study for addressing appropriate psychosocial care guidelines for children and adolescents after a disaster. The agreed-upon measures could constitute a standardized approach to initial clinical evaluation and intervention to help identify individuals in need after a disaster.[10] A three-round Delphi study was undertaken to elicit a prioritized list of research topics to guide future research efforts and thus obtain meaningful results.[11] Consequently, using the Delphi survey technique, this study aimed to evaluate the usefulness and direction of the development of post-traumatic assessment and intervention based on the opinions of pediatric and disaster- and trauma-related experts.
Methods
The Delphi study consisted of three consultation rounds from January to May 2016. In each Delphi round, we provided the panel with feedback on the results of the previous consultation, and routine communications with panel experts were conducted by e-mail. The study was approved by Eulji University's Institutional Review Board (IRB No. EMCS 2015-12-004).
Delphi study
A Delphi study is a structured process that invites experts to complete a series of ‘rounds’ to gather and refine information related to the study question until an expert consensus is reached.[12] A commonly used formal consensus method is the Delphi technique, which involves two or more rounds of postal or online questionnaires.[13]
According to previous studies, two or three rounds are frequently used in the Delphi process.[12] The survey rounds interactively ask experts to prioritize issues or rate them on implementation-related scales, such as scales measuring feasibility or desirability, enabling controlled feedback on the previous round’s group results.[14] This group facilitation technique aims to obtain consensus among the opinions of ‘experts’ through a series of structured questionnaires.[15]
Delphi panel
A Delphi study is conducted with a group of individuals considered to have expertise (both professional and experience-based) in the field under investigation.[16] The Delphi panel in this study consisted of experts in child and adolescent mental health, professionals providing disaster psychological support, and related practitioners with experience in disasters. Our survey included a range of mental health professionals.[13]
The Delphi technique allows for the selection of experts and does not require a representative sample of the population. We note that the literature on Delphi surveys traditionally recommends a panel of 10 to 15 experts, typical of most qualitative research.[17] However, a panel size ranging from 20 to 50 has been deemed appropriate.[18] Therefore, the present study is informed by recommendations of a sample size from 10 to 50 for qualitative research and Delphi surveys designed to generate hypotheses.[19]
The Delphi panel participants were also required to provide basic demographic information and professional characteristics.[20] Anonymity was assured for all participants during the study; anonymity prevents the influence of the authority, status, personality, or reputation of group members in the process, thereby preventing biased outcomes.[21]
First-round questionnaire
The Round 1 survey consisted of 20 open-ended questions grouped into four themes (S1 Appendix). Several open-ended questions were included to ensure that the survey accommodated the opinions of professionals from a multidisciplinary team. After confirming participation, panel participants were e-mailed an invitation to activate the Round 1 questionnaire. We conducted the online interview and received informed consent from all participants on the expert panel before interviewing them. The responses had no word limits, and participants were encouraged to give their opinions freely. Reminders were sent if the survey had not been returned. The survey was open for one month.
Second-round questionnaire
Questions for Rounds 2 were developed based on the participants' responses in the previous round. Converged answers in Round 1 were classified as evaluation and intervention, and freely presented expert opinions were based on detailed questions. The Round 2 survey consisted of 156 closed-ended questions with responses grouped into 27 themes. The experts received the second-round questionnaire by e-mail and were instructed to rate and score the importance of each indicator on a five-point Likert scale (1 = very unimportant, 3 = neutral and 5 = very important). An item was considered important if ≥80% of the respondents awarded it a score of 4 or 5; otherwise, the item was removed. The experts were encouraged to provide comments freely on each indicator and/or to propose indicators that they considered important. Routine communication with panel experts was conducted by e-mail.
Third-round questionnaire
Round 3 excluded 44 items that did not receive a consensus in Round 2. For 112 items, 80% agreement was reached. In Round 2, the experts freely commented on each indicator that they considered important. Based on these responses, 11 items were modified, and 63 items were added.
Ultimately, 175 items were composed and grouped into 25 themes. In the third round, we asked the panel to rate the importance of each topic on a 5-point Likert scale from 1 (not important) to 5 (very important). The level of consensus was set to 80% of respondents indicating agreement.[9] Individual and anonymous opinions were solicited via e-mail.
Data analysis
Delphi questionnaires were coded individually. Members of the research team alone had access to the codes to facilitate follow-up. Any published data identified individuals, their institution, or organizations.
In Round 1, all topics suggested by the panel experts were categorized using content analysis. We identified words or expressions in conceptual categories to understand and identify the relationships among themes. We performed categorization by removing irrelevant, overlapping and repeated content; looking for common viewpoints; and identifying responses. To analyze the Round 2 and 3 responses, we calculated content validity ratios (CVRs). The minimum CVR was determined by the number of experts participating in each round.
We used the formula CVR = (ne -N/2)/ (N/2), where ne represents the number of panel experts rating an item as ‘essential’ (score of 4 or 5) and N represents the entire number of panelists.[22] The CVR ranges from +1 to −1. A high positive value indicates that the survey experts agreed that a factor or item was essential.[23]
Therefore, in Round 2, the CVR values of all items were set to 0.49 for the 16 panels. Additionally, in Round 3, the minimum CVR value was set to 0.38 for the 24 panels.
Results
Demographics of the panel experts
The demographic characteristics of the experts are described in Table 1.
In Round 1, 18 experts registered to be members of the Delphi panel, and 15 of them (83%) (10 female, 5 males) returned the Round 1 questionnaire. The mean age of the experts was 44.07 years (standard deviation: 6.84 years). Approximately 10 (66.67%) of the respondents had earned a Ph.D.
In Round 2, 20 participants were included, and 16 (80%) responded; the respondents included psychiatrists (10), psychologists (5), and a social worker (1). The mean age of the experts was 43.75 years (standard deviation: 7.14 years). Approximately 11 (68.75%) of the panel experts were women, and 9 (56.25%) had earned a Ph.D. as their highest level of education.
In Round 3, 28 psychiatric professionals registered to be members of the expert panel, and 24 (86%) returned the questionnaires. The mean age of the experts was 43.83 years (standard deviation: 8.33 years); the experts included psychiatrists (17), psychologists (5), and social workers (2). Most of the experts were females (19), and 15 (62.50%) had earned a Ph.D. Round 3 experts showed an adequate level of agreement on the research topics (Table 1).
Results of first-round Delphi survey
Qualitative content analysis was used in Round 1. The Round 1 results are described in detail in a previously published paper.[24] We found that the following issues have a strong effect on post-disaster interventions: proper timing of the initial interview in the event of a disaster, assessment notification, assessment services for individuals, mandatory enforcement measures, scale screening and treatment intervention elements, symptom degree classification, intervention standardization, program level, care unit environments, and operation plans.
The table in the preliminary research paper that included the Round 1 items and content has been reproduced. We sought permission from previous journals to re-use the table and to add a reference (Table 2).
Results of second-round Delphi survey
The categories and items on the Delphi panel survey are described in Table 3.
Tables 4 and 5 show the evaluation items and intervention items, respectively, for Round 2.
In the conceptual and semantic domain of trauma in children and adolescents, the CVR was 0.49 or higher, and the content validity was verified for all items. The average value and the CVR were the highest in the ‘self-report’ and ‘teacher-report’ assessments. In contrast, the CVR for ‘the importance of evaluating an acquaintance (or a friend) of victims from the disaster’ was less than 0.49 (Table 4).
The screening questionnaire items ‘necessary to meet a family member at the time of screening’ and ‘caution when interviewing children and adolescents’ were validated. The CVR was the highest for ‘trauma, depression, anxiety, suicide, physical symptoms, social support, adaptation, and mood response should be included in the screening test’. Nevertheless, the CVR was less than 0.49 for ‘20 minutes of screening time is needed’ and ‘children's developmental considerations must be considered’. Therefore, the items with low CVRs were excluded in the third round, and supplementary items were developed (Table 4).
In the high-risk group, the CVR was highest for ‘child, adolescent, family, teacher evaluation’. However, the CVR for the item ‘It takes about one hour to interview the high-risk group’ was less than 0.49. Based on an additional comment from the expert panels, it was decided that the third round should include ‘30 minutes to 1 hour is most appropriate when evaluating a high-risk group’. In addition, many opinions suggested that ‘they should evaluate trauma, depression, anxiety, suicide, and social support’. However, the item ‘intelligence, projection test, and neuropsychological evaluation are necessary’ was excluded from the third round because the CVR was less than 0.49 (Table 4).
In addition, the CVR was lower than 0.49 for ‘the number of program sessions is “5 to 8 sessions”, “9 to 12 sessions”, and “13 sessions or more” is required’ if the intervention program is implemented after a disaster. The CVR was also low for ‘the treatment was terminated if the child had recovered the level of functioning’. These items should be excluded because of CVR validity; however, we revised those items based on additional comments from the experts, and the revised items were used in the third round (Table 5).
The CVR for the ‘need for standardized PFA (psychological first aid) and TRT (teaching recovery techniques)’ for the Korean version for infants and children was higher than 0.49. However, the CVRs for ‘SSET (support for students exposed to trauma), TF-CBT (trauma-focused cognitive behavior therapy), EMDR (eye movement desensitization and processing), PE (prolonged exposure therapy), trauma-focused play therapy and art therapy’ were low. In this case, the opinion of experts on Korean culture was reflected in the third round. However, the need for the Korean version of the PFA, TRT, SSET, TF-CBT, and EMDR was associated with a CVR higher than 0.49 (Table 5).
Results of third-round Delphi survey
The evaluation items and intervention items for Round 3 are described in detail in Tables 6 and 7, respectively.
The CVR for Round 3 was 0.38 or higher, and the content validity was verified for nearly all items. The major items with high CVRs are described as follows.
The CVRs were higher than 0.38 for the following items: ‘children and adolescents experiencing trauma should adjust to their current life to recover from trauma’, ‘stabilize their social and interpersonal functions’, and ‘fulfill their developmental tasks in the long term’ (Table 6).
In particular, in the high-risk group, the average value and the content CVR were the highest for the item ‘the child and the family should be evaluated’. The highest CVR was observed for the opinion that a trauma-related scale and scales for depression, anxiety, suicide, sleep, and social resources are needed. The CVR was 0.38 or higher for the items indicating that specialists who perform a psychological assessment in a disaster need ‘crisis management training’ and the ‘ability to cope with various responses of clients’ (Table 6).
In terms of the intervention program, the CVR was the highest for ‘psychological education for the post-traumatic response, normalization, stabilization, physical stability training, family and teacher education, and emotion education should be included.’ With respect to the elements of a therapy program, a high CVR was observed for ‘requiring PFA, TRT, SSET, TF-CBT, EMDR, PE, a family participation program, a mourning-themed program, individual psychotherapy and medication’. Opinions suggesting that ‘individual psychotherapy and medication are needed’ were most frequently observed. In addition, some comments indicated that ‘child-parent psychotherapy might be more appropriate than PFA and TRT for toddlers and preschoolers’ (Table 7).
With respect to the termination of therapy, CVRs higher than 0.38 were observed for the following items: ‘the intervention should be terminated after the prescribed therapy sessions’ and ‘referrals should be determined thereafter’ (Table 7).
A high CVR was found for the item regarding the intervention development strategy: 'establish a therapeutic linkage system based on national support, educate and inform the whole school, support medical expenses (such as with government subsidies), connect with the community, consider the persistence of treatment cases, and reduce the stigma of psychiatry’ (Table 7).
Discussion
In South Korea, the dispute over how to evaluate and intervene in the aftermath of the Sewol Ferry Disaster required a consensus regarding the need for disaster planning.[24] The Delphi process was a suitable method for surveying experts on this topic.[25] Using this method, we propose a multidisciplinary recommendation for treating children exposed to disasters. The results of qualitative and quantitative analyses conducted through the Delphi panel survey demonstrate that psychosocial assessment and intervention are essential to early mental health services following a disaster. We discuss suggestions based on the consensus of the experts involved in the study.
We found that in the event of a disaster, intervention factors such as ‘appropriate time for assessment after the disaster’, ‘prerequisites for screening and in-depth intervention’, ‘classifying the degree of psychosocial symptoms’, and ‘social and mental health services’ are very important. Recovery from psychological trauma after a disaster means mental stability as well as the recovery of physical health. Screening tests are recommended for all children exposed to disasters, particularly during acute periods of disaster. After the completion of screening tests, assessment should include in-depth interviews and interventions for the high-risk group. First, however, we must distinguish between brief screening and in-depth evaluation. As in our study, many previous studies have suggested mental health assessments and interventions for children.[26] These findings are consistent with research findings indicating that screening is appropriate when large numbers of children are exposed to an event or when the level of exposure among a population is unknown.[27]
The actual screening assessment performed after a disaster requires the consideration of each stage of the disaster and should consist of appropriate questions.[28] In the disaster context, screening tools should reflect the needs of children with mental health problems, including consideration of children’s exposure, experience, and subjective reactions to traumatic events and conditions.[29, 30]
Evaluation of children, families, and teachers during the acute phase of a disaster is important. Above all, consensus among experts on the selection of children exposed to a disaster is required. Families and teachers should be evaluated together. The use of multiple informants, such as parents, teachers, and other professionals, as collateral sources of information enables the most comprehensive appraisal of children’s reactions and functioning.[28] These results are consistent with the opinion that it is important for parents and/or caregivers to participate together in a child's treatment session to recover from PTSD symptoms.[26] When interviewing a family member, we must check for signs of psychological crisis among family members. This finding is consistent with studies of the family environment, social support, and supportive quality.[31] However, it is not necessary to evaluate acquaintances or friends. Furthermore, assessments of grief, depression, anxiety, and suicide risk, as well as trauma-related scales, need to measure PTSD and other psychosocial symptoms. This finding is largely consistent with a previous report that disaster exposure is correlated with PTSD, depression, anxiety, functional impairment, and behavioral problems.[32] In addition, trauma assessment of children and adolescents should consider their developmental stage. When treating a child who has experienced trauma, the clinician must understand the child's existing psychopathological symptoms and provide appropriate interventions, such as trauma-focused therapy.[26] Our results suggest the need to develop a crisis intervention model for children and adolescents.[33]
Psychosocial assessments should be conducted in a safe environment and at appropriate durations of 30–60 minutes. Approximately 30 to 60 minutes is needed for screening a high-risk group.
Psychoeducation is also beneficial to children. A post-disaster intervention program should include the following: psychoeducation, guidelines for coping with the media, normalization, stabilization, techniques for handling survivor’s guilt and emotion-focused coping strategies. Appropriate access phases can be classified as hyper-acute, acute, sub-acute or chronic stages. Stabilization and psychological support should be provided immediately after a disaster along with intervention to help children adapt to everyday life. This finding is consistent with a report that most interventions are multimodal, incorporating common elements to educate children, normalize their reactions, process their emotions and manage stress, enhance coping and provide social support.[27] In addition, the development stage, age, trauma symptoms, and traits of a group should be considered. The number of children participating in a group may vary depending on the type of disaster. In general, psychoeducation can be provided in the class setting at school. For prevention education, holding one to four sessions is recommended, whereas for therapeutic intervention, five to eight sessions are appropriate. If the child is exposed to a national large-scale disaster, intervention to address brief trauma may not be sufficient. Therefore, professional intervention should be provided, particularly for children with symptoms of PTSD.[26]
For a preschooler, the appropriate duration of an intervention is 30 minutes with caregiver participation. A proper duration of 30 to 40 minutes is suitable for elementary school students in lower grades. An intermediate duration of 40 minutes is suitable for elementary school students in higher grades. For middle and high school students, intervention programs could last 45 to 50 minutes. The optimal intervention components may not be the same for all children or all situations, which should be examined in future work.[34]
We recommend the following available intervention programs: PFA[35], TRT[36], SSET[37], and TF-CBT[38]. In South Korea, the South Korean versions of PFA, TRT, and TF-CBT should be standardized for children and adolescents. However, the study findings provided no suggestion related to narrative therapy. Furthermore, an intervention for toddlers and preschoolers should be considered. Multiple evidence-based programs should be considered as well, and an intervention protocol that includes a standardized South Korean version can then be implemented. These results provide a framework for further research. Accordingly, the CIDER (Children In Disaster: Evaluation & Recovery) protocol developed by the authors of this study will be made available. Additionally, we must include not only child-focused therapy but also long-term mental health services. These findings are partially consistent with a prior study.[39]
The professionals providing disaster interventions vary with respect to factors such as availability, training, and experience, and the goals and complexity of the intervention differ as well.[27] Nevertheless, affected communities do not have enough therapists trained in evidence-based treatments to be able to provide every child with individual therapy.[39] It is not necessary for all mental health workers to conduct evaluations and interventions after a disaster. Therefore, disaster experts with experience working in a clinical environment should be called upon; a training and education system for professionals is needed. Such professionals may need additional support and guidance to address their own emotional responses.[27] This support can be incorporated into supervision as well as peer support groups. Additionally, the present study shows that good relationships should be cultivated within professional networks of information related to in-depth therapy.
Above all, interventions delivered in groups are particularly well suited for school settings.[27] Schools are among the most important links in the chain of public health education for children and adolescents.[40] School-based interventions should be developed, regular training in disaster safety measures for school personnel should be mandated, and training programs for children should be established. Moreover, teachers should receive advice on coping with emergencies in either their basic teacher training or in-service training. In summary, schools should identify school crisis emergencies and clearly delineate the roles of children and teachers in coping with disaster. Based on the abovementioned considerations, psychiatric and psychological support should be accessible. Additionally, guiding children to use positive coping strategies and encouraging a warm community atmosphere are recommended.[32] Consequently, our confidence in reaching consensus means that we now have a comprehensive framework of competency statements that describe what psychiatric professionals working in the aftermath of a disaster must do. As the National Child Traumatic Stress Network has coordinated collaboration among 10 research development and evaluation sites and 26 community mental health centers across the United States, it is also essential to establish sensible governance between central and local governments, between administrative institutions and institutions that provide services, and between public and civic organizations.[41]
This study proposed effective mental health intervention measures and described the implications for developing a post-disaster evaluation treatment protocol. The main strengths of our study include its responses from a panel of defined experts, good response rates and framework of competencies that describe attributes of professionals working within the disaster field. However, some limitations also need to be recognized.
First, the study findings suggest that children in South Korean cultures require disaster-related psychosocial evaluation and interventions, but modifications may be needed to address other cultural issues.
Second, our expert panel was determined by our approach to sampling. E-mails may not have been distributed by some of the professional groups we contacted, and other experts not publishing their work may have been missed. The rich qualitative and quantitative data obtained from this study are very useful for understanding why certain topics are research priorities.[21]
Third, the experts who conducted psychological intervention at Danwon High School after the Sewol Ferry Disaster in South Korea were all psychiatrists, except for two psychologists.[42] The primary aim was to gather psychiatrists’ opinions and experience from the disaster environment. In Round 1, we had limitations in distinguishing between the related areas of expertise in disaster and trauma for the psychological specialists, and these limitations might be reflected in the medical opinions of the panel.
In conclusion, we suggest the need for informed evidence-based assessments, interventions, and treatments for children and adolescents who experience disasters. This survey presents important opinions from trauma care experts and should be utilized by psychiatrists to develop a meaningful protocol for PTSD assessment and treatment. Hence, the results can be applied to existing and future disaster management.
Supporting information
S1 Appendix. The specific 20 questions in Round 1.
https://doi.org/10.1371/journal.pone.0195235.s001
(DOCX)
Acknowledgments
We would like to give our heartfelt thanks to all the panel experts who participated in this study.
References
- 1. Bolton D, O'Ryan D, Udwin O, Boyle S, Yule W. The long-term psychological effects of a disaster experienced in adolescence: II: General psychopathology. Journal of child psychology and psychiatry, and allied disciplines. 2000;41(4):513–23. pmid:10836681.
- 2. Cohen JA, Jaycox LH, Walker DW, Mannarino AP, Langley AK, DuClos JL. Treating traumatized children after Hurricane Katrina: Project Fleur-de lis. Clinical child and family psychology review. 2009;12(1):55–64. pmid:19224365; PubMed Central PMCID: PMC2927211.
- 3. La Greca AM, Silverman WK, Lai B, Jaccard J. Hurricane-related exposure experiences and stressors, other life events, and social support: concurrent and prospective impact on children's persistent posttraumatic stress symptoms. Journal of consulting and clinical psychology. 2010;78(6):794–805. pmid:20939624.
- 4. Demyttenaere K, Bruffaerts R, Posada-Villa J, Gasquet I, Kovess V, Lepine JP, et al. Prevalence, severity, and unmet need for treatment of mental disorders in the World Health Organization World Mental Health Surveys. Jama. 2004;291(21):2581–90. pmid:15173149.
- 5. van Ommeren M, Hanna F, Weissbecker I, Ventevogel P. Mental health and psychosocial support in humanitarian emergencies. Eastern Mediterranean health journal = La revue de sante de la Mediterranee orientale = al-Majallah al-sihhiyah li-sharq al-mutawassit. 2015;21(7):498–502. pmid:26442890.
- 6. Barrett T, Fox J, Shriberg J, Aden K, Eich W, Wolfe A. The IASC Guidelines and the International Disaster Psychology Program at the University of Denver. Intervention. 2011;9(2):159–63.
- 7. Wang HR, Bahk WM, Park YM, Lee HB, Song HR, Jeong JH, et al. Korean medication algorithm for depressive disorder: comparisons with other treatment guidelines. Psychiatry Investig. 2014;11(1):1–11. pmid:24605117; PubMed Central PMCID: PMC3942544.
- 8. Powell C. The Delphi technique: myths and realities. Journal of advanced nursing. 2003;41(4):376–82. pmid:12581103.
- 9. Birko S, Dove ES, Ozdemir V. A Delphi Technology Foresight Study: Mapping Social Construction of Scientific Evidence on Metagenomics Tests for Water Safety. PloS one. 2015;10(6):e0129706. pmid:26066837; PubMed Central PMCID: PMC4465892.
- 10. Jackson KA, Glyn-Jones S, Batt ME, Arden NK, Newton JL, Delphi P. Assessing risk factors for early hip osteoarthritis in activity-related hip pain: a Delphi study. BMJ open. 2015;5(9):e007609. pmid:26419679; PubMed Central PMCID: PMC4593156.
- 11. Schoenly L. Research Priorities in Correctional Nursing Practice: Results of a Three-Round Delphi Study. Journal of correctional health care: the official journal of the National Commission on Correctional Health Care. 2015;21(4):400–7. pmid:26285595.
- 12. Maarsingh OR, Dros J, van Weert HC, Schellevis FG, Bindels PJ, van der Horst HE. Development of a diagnostic protocol for dizziness in elderly patients in general practice: a Delphi procedure. BMC family practice. 2009;10:12. pmid:19200395; PubMed Central PMCID: PMC2660288.
- 13. Taylor RM, Feltbower RG, Aslam N, Raine R, Whelan JS, Gibson F. Modified international e-Delphi survey to define healthcare professional competencies for working with teenagers and young adults with cancer. BMJ open. 2016;6(5):e011361. pmid:27142859; PubMed Central PMCID: PMC4861123.
- 14. Twycross A. Achieving consensus about pain content for child branch curricula. Journal of advanced nursing. 2001;34(1):51–60. pmid:11430606.
- 15. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. Journal of advanced nursing. 2000;32(4):1008–15. pmid:11095242.
- 16. Ager A, Stark L, Akesson B, Boothby N. Defining best practice in care and protection of children in crisis-affected settings: a Delphi study. Child development. 2010;81(4):1271–86. pmid:20636695.
- 17. Green B, Jones M, Hughes D, Williams A. Applying the Delphi technique in a study of GPs' information requirements. Health & social care in the community. 1999;7(3):198–205. pmid:11560634.
- 18. Aronson BD, Janke KK, Traynor AP. Investigating student pharmacist perceptions of professional engagement using a modified Delphi process. American journal of pharmaceutical education. 2012;76(7):125. pmid:23049097; PubMed Central PMCID: PMC3448463.
- 19. Birko S, Dove ES, Ozdemir V. Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation. PloS one. 2015;10(8):e0135162. pmid:26270647; PubMed Central PMCID: PMC4535950.
- 20. Kong LN, Guo Y, Qin B, Peng X, Zhu WF. Development of Self-Management Indicators for Chronic Hepatitis B Patients on Antiviral Therapy: Results of a Chinese Delphi Panel Survey. PloS one. 2015;10(9):e0134125. pmid:26327606; PubMed Central PMCID: PMC4556706.
- 21. van de Glind I, Berben S, Zeegers F, Poppen H, Hoogeveen M, Bolt I, et al. A national research agenda for pre-hospital emergency medical services in the Netherlands: a Delphi-study. Scandinavian journal of trauma, resuscitation and emergency medicine. 2016;24:2. pmid:26746873; PubMed Central PMCID: PMC4706720.
- 22. Lawshe CH. A quantitative approach to content validity. Personnel Psychology. 1975;28(4):563–75.
- 23. Wilson FR, Pan W, Schumsky DA. Recalculation of the critical values for Lawshe's content validity ratio. Measurement and Evaluation in Counseling and Development. 2012;45(3):197–210.
- 24. Park JH, Lee MS, Chang HY, Hwang JW, Lee JH, Kim JY, et al. The Major Elements of Psychological Assessment and Intervention for Children and Adolescents after a Disaster: A Professional Delphi Preliminary Survey. J Korean Acad Child Adolesc Psychiatry. 2016;27(3):164–72.
- 25. Colagiuri R, Boylan S, Morrice E. Research Priorities for NCD Prevention and Climate Change: An International Delphi Survey. International journal of environmental research and public health. 2015;12(10):12941–57. pmid:26501301; PubMed Central PMCID: PMC4627009.
- 26. Practice parameters for the assessment and treatment of children and adolescents with posttraumatic stress disorder. Journal of the American Academy of Child and Adolescent Psychiatry. 1998;37(10 Suppl):4S–26S. pmid:9785726.
- 27. Pfefferbaum B, Varma V, Nitiema P, Newman E. Universal preventive interventions for children in the context of disasters and terrorism. Child and adolescent psychiatric clinics of North America. 2014;23(2):363–82, ix-x. pmid:24656585; PubMed Central PMCID: PMC3964365.
- 28. Pfefferbaum B, Noffsinger MA, Sherrieb K, Norris FH. Framework for research on children's reactions to disasters and terrorist events. Prehospital and disaster medicine. 2012;27(6):567–76. pmid:23034149; PubMed Central PMCID: PMC3526372.
- 29.
Saylor C, Deroma V. Assessment of children and adolescents exposed to disaster. In: La Greca AM, Silverman WK, Vernberg EM, Roberts MC, editors. Helping Children Cope with Disasters and Terrorism. Washington, DC: American Psychological Association. 2002:35–53.
- 30. Ohan JL, Myers K, Collett BR. Ten-year review of rating scales. IV: scales assessing trauma and its effects. Journal of the American Academy of Child and Adolescent Psychiatry. 2002;41(12):1401–22. pmid:12447027.
- 31. Ayub M, Poongan I, Masood K, Gul H, Ali M, Farrukh A, et al. Psychological morbidity in children 18 months after Kashmir Earthquake of 2005. Child psychiatry and human development. 2012;43(3):323–36. pmid:22080367.
- 32. Dimitry L. A systematic review on the mental health of children and adolescents in areas of armed conflict in the Middle East. Child: care, health and development. 2012;38(2):153–61. pmid:21615769.
- 33. Adams ZW, Sumner JA, Danielson CK, McCauley JL, Resnick HS, Gros K, et al. Prevalence and predictors of PTSD and depression among adolescent victims of the Spring 2011 tornado outbreak. Journal of child psychology and psychiatry, and allied disciplines. 2014;55(9):1047–55. pmid:24580551; PubMed Central PMCID: PMC4133357.
- 34. Pfefferbaum B, Sweeton JL, Nitiema P, Noffsinger MA, Varma V, Nelson SD, et al. Child disaster mental health interventions: therapy components. Prehospital and disaster medicine. 2014;29(5):494–502. pmid:25225954; PubMed Central PMCID: PMC4405161.
- 35. Cain DS, Plummer CA, Fisher RM, Bankston TQ. Weathering the storm: Persistent effects and psychological first aid with children displaced by Katrina. 2010; 3:330–43. J Child Adolesc Trauma. 2010;3(4):330–43.
- 36. Diab M, Peltonen K, Qouta SR, Palosaari E, Punamaki RL. Effectiveness of psychosocial intervention enhancing resilience among war-affected children and the moderating role of family factors. Child abuse & neglect. 2015;40:24–35. pmid:25534065.
- 37. Jaycox LH, Langley AK, Stein BD, Wong M, Sharma P, Scott M, et al. Support for Students Exposed to Trauma: A Pilot Study. School mental health. 2009;1(2):49–60. pmid:20811511; PubMed Central PMCID: PMC2930829.
- 38. Lucio R, Nelson TL. Effective Practices in the Treatment of Trauma in Children and Adolescents: From Guidelines to Organizational Practices. Journal of evidence-informed social work. 2016;13(5):469–78. pmid:27104619.
- 39. Jaycox LH, Cohen JA, Mannarino AP, Walker DW, Langley AK, Gegenheimer KL, et al. Children's mental health care following Hurricane Katrina: a field trial of trauma-focused psychotherapies. Journal of traumatic stress. 2010;23(2):223–31. pmid:20419730; PubMed Central PMCID: PMC2860874.
- 40. Ostad Taghizadeh A, Mowafi H, Ardalan A. School fire in Iran: simple actions save lives. Annals of burns and fire disasters. 2013;26(1):44–7. pmid:23966899; PubMed Central PMCID: PMC3741008.
- 41. Yoon MK, Kim SY, Ko HS, Lee MS. System effectiveness of detection, brief intervention and refer to treatment for the people with post-traumatic emotional distress by MERS: a case report of community-based proactive intervention in South Korea. International journal of mental health systems. 2016;10:51. pmid:27504141; PubMed Central PMCID: PMC4976505.
- 42. Association TKN. The Korean Neuropsychiatric Association Newsletter. 54. 2014;8(30 August):6–7.