Development of post-disaster psychosocial evaluation and intervention for children: Results of a South Korean delphi panel survey

Objective This study aimed to administer a Delphi panel survey and provide evidence for the development of a psychological intervention protocol for use after disasters in South Korea. Method A three-round Delphi survey was conducted. In all rounds, respondents answered open- or closed-ended questions regarding their views on i) the concept of disaster, ii) evaluation, iii) intervention, and iv) considerations in a disaster. Data from Round 1 were subjected to content analysis. In Round 2, items with content validity ratios (CVRs) greater than 0.49 were included, and in Round 3, items with a CVR≥0.38 were accepted. Results The response rates for the Delphi survey were high: 83% (n = 15, Round 1), 80% (n = 16, Round 2), and 86% (n = 24, Round 3). The data collected during this survey showed a need for a support system for children; for preventive strategies, including disaster readiness plans; for the protection of children's safety; and for the development of post-disaster psychosocial care. Conclusions The panel experts reached a consensus regarding the steps they considered critical in post-disaster evaluation and intervention. The findings suggest a unified model for advancing the development of the Korean version of an intervention protocol for children and adolescents exposed to traumatic events.

Introduction Natural and man-made disasters are common worldwide. Various disasters have occurred in South Korea in the twenty-first century, such as typhoons, floods, and subway fires. Among them is the sinking of the Sewol ferry, which occurred on the morning of April 16, 2014. The ferry capsized while carrying 476 people, mostly secondary school students from Danwon High School. In total, 304 passengers and crew members died in the disaster. The Sewol ferry disaster severely shocked Korean society and resulted in widespread social and political reactions in South Korea.
Traumatic symptoms in children and adolescents are expressed in a variety of forms depending on their developmental stage. Children can develop PTSD (Post-Traumatic Stress Disorder) and other mental health problems following traumatic events. [1] Moreover, a significant minority of children who are particularly vulnerable have ongoing difficulties. [2] Compared with studies of adult samples, studies of youth outcomes after a disaster generally report higher estimates for the prevalence of mental health disorders. [3] Therefore, to help children and adolescents, it is very important to evaluate and intervene in situations of psychological trauma.
In South Korea, before April 16, 2014, there were no efforts to prepare the population for coping with disaster. Systematic psychological intervention guides for disaster situations have never been provided.
We searched through guidelines such as the WHO guidelines [4], the Mental Health Gap Action Programme (mhGAP) Humanitarian Intervention Guide [5] and recommendations by the Inter-Agency Standing Committee (IASC) [6]. However, the use of available practical guidelines for disaster and trauma patients might be limited due to cultural differences in medical situations and clinical environments. Therefore, protocols that can more aptly respond to culturally specific situations and issues in South Korea are required. [7] The country has suffered from a lack of crisis intervention approaches to follow after disasters. For these reasons, confusion arose when the sinking of MV Sewol occurred on April 16, 2014. Therefore, we seek to study and suggest practical directions for establishing guidelines in South Korea.
In this regard, a Delphi study for disaster care is necessary. The Delphi methodology is a widely used group survey technique typically conducted in three consecutive rounds to evaluate consensus among experts in a field. The quality of the panel of experts and their opinions on the given topic is considered a strength of the Delphi technique. [8] The approach has the advantage of obtaining expert opinion with a guarantee of anonymity, thus avoiding potential distortion caused by peer pressure in group situations such as focus group analysis. [9] Above all, this technique is most effective when there is a lack of information or only inadequate information on a particular issue.
In this context, it is particularly important to monitor the psychosocial care guidelines for children after a disaster. However, to our knowledge, no researchers have examined expert opinion via a Delphi study in post-disaster situations in South Korea.
This survey details the design of a Delphi study for addressing appropriate psychosocial care guidelines for children and adolescents after a disaster. The agreed-upon measures could constitute a standardized approach to initial clinical evaluation and intervention to help identify individuals in need after a disaster. [10] A three-round Delphi study was undertaken to elicit a prioritized list of research topics to guide future research efforts and thus obtain meaningful results. [11] Consequently, using the Delphi survey technique, this study aimed to evaluate the usefulness and direction of the development of post-traumatic assessment and intervention based on the opinions of pediatric and disaster-and trauma-related experts.

Methods
The Delphi study consisted of three consultation rounds from January to May 2016. In each Delphi round, we provided the panel with feedback on the results of the previous consultation, and routine communications with panel experts were conducted by e-mail. The study was approved by Eulji University's Institutional Review Board (IRB No. EMCS 2015-12-004).

Delphi study
A Delphi study is a structured process that invites experts to complete a series of 'rounds' to gather and refine information related to the study question until an expert consensus is reached. [12] A commonly used formal consensus method is the Delphi technique, which involves two or more rounds of postal or online questionnaires. [13] According to previous studies, two or three rounds are frequently used in the Delphi process. [12] The survey rounds interactively ask experts to prioritize issues or rate them on implementation-related scales, such as scales measuring feasibility or desirability, enabling controlled feedback on the previous round's group results. [14] This group facilitation technique aims to obtain consensus among the opinions of 'experts' through a series of structured questionnaires. [15] Delphi panel A Delphi study is conducted with a group of individuals considered to have expertise (both professional and experience-based) in the field under investigation. [16] The Delphi panel in this study consisted of experts in child and adolescent mental health, professionals providing disaster psychological support, and related practitioners with experience in disasters. Our survey included a range of mental health professionals. [13] The Delphi technique allows for the selection of experts and does not require a representative sample of the population. We note that the literature on Delphi surveys traditionally recommends a panel of 10 to 15 experts, typical of most qualitative research. [17] However, a panel size ranging from 20 to 50 has been deemed appropriate. [18] Therefore, the present study is informed by recommendations of a sample size from 10 to 50 for qualitative research and Delphi surveys designed to generate hypotheses. [19] The Delphi panel participants were also required to provide basic demographic information and professional characteristics. [20] Anonymity was assured for all participants during the study; anonymity prevents the influence of the authority, status, personality, or reputation of group members in the process, thereby preventing biased outcomes. [21] First-round questionnaire The Round 1 survey consisted of 20 open-ended questions grouped into four themes (S1 Appendix). Several open-ended questions were included to ensure that the survey accommodated the opinions of professionals from a multidisciplinary team. After confirming participation, panel participants were e-mailed an invitation to activate the Round 1 questionnaire. We conducted the online interview and received informed consent from all participants on the expert panel before interviewing them. The responses had no word limits, and participants were encouraged to give their opinions freely. Reminders were sent if the survey had not been returned. The survey was open for one month.

Second-round questionnaire
Questions for Rounds 2 were developed based on the participants' responses in the previous round. Converged answers in Round 1 were classified as evaluation and intervention, and freely presented expert opinions were based on detailed questions. The Round 2 survey consisted of 156 closed-ended questions with responses grouped into 27 themes. The experts received the second-round questionnaire by e-mail and were instructed to rate and score the importance of each indicator on a five-point Likert scale (1 = very unimportant, 3 = neutral and 5 = very important). An item was considered important if !80% of the respondents awarded it a score of 4 or 5; otherwise, the item was removed. The experts were encouraged to provide comments freely on each indicator and/or to propose indicators that they considered important. Routine communication with panel experts was conducted by e-mail.

Third-round questionnaire
Round 3 excluded 44 items that did not receive a consensus in Round 2. For 112 items, 80% agreement was reached. In Round 2, the experts freely commented on each indicator that they considered important. Based on these responses, 11 items were modified, and 63 items were added.
Ultimately, 175 items were composed and grouped into 25 themes. In the third round, we asked the panel to rate the importance of each topic on a 5-point Likert scale from 1 (not important) to 5 (very important). The level of consensus was set to 80% of respondents indicating agreement. [9] Individual and anonymous opinions were solicited via e-mail.

Data analysis
Delphi questionnaires were coded individually. Members of the research team alone had access to the codes to facilitate follow-up. Any published data identified individuals, their institution, or organizations.
In Round 1, all topics suggested by the panel experts were categorized using content analysis. We identified words or expressions in conceptual categories to understand and identify the relationships among themes. We performed categorization by removing irrelevant, overlapping and repeated content; looking for common viewpoints; and identifying responses. To analyze the Round 2 and 3 responses, we calculated content validity ratios (CVRs). The minimum CVR was determined by the number of experts participating in each round.
We used the formula CVR = (n e -N/2)/ (N/2), where n e represents the number of panel experts rating an item as 'essential' (score of 4 or 5) and N represents the entire number of panelists. [22] The CVR ranges from +1 to −1. A high positive value indicates that the survey experts agreed that a factor or item was essential. [23] Therefore, in Round 2, the CVR values of all items were set to 0.49 for the 16 panels. Additionally, in Round 3, the minimum CVR value was set to 0.38 for the 24 panels.
In Round 1, 18 experts registered to be members of the Delphi panel, and 15 of them (83%) (10 female, 5 males) returned the Round 1 questionnaire. The mean age of the experts was 44.07 years (standard deviation: 6.84 years). Approximately 10 (66.67%) of the respondents had earned a Ph.D.
In Round 2, 20 participants were included, and 16 (80%) responded; the respondents included psychiatrists (10), psychologists (5), and a social worker (1). The mean age of the experts was 43.75 years (standard deviation: 7.14 years). Approximately 11 (68.75%) of the panel experts were women, and 9 (56.25%) had earned a Ph.D. as their highest level of education.
In Round 3, 28 psychiatric professionals registered to be members of the expert panel, and 24 (86%) returned the questionnaires. The mean age of the experts was 43.83 years (standard deviation: 8.33 years); the experts included psychiatrists (17), psychologists (5), and social workers (2). Most of the experts were females (19), and 15 (62.50%) had earned a Ph.D. Round 3 experts showed an adequate level of agreement on the research topics ( Table 1).

Results of first-round Delphi survey
Qualitative content analysis was used in Round 1. The Round 1 results are described in detail in a previously published paper. [24] We found that the following issues have a strong effect on post-disaster interventions: proper timing of the initial interview in the event of a disaster, assessment notification, assessment services for individuals, mandatory enforcement measures, scale screening and treatment intervention elements, symptom degree classification, intervention standardization, program level, care unit environments, and operation plans.
The table in the preliminary research paper that included the Round 1 items and content has been reproduced. We sought permission from previous journals to re-use the table and to add a reference ( Table 2).

Results of second-round Delphi survey
The categories and items on the Delphi panel survey are described in Table 3.  Tables 4 and 5 show the evaluation items and intervention items, respectively, for Round 2.
In the conceptual and semantic domain of trauma in children and adolescents, the CVR was 0.49 or higher, and the content validity was verified for all items. The average value and the CVR were the highest in the 'self-report' and 'teacher-report' assessments. In contrast, the CVR for 'the importance of evaluating an acquaintance (or a friend) of victims from the disaster' was less than 0.49 ( Table 4).
The screening questionnaire items 'necessary to meet a family member at the time of screening' and 'caution when interviewing children and adolescents' were validated. The CVR was the highest for 'trauma, depression, anxiety, suicide, physical symptoms, social support, adaptation, and mood response should be included in the screening test'. Nevertheless, the CVR was less than 0.49 for '20 minutes of screening time is needed' and 'children's developmental considerations must be considered'. Therefore, the items with low CVRs were excluded in the third round, and supplementary items were developed (Table 4).
In the high-risk group, the CVR was highest for 'child, adolescent, family, teacher evaluation'. However, the CVR for the item 'It takes about one hour to interview the high-risk group' was less than 0.49. Based on an additional comment from the expert panels, it was decided that the third round should include '30 minutes to 1 hour is most appropriate when evaluating a high-risk group'. In addition, many opinions suggested that 'they should evaluate trauma, Post-disaster psychosocial evaluation and intervention for children Table 3. Categories and items of the second and third rounds of the Delphi study.

Categories Items Details
Concept of child-Adolescent trauma in disasters depression, anxiety, suicide, and social support'. However, the item 'intelligence, projection test, and neuropsychological evaluation are necessary' was excluded from the third round because the CVR was less than 0.49 (Table 4).

Categories Items Details
Child-adolescent treatment programs after disasters Post-disaster psychosocial evaluation and intervention for children In addition, the CVR was lower than 0.49 for 'the number of program sessions is "5 to 8 sessions", "9 to 12 sessions", and "13 sessions or more" is required' if the intervention program is implemented after a disaster. The CVR was also low for 'the treatment was terminated if the child had recovered the level of functioning'. These items should be excluded because of CVR validity; however, we revised those items based on additional comments from the experts, and the revised items were used in the third round ( Table 5).
The CVR for the 'need for standardized PFA (psychological first aid) and TRT (teaching recovery techniques)' for the Korean version for infants and children was higher than 0.49. However, the CVRs for 'SSET (support for students exposed to trauma), TF-CBT (traumafocused cognitive behavior therapy), EMDR (eye movement desensitization and processing), PE (prolonged exposure therapy), trauma-focused play therapy and art therapy' were low. In this case, the opinion of experts on Korean culture was reflected in the third round. However, the need for the Korean version of the PFA, TRT, SSET, TF-CBT, and EMDR was associated with a CVR higher than 0.49 ( Table 5).

Results of third-round Delphi survey
The evaluation items and intervention items for Round 3 are described in detail in Tables 6  and 7, respectively.
The CVR for Round 3 was 0.38 or higher, and the content validity was verified for nearly all items. The major items with high CVRs are described as follows.
The CVRs were higher than 0.38 for the following items: 'children and adolescents experiencing trauma should adjust to their current life to recover from trauma', 'stabilize their social and interpersonal functions', and 'fulfill their developmental tasks in the long term' ( Table 6). Post-disaster psychosocial evaluation and intervention for children In particular, in the high-risk group, the average value and the content CVR were the highest for the item 'the child and the family should be evaluated'. The highest CVR was observed for the opinion that a trauma-related scale and scales for depression, anxiety, suicide, sleep, and social resources are needed. The CVR was 0.38 or higher for the items indicating that specialists who perform a psychological assessment in a disaster need 'crisis management training' and the 'ability to cope with various responses of clients' ( Table 6).
In terms of the intervention program, the CVR was the highest for 'psychological education for the post-traumatic response, normalization, stabilization, physical stability training, family and teacher education, and emotion education should be included.' With respect to the elements of a therapy program, a high CVR was observed for 'requiring PFA, TRT, SSET, TF-CBT, EMDR, PE, a family participation program, a mourning-themed program, individual psychotherapy and medication'. Opinions suggesting that 'individual psychotherapy and medication are needed' were most frequently observed. In addition, some comments indicated that 'child-parent psychotherapy might be more appropriate than PFA and TRT for toddlers and preschoolers' (Table 7).
With respect to the termination of therapy, CVRs higher than 0.38 were observed for the following items: 'the intervention should be terminated after the prescribed therapy sessions' and 'referrals should be determined thereafter' (Table 7).
A high CVR was found for the item regarding the intervention development strategy: 'establish a therapeutic linkage system based on national support, educate and inform the whole school, support medical expenses (such as with government subsidies), connect with the community, consider the persistence of treatment cases, and reduce the stigma of psychiatry' (Table 7).

Discussion
In South Korea, the dispute over how to evaluate and intervene in the aftermath of the Sewol Ferry Disaster required a consensus regarding the need for disaster planning. [24] The Delphi process was a suitable method for surveying experts on this topic. [25] Using this method, we propose a multidisciplinary recommendation for treating children exposed to disasters. The results of qualitative and quantitative analyses conducted through the Delphi panel survey demonstrate that psychosocial assessment and intervention are essential to early mental health services following a disaster. We discuss suggestions based on the consensus of the experts involved in the study.
We found that in the event of a disaster, intervention factors such as 'appropriate time for assessment after the disaster', 'prerequisites for screening and in-depth intervention', 'classifying the degree of psychosocial symptoms', and 'social and mental health services' are very Post-disaster psychosocial evaluation and intervention for children  Post-disaster psychosocial evaluation and intervention for children Post-disaster psychosocial evaluation and intervention for children important. Recovery from psychological trauma after a disaster means mental stability as well as the recovery of physical health. Screening tests are recommended for all children exposed to disasters, particularly during acute periods of disaster. After the completion of screening tests, assessment should include in-depth interviews and interventions for the high-risk group. First, however, we must distinguish between brief screening and in-depth evaluation. As in our study, many previous studies have suggested mental health assessments and interventions for children. [26] These findings are consistent with research findings indicating that screening is appropriate when large numbers of children are exposed to an event or when the level of exposure among a population is unknown. [27] The actual screening assessment performed after a disaster requires the consideration of each stage of the disaster and should consist of appropriate questions. [28] In the disaster context, screening tools should reflect the needs of children with mental health problems, including consideration of children's exposure, experience, and subjective reactions to traumatic events and conditions. [29,30] Evaluation of children, families, and teachers during the acute phase of a disaster is important. Above all, consensus among experts on the selection of children exposed to a disaster is required. Families and teachers should be evaluated together. The use of multiple informants, such as parents, teachers, and other professionals, as collateral sources of information enables the most comprehensive appraisal of children's reactions and functioning. [28] These results are consistent with the opinion that it is important for parents and/or caregivers to participate together in a child's treatment session to recover from PTSD symptoms. [26] When interviewing a family member, we must check for signs of psychological crisis among family members. This finding is consistent with studies of the family environment, social support, and supportive quality. [31] However, it is not necessary to evaluate acquaintances or friends. Furthermore, assessments of grief, depression, anxiety, and suicide risk, as well as trauma-related scales, need to measure PTSD and other psychosocial symptoms. This finding is largely consistent with a previous report that disaster exposure is correlated with PTSD, depression, anxiety, functional impairment, and behavioral problems. [32] In addition, trauma assessment of children and adolescents should consider their developmental stage. When treating a child who has experienced trauma, the clinician must understand the child's existing psychopathological symptoms and provide appropriate interventions, such as trauma-focused therapy. [26] Our results suggest the need to develop a crisis intervention model for children and adolescents. [33] Psychosocial assessments should be conducted in a safe environment and at appropriate durations of 30-60 minutes. Approximately 30 to 60 minutes is needed for screening a highrisk group.
Psychoeducation is also beneficial to children. A post-disaster intervention program should include the following: psychoeducation, guidelines for coping with the media, normalization, stabilization, techniques for handling survivor's guilt and emotion-focused coping strategies. Appropriate access phases can be classified as hyper-acute, acute, sub-acute or chronic stages. Stabilization and psychological support should be provided immediately after a disaster along with intervention to help children adapt to everyday life. This finding is consistent with a report that most interventions are multimodal, incorporating common elements to educate children, normalize their reactions, process their emotions and manage stress, enhance coping and provide social support. [27] In addition, the development stage, age, trauma symptoms, and traits of a group should be considered. The number of children participating in a group may vary depending on the type of disaster. In general, psychoeducation can be provided in the class setting at school. For prevention education, holding one to four sessions is recommended, whereas for therapeutic intervention, five to eight sessions are appropriate. If the child is exposed to a national large-scale disaster, intervention to address brief trauma may not be sufficient. Therefore, professional intervention should be provided, particularly for children with symptoms of PTSD. [26] For a preschooler, the appropriate duration of an intervention is 30 minutes with caregiver participation. A proper duration of 30 to 40 minutes is suitable for elementary school students in lower grades. An intermediate duration of 40 minutes is suitable for elementary school students in higher grades. For middle and high school students, intervention programs could last 45 to 50 minutes. The optimal intervention components may not be the same for all children or all situations, which should be examined in future work. [34] We recommend the following available intervention programs: PFA [35], TRT [36], SSET [37], and TF-CBT [38]. In South Korea, the South Korean versions of PFA, TRT, and TF-CBT should be standardized for children and adolescents. However, the study findings provided no suggestion related to narrative therapy. Furthermore, an intervention for toddlers and preschoolers should be considered. Multiple evidence-based programs should be considered as well, and an intervention protocol that includes a standardized South Korean version can then be implemented. These results provide a framework for further research. Accordingly, the CIDER (Children In Disaster: Evaluation & Recovery) protocol developed by the authors of this study will be made available. Additionally, we must include not only child-focused therapy but also long-term mental health services. These findings are partially consistent with a prior study. [39] The professionals providing disaster interventions vary with respect to factors such as availability, training, and experience, and the goals and complexity of the intervention differ as well. [27] Nevertheless, affected communities do not have enough therapists trained in evidence-based treatments to be able to provide every child with individual therapy. [39] It is not necessary for all mental health workers to conduct evaluations and interventions after a disaster. Therefore, disaster experts with experience working in a clinical environment should be called upon; a training and education system for professionals is needed. Such professionals may need additional support and guidance to address their own emotional responses. [27] This support can be incorporated into supervision as well as peer support groups. Additionally, the present study shows that good relationships should be cultivated within professional networks of information related to in-depth therapy.
Above all, interventions delivered in groups are particularly well suited for school settings. [27] Schools are among the most important links in the chain of public health education for children and adolescents. [40] School-based interventions should be developed, regular training in disaster safety measures for school personnel should be mandated, and training programs for children should be established. Moreover, teachers should receive advice on coping with emergencies in either their basic teacher training or in-service training. In summary, schools should identify school crisis emergencies and clearly delineate the roles of children and teachers in coping with disaster. Based on the abovementioned considerations, psychiatric and psychological support should be accessible. Additionally, guiding children to use positive coping strategies and encouraging a warm community atmosphere are recommended. [32] Consequently, our confidence in reaching consensus means that we now have a comprehensive framework of competency statements that describe what psychiatric professionals working in the aftermath of a disaster must do. As the National Child Traumatic Stress Network has coordinated collaboration among 10 research development and evaluation sites and 26 community mental health centers across the United States, it is also essential to establish sensible governance between central and local governments, between administrative institutions and institutions that provide services, and between public and civic organizations. [41] This study proposed effective mental health intervention measures and described the implications for developing a post-disaster evaluation treatment protocol. The main strengths of our study include its responses from a panel of defined experts, good response rates and framework of competencies that describe attributes of professionals working within the disaster field. However, some limitations also need to be recognized.
First, the study findings suggest that children in South Korean cultures require disasterrelated psychosocial evaluation and interventions, but modifications may be needed to address other cultural issues.
Second, our expert panel was determined by our approach to sampling. E-mails may not have been distributed by some of the professional groups we contacted, and other experts not publishing their work may have been missed. The rich qualitative and quantitative data obtained from this study are very useful for understanding why certain topics are research priorities. [21] Third, the experts who conducted psychological intervention at Danwon High School after the Sewol Ferry Disaster in South Korea were all psychiatrists, except for two psychologists. [42] The primary aim was to gather psychiatrists' opinions and experience from the disaster environment. In Round 1, we had limitations in distinguishing between the related areas of expertise in disaster and trauma for the psychological specialists, and these limitations might be reflected in the medical opinions of the panel.
In conclusion, we suggest the need for informed evidence-based assessments, interventions, and treatments for children and adolescents who experience disasters. This survey presents important opinions from trauma care experts and should be utilized by psychiatrists to develop a meaningful protocol for PTSD assessment and treatment. Hence, the results can be applied to existing and future disaster management.