Introduction

The Committee on Disaster Risk Reduction (UNDRR, UN Office for Disaster Risk Reduction) has defined a disaster as a serious disruption of the functioning of a community or society involving widespread human, material, economic or environmental losses and impacts, which exceeds the ability of the affected community or society to cope using its own resources [1] On that basis, it is noteworthy that disasters that occur in transborder areas are especially complex, especially when international intervention is required [2, 3]. In addition to the complexities of multi-victim incidents, cross-border events require the collaboration of different emergency providers, of different nationalities, speaking different languages, with different skills (emergency vehicles with medical staff are not always provided) [4] and with first responder systems that differ from one another (sometimes police or firefighters, even the military are the first on the scene) [5] in addition to the differences related to the prevention and promotion programmes of the different countries, the health systems they have (public and/or private) and the implication for the health of the patients attended [6], which is related to survival in the event of an event of these characteristics. Including the risk associated with the safety of staff working in this type of event.

Here we must acknowledge the fundamental role of the emergency medical services (EMS) in disaster situations with multiple casualties, when the lack of resources identified in the UN’s definition is particularly apparent along with the need for collaborative working across differing action protocols, means of communication and even laws. Such scenarios raise significant ethical issues, because critical decisions have to be taken quickly by responders, who may also become direct or indirect casualties in the course of their efforts.

It is in these scenarios that a highly effective chain of command and control translates into highly effective use of resources, the shortest time to intervention and, finally, the saving of human lives. There is a need to develop harmonized procedures and a Minimum Data Set (MDS) for cross-border Multi Casualty Incidents (MCI) to ensure appropriate management of such incidents, regardless of place, language and internal processes of the institutions involved. Such transnational incident management will be possible and have a significant social impact if there is prior agreement as to the information required, making use of the opportunities offered in terms of technology, procedures, collaboration, and training, by the Security Market of the European Union (EU), thus far very fractured.

Today there are already MDS as tools for the systematic collection of minimal basic information about episodes of care for individual patients to have a system of high-quality exhaustive and consistent information that allows improved process management [7, 8]. Thus, various works have validated MDS and so highlighted the need for consensus on the critical information required to support the management of disaster scenarios [9, 10].

Although great progress is being made in standardising processes and data collection through MDS, there are important differences not only between countries, but also between regions within the same country, especially in lower-middle-income countries [11]. There are also inequalities in terms of resources used, with large differences between hospitals, primary care centres or nursing homes, even within the same cities [12]. This is why, in 2015, the UN developed a document to help build and improve health electronic systems through the MDSs to improve data interoperability at national and subnational levels [13].

It is crucial to adopt clinical information structures that allow interoperability among the EU’s Member States under laws concerning patients’ rights in cross-border healthcare (Directive (EU) 2011/24 of the European Parliament and of the Council of 9 March 2011) [14]. This advance will also facilitate the availability of high-quality normalized data that may be useful in the context of the European Health Data Space [15], which is a health-specific ecosystem comprised of rules, common standards and practices, infrastructures and a governance framework that aims at empowering individuals through increased digital access to and control of their electronic personal health data, at the national level and EU-wide, and support to their free movement, as well as fostering a genuine single market for electronic health record systems, relevant medical devices and high-risk AI systems, providing a consistent, trustworthy and efficient set-up for the use of health data for research, innovation, policy-making and regulatory activities [16].

The EU has developed various tools for collective collaboration and cooperation between Member States and across industries. Of particular note is the EU’s Mandate M/487 [17], adopted to develop work plans and roadmaps for the normalization of standards in security and the protection of critical infrastructure in the EU. Mandate M/487 requires the identification of the challenges to be met in the use of terminology, planning, resilience and institutional interoperability.

In response to that need and in order to identify a set of solutions to improve collective resilience in the face of MCI, Project VALKYRIES was launched under the EU’s Horizon 2020 Framework Programme (Harmonization and Pre-Standardization of Equipment, Training and Tactical Coordinated procedures for First Aid Vehicles deployment on European MCI. Project VALKYRIES falls under the programme entitled ‘Secure Societies—Protecting freedom and security of Europe and its citizens’ and more specifically under the sub-programme ‘Increase Europe’s resilience to crises and disasters’, Sub-topic 3 ‘First Aid vehicles deployment, training, maintenance, logistic and remote centralized coordination means’, under procurement process SU-DRS03-2018-2019-2020, ‘Pre-normative research and demonstration for disaster-resilient societies’. This project was active from October 2021 to October 2023 a comprised a set of partners from a wide variety of EU countries and different expertise.

In that context and in light of the salient issues identified in EU M/487 [18] in terms of the gaps to be bridged in interoperability, incident management, organizational interoperability, and operational effectiveness, H2020-VALKYRIES identified the following objectives inspired by the Observe-Orient-Decide-Act loop (Boyd’s OODA Loop [19]).

Materials and methods

To that end, a group of 6 researchers from the VALKYRIES project carried out a consensus data harmonization process to determine an MDS that could be collected and communicated electronically in real time to the chain of command and control, seeking optimal management of an MCI in cross-border areas regardless of the particular procedures, protocols, languages and location of the event.

That exercise was carried out through a combination of a variant of the Delphi method (modified Delphi) [20, 21] as a validated consensus tool and the Utstein technique or style [22] to select the essential data, as described above.

The basic principles that guided us in the selection of the final items were: the feasibility of collecting the data, real-time onward communication of the data to the whole chain of command and control and being thought to be essential to good management of an MCI.

Modified Delphi is an abbreviated version of the classic Delphi modified to the specific needs and resources of the project. Subsequently, abbreviated Delphi was complemented by the Utstein method to improve the quality of the data gathered and allow more precise comparison of results across different studies concerning emergency response.

For this purpose, the researchers were supported by 17 experts (with more than 20 years of experience in their field of expertise) from 17 agencies and institutions of 8 different countries, involved in the Valkyries project.

The MDS was presented and validated by external experts of the Valkyries project before being tested in the 4 major cross-border simulations foreseen in the project. The list and profile of researchers, internal and external experts can be seen in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Study participants classified by area of work and country of activity.

https://doi.org/10.1371/journal.pone.0305699.t001

The consensus process in four phases described below (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Study development process.

*MDS: minimum data set.

https://doi.org/10.1371/journal.pone.0305699.g001

Phase 1—Logistical and operational framework of the process: In this phase, carried out by the consortium, we determined the shared, essential principles to govern the whole process. They were the minimum requirements to be met by the items selected (being thought to be essential to management of MCIs in cross-border areas; suitability for collection, and onward communication of items without any need to change the protocols and procedures of first responders (regardless of place and country)), qualifications and experience of group of experts.

After that, it was necessary to pre-identify the data. This information comes from previous studies [9, 23, 24] and the information currently collected in emergency management in different countries.

Phase 2 –Modified Delphi. It was led by the study team and the internal experts also participated in this phase. There were several face-to-face and online meetings as described below. In the first round of the Delphi method, the scope and structure developed at earlier stages were validated, the researchers organised the preliminary variables into 6 groups based on the literature consulted. Subsequently, each member of the participating suggested new items that they considered important for review. In the first meetings, the objectives and criteria determined in Phase 1 were validated and new items were added at the request of the internal experts. There were then two rounds of voting. Voting required each expert to score each candidate variable from 1 (the item is not material to management of an MCI) to 10 (the item is material for management of an MCI). In Round 1 of voting, items with no votes in favour were discarded. The objectives of the project were reviewed, and a new vote was taken on the variables.

Phase 3 (Utstein Method), led by the researchers, several rounds of discussions were held to choose the essential variables of the MDS. The internal experts who had assigned each variable a value between 1 (not essential for MDS management) and 10 (essential for MDS management) agreed to eliminate those variables with less than 5 points. As a result, the data were ordered from most to least essential for incident management. The internal experts decided that the data with an average score below five should not be considered as essential and, therefore, should not be part of the MDS. This resulted in a digital CMD agreed upon by all the experts in the project.

In Phase 4, the researchers submitted the CMD to external experts, consisting of disaster management and MCI experts from the institutions and agencies involved in the four major simulations foreseen in the project, for final assessment and validation prior to its use in the simulations.

Finally, as part of our modified Delphi strategy, we determined the degree of agreement among the experts by calculating Cohen’s kappa [25, 26].

To that end, we considered agreement between three or more raters (experts) with three or more possible scores (degrees of agreement). In this case, we used a weighted kappa given the number of categories of degree of agreement equivalent to an ordinal scale. The two types of weighting used were squared weights and Cicchetti weights, based on relative distances between categories [27].

The presentation of values of kappa complemented the presentation of the consensus showing degrees of agreement of 55% and above. Data were analysed using IBM’s SPSS Statistics version 2.0. This study did not require consideration and approval by the Ethics Committee of Research with Medicines.

The present study did not need to be submitted to the evaluation of an Ethics Committee since the VALKYRIES Project did not deal with personal data, only simulated data were applied. The Panel (specific working group to ensure legal-ethical compliance with the EU Regulation, which all consortium members signed up to) was made up of ethical-legal experts, who were involved in both compliance and regulatory and standardization tasks, and a declaration was signed by those responsible for the Panel, showing the commitment to comply with all ethical-legal issues. Within this declaration, the decision of the Panel to develop the work without hiring volunteers, not processing personal data, is stated. The case study scenarios involved first responders to receive feedback on the effectiveness of the interoperable platform through the application of simulated and unreal data. The simulated data were generated directly by the Panel.

With regard to inclusivity in global research additional information regarding the ethical, cultural, and scientific considerations specific to inclusivity in global research is included in the S1 Dataset.

Results

Six data entities were defined with 31 sub-entities (pre-incident data, incident, vehicles, first responders, casualties, and hospitals) that generated the 127 items in the final MDSs, comprising data essential to incident management, capable of being collected and communicated electronically in real-time to the chain of command and control as shown in Table 2. This table presents each data entity and sub-entity identified, in addition to each data entry selected by the organizations participating in the voting process. Moreover, this table contains the data type associated with each data entry to better understand the nature of the information displayed. Furthermore, the table presents one or more examples per data entry to ease the understanding of the content. It is relevant to note that these examples are not agreed values and merely serve as an orientation for interpretation. Finally, Table 2 presents the degree of agreement between voting organizations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Presentation and classification of data: Data entities, subentities and items.

https://doi.org/10.1371/journal.pone.0305699.t002

Therefore, the data obtained is presented as follows:

6. Hospital MDS

Finally, hospital-related information, such as identification, contact route, hospital incident manager, statistical data and dynamic data needed for resource and incident management were specified.

Here the sub-entities identification, general contact, hospital incident manager, static data, dynamic data and task assigned were generated with 46 items that obtained a degree of agreement of 10/10 (“agency”, “city”, “address”, “hospital location”, “general phone number”, “available adult burn unit beds”, “available pediatric burn unit beds”, “available cardiac surgery op rooms number”, “available plastic surgery op rooms number”, “available pediatric surgery op rooms number”, “available obstetric op rooms number”, “available CT number (computer tomography)”, “available MRI number (magnetic resonance)”, “available dialysis posts number”, “available isolation rooms number”, “available hyperbaric chambers number”, “task ID”, “incident ID”, “hospital ID”, “victims assigned”), 7.5/10 (“hospital name, country”, “ICU phone number”, “total adult BURN unit beds”, “total pediatric burn unit beds”, “available adult ICU posts”, “available pediatric hospital beds”, “available pediatric ICU posts”, “available operating rooms number”, “available thoracic surgery op rooms number”, “vehicle ID”) 8/10 (“ED phone number (emergency department)”) and from 5/10 (“hospital incident manager category”, “total adult emergency department posts”, “total adult ICU post”, “total pediatric emergency department posts”, “total pediatric ICU posts”, “total general surgery op rooms number”, “total pediatric surgery op rooms number”, “total CT number”, “available adult emergency department posts”, “available pediatric emergency department posts”, “available orthopedic surgery op rooms number”, “available general surgery op rooms number”, “available neurosurgery op rooms number”, “available vascular surgery op rooms number”).

The 127 items finally agreed for inclusion in the MDS can be readily recorded in real time by first responders to an MCI thanks to a system for the collection and communication of digital data called SIGRUN, developed under Project VALKYRIES. In fact, the agreed MDS was tested in the four simulations run by the EU’s Project Valkyries. The medical emergency response services of the eight countries involved in the simulations at four cross-border and cross-sectorial scenarios (Spain-Portugal, Slovenia-Italy, Bulgaria-Greece, Norway-Netherlands) were able to gather and communicate the MDS items to the chain of command and control for each incident in real time. They were able to confirm the effectiveness of this new harmonized tools in different highly complex scenarios, such as: forest fires; earthquake; chemical, biological, radiological and nuclear (CBRN) and offshore disasters. The results of those simulations will be published in the coming months.

The results with the highest level of consensus in our study are for the MDS for the category ‘Incident’ with overall kappa of 0.7401 (95% CI 0.1265–0.5812, p 0.000), a good level of consensus in the Landis and Koch model [25]. The items with the greatest degree of consensus (10 out of 10) were those relating to location, type of incident, time and identification of the incident.

With moderate values (kappa of 0.4–0.6) in the Landis and Koch model, we find the description of the situation prior to the incident, with overall kappa of 0.44 (95% CI 0.0114–0.2531, p 0.0093) and ‘first responders’ with overall kappa of 0.4318 (95% CI 0.0599–0.5928, p 0.0000). The degree of agreement on the items concerning pre-incident information was similar (6 out of 10). Conversely, the disposition, function and type of first responders garnered the highest level of agreement, in excess of eight out of ten.

The categories that had low levels of agreement (kappa 0.2–0.4) were vehicles with kappa of 0.2696 (95% CI 0.0780–0.2482, p 0.0000); casualties, with kappa 0.2320 (95% CI 0.0261–0.1393, p 0.0006); and hospitals, with kappa of 0.3241 (95% CI 0.1076–0.1500, p 0.0000). However, the degree of agreement was maximal for some items, including the personal details of patients and their patient reference numbers to ensure maximum traceability; patient age band and gender; location; and triage colour code. All results for degrees of agreement are shown in the Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Levels of agreement between participating agencies.

https://doi.org/10.1371/journal.pone.0305699.t003

Discussion

Scientific publications on MCI are still very scarce, even more so when it comes to cross-border events involving international organisations. To date, there is no evidence that determines what MDS should be collected or how that data should be collected and managed. Consequently, the principal strength that can be claimed for the method employed is that it achieved consensus in areas where there is no certainty or where evidence on a topic is empirical [9].

In addition, the results advance the principal objective of the project because although there are a number of limited-scope MDSs, there is currently no exhaustive MDS and no analysis across a range of cross-border disaster scenarios, given that information has traditionally been collected from observation or analogue capture. However, this study worked in parallel on the capture of digital data, as will be described in subsequent articles.

One of the problems observed in the management of MCI is variability in the way in which information is gathered, which may in turn lead to divergent results and so hinder comparison and standardization of those results [28]. That is all the more so in cross-border (MCI), such as the simulations that were conducted in an earlier phase of this project.

Another problem in the management of disasters seen in connection with MDSs for casualty information and how to collect that information is that not all the information about a casualty should be shared with all the agencies involved in an MCI. However, some part of that information may be important for command-and-control decision-making [29].

Modified Delphi and subsequent Utstain-style technique allowed a digital MDS to be developed with the involvement of 17 experts from eight organizations and institutions involved in the Valkyries project. Experts from eight different countries with extensive experience of emergency and disaster management, IT and cybersecurity, standardisation, etc. validated this MDS, which was subsequently tested in four large-scale simulations in eight European countries.

This tool complies with the Health Disaster Management: Guidelines for Evaluation and Research issued by the Working Group on Quality Assurance in Disaster Management (WADEM) [30] and the tool developed by the Academy for Emergency Management and Disaster Medicine (EMDM Academy) [9] which provides standardised systematic empirical information on disaster medical response to support monitoring for further evaluation, training and research.

The variability across procedures and action protocols of organizations with responsibility for MCI/disaster management and the variability of technical and human resources, with different degrees of training, education and specialisation, can make it difficult to provide an optimal joint response. This further complicates the evaluation, training and investigation of such incidents [31–35]. The digital MDS that has been developed and tested in the four simulations makes possible standardized systematic collection of data that supports the monitoring of many aspects of incident management and so in turn undoubtedly lead to the identification and assessment of areas for improvement, training, guidance and research in this field.

Data collection in the course disaster management has not thus far been regulated [34, 36–38]. The MDS in this study is based on current scientific evidence and allows for the systematic collection of data in real time, as shown by the collection of data in large-scale simulations involving eight European countries–in a field of research sorely lacking empirical research [28, 39]. The automatic digital collection of data speeds the provision of healthcare, which is one of the most significant issues in the reporting of medical care in such incidents [36, 40, 41].

It is also significant that information about casualties does not always originate from health services. Casualties may receive initial care from other agencies, such as firefighters, police and volunteers and that information is not recorded in support systems, leading to loss of information as a casualty passes from the care of one service into the care of another service. Consequently, the way in which information should be processed is not always the same and applicable laws relating to the collection of sensitive personal data should be followed at all times and the rights of casualties under the General Data Protection Regulation (GDPR) should be given full effect.

The advantage of the tool lies in its neutrality with the way of working of first responders, their action protocols, language, etc. These MDS is put forward as the minimum essential data for optimal MCI management. Another advantage of the digital MDS presented is the possibility of combined collection and digital dissemination in real time to the entire chain of command and control, both in the country of collection and to other countries. For example, in the simulation exercise carried out in May 2023 on the border between Spain and Portugal, it was possible to record all the items digitally in real time and international command across the two countries had information on the number of casualties, their characteristics and the resources involved at all times. The data were stratified and protected so that each responder could only access data related to their specialism.

The healthcare coordinators, like the security services, did not have access to specific medical data for patients, but they were informed in real time of the number and severity (colour) of each Triage station. Conversely, doctors and nurses at the medical stations could access patients’ medical report, but not other non-medical data such as numbers and types of resources known to the incident commander.

We thus eliminated the need to use telephones or radio to communicate the information needed to manage the incident with a significant impact on delays to incident management pending availability of information.

For that reason, in this study the expert consensus reached through Delphi focused on optimizing incident management and minimizing the need to communicate information by voice and maximizing the traceability of casualties and the management of transport, among other data. It was not therefore focused on the medical treatment or management of individual patients. Interventions of this nature are particularly useful when there are issues such as remoteness and challenging geographies, requiring the intervention of teams from different countries, with different languages and ways of working. The scope for error is reduced by a standardized procedure in especially complicated situations such as (MCI), which can reduce the time taken by healthcare staff in treating casualties and so reduce morbimortality and allow investigators to collect and interpret reliable evidence to improve survival and reduce morbidity among patients treated in an MCI.

As can be seen from Table 3, the degree of consensus was as high as possible (10/10) among experts for the basic items that directly affect incident management, such as identification of the incident, its location, type of resources, casualty triage, etc. We should remember that all data was gathered and communicated in real time to the whole chain of command and control, greatly improving the management of the four large simulations run to confirm the effectiveness of this tool. It is also important that it proved possible to collect the data regardless of place and language, with full respect throughout for the procedures, protocols, and laws of each country.

The value for Cohen’s kappa to be taken as showing a good level of agreement varies depending on the problem at hand. In practice, we have used a scale for the interpretation of kappa values from Acceptable as equal to or greater than 0.40 (First Responder) to Excellent as above 0.75 (for Incident) [26].

However, our results (particularly in relation to casualties and hospitals) show that it is possible for there to be a high level of observed agreement and, at the same time, a kappa of close to zero. The apparent paradox of high levels of observed agreement and low kappa values can be explained by the fact that for a given level of observed agreement, the value of kappa depends on the frequency or rarity of the phenomenon under study [27]. Among the solutions proposed for this problem is to state the maximum and minimum values of kappa with the calculated value of kappa. We must also bear in mind that as the number of categories increases, so does the difficulty of correctly classifying the observed items, which generally leads to lower values of kappa [42].

It is now worth noting a peculiarity of this study, namely the use of Kappa to measure the degree of agreement of the evaluations. That this is uncommon can be confirmed from a review of prior similar studies in the same area [43–46] where such calculations were not carried out. In another case the Kendal coefficient was used [47]. The Kendal coefficient is used to measure associations between ratings and is suitable for ordinal rankings (ratings on a scale), which is not the case in this study [48]. As noted above, Kappa is particularly suitable for use with a Delphi-type consensus method.

Studies such as that by Debacker M. et al. [9], involving the development of a standardized model for the management of data in catastrophes, proposed testing such models in the field. That is where the novelty of this study lies, that we have tested the degree of agreement of this gathering of Utstein-style MDS not only in situ but in cross-border MCIs, thus giving a good representation of the operation of the model.

On the basis of the data obtained through Delphi, we propose conducting further projects to address the challenges that we encountered. To assess the potential usability of metadata where data are collected electronically. Analysis should be carried out after the fact with exact response times. A trial should be conducted to determine the format of such information: there are many choices, such as triage colour, age band, numerical values such as age and alphanumeric information such as name and surname(s). In conclusion, to find a solution to the problem of data segmentation.

In terms of limitations, while a major step forward has been made in bringing this method to the field, it is still a simulation. To assess its full potential, the MDS would have to be tested in real-world multi-casualty incidents. A second limitation is testing only in European countries: it should also be tested in areas undergoing development to assess its suitability. The modified approach adopted in this Delphi survey is considered superior to the original approach because it is highly effective and less time consuming. Having an “expert panel” is central to the process of the Delphi technique, although there are no standard criteria for determining expertise. In the current study, the panel of experts comprised people from several countries/cities who are in diverse professions, such as university academics, physicians, and nurses.

Conclusion

This study documents the development of an MDS through consensus with a high degree of agreement in a group of experts of different nationalities working in different fields. All items in the MDS were digitally collected and communicated in real time to the chain of command and control. This tool has demonstrated its validity in four large cross-border simulations involving more than eight countries and their emergency services using a method that generates proposals for further investigation. The work falls within the framework of proposals made in the Conclusions of the Council of Europe for Disaster Risk Reduction in EU external action (28 November 2022).

Supporting information

Acknowledgments

We thank the Project VALKYRIES team, the Foundation for Biosanitary Research and Innovation in Primary Care (FIIBAP), the European Programmes Technical Support Unit (UTAPE) at the Ministry of Health of the Autonomous Community of Madrid for making this study possible and to all the organisations and institutions involved in the 4 major drills, together with the experts participating in these drills.