Introduction

Human identification is crucial when catastrophes cause large-scale deaths. After the 2011 east Japan earthquake and tsunami, approximately 10% of the total number of victims whose identities were confirmed were identified by dental examination. In the 2003 Daegu subway fire disaster in Korea, a similar pattern was observed. Disaster victim identification (DVI) by dentists has therefore garnered growing attention. Teeth and dental treatment types are useful in human identification because they are resistant to destruction by fire [1–3]. The use of teeth, one of the hard tissues in the human body, helps reduce the number of potential candidates and accurately confirms their identity.

With an increasingly aging society, identifying individuals by dental restorations is more likely because the number and complexity of dental restorations increase with age [4]. A previous study [5] reported that individuals who had received several complicated dental treatments were more easily identified than individuals who had not received or rarely received such treatments. A dental panoramic radiograph (DPR) is a reliable tool for human identification because it is regularly filmed and updated. DPRs, which contain most of the major dental information, are widely used in clinical practice and therefore have a high potential for clinical use.

In recent years, artificial intelligence (AI) has assumed an important role in the types of image processing that are cumbersome or time-consuming for the observer [6]. Object detection, image classification, and pattern recognition using deep learning have developed rapidly and are applied to medical diagnoses using dental imaging [7,8]. The convolutional neural network (CNN) method is prominent in the image domain and is particularly useful in finding patterns for image recognition [9], including radiological applications in general medical fields [9,10]. In particular, object detection has recently undergone remarkable development, and has achieved the establishment of an efficient model variously named “end-to-end learning” [11–13], “transfer learning” [14,15], “active learning” [16,17], and “explainable deep learning” [18,19]. Recently, explainable deep learning models have been increasingly introduced into biomedical image analysis to overcome the inherent black box problem.

AI has been most actively introduced in diagnostic imaging in the medical imaging area. Research on human identification conducted using deep neural networks began with detecting teeth and dental prostheses from DPRs [20–32]. Going further from the excellent research results on detection, the research trend of deep learning in the field of human identification is the comprehensive establishment and automation of human identification processes. Studies related to human identification that simply compared DPR images predominate. The mainstream themes of existing studies have been to find possible candidates based on the similarity of images or to measure similarities, based on various identifiers by extracting certain features or patterns from DPRs. Extracting features or performing comparisons on images could be intuitive and fast, but it may also be affected by several endogenous variables such as distortion, contrast, and resolution of DPRs [33].

To date, few studies have conducted comprehensive simulations of the human identification process using deep neural networks. To achieve the ultimate comprehensive human identification, an approach that employs database construction is essential; it takes one step forward from existing studies of human identification that are primarily based on traditional comparisons of the postmortem (PM) and antemortem (AM) images. When the database is constructed, various parameters can be extracted from the images, and developing a human identification methodology from diverse viewpoints would ultimately be possible. This study aimed to develop a novel automated system that can identify humans as a basis for DVI in the field of forensic science.

Material and methods

DPR-pair dataset for constructing a database for the human identification model

The Institutional Review Board of Seoul National University Dental Hospital (SNUDH) approved this study (ERI20032) and waived the need for informed consent. The entire experiments including data acquisition were conducted under the relevant research regulations and guidelines. DPRs of 1,029 individuals were used to simulate the postulated human identification process. The dataset contained 2,058 DPRs comprising paired images of the recent and past DPRs for each individual. These images were randomly selected after removing identifiable patient information and were retrospectively reviewed from the picture archiving and communication system at SNUDH. All authors had not access to information that could identify individual participants during or after data collection. The radiographic images were obtained using panoramic radiographic machines OP-100 (Imaging Instrumentarium, Tuusula, Finland) and Rayscan Alpha P (Ray, Gyeonggi, Korea).

From January 2000 to November 2020, DPRs with information that include age and sex were collected for dental treatment or diagnosis. Recent and past DPRs of adults aged 20–49 years (465 men and 564 women) who had more than two DPRs were assumed as the PM and AM images, respectively. The timepoints at which the PM and AM DPRs were recorded were also collected as six digits in the format of two-digit year, two-digit month, and two-digit day. The age range was limited to 20–49 years to minimize the impact of dentition changes and image distortions over time, which could lead to false identifications. This range was chosen based on the exclusion of younger individuals with deciduous teeth and older individuals with higher rates of tooth loss, ensuring a more consistent and reliable analysis. Additionally, the age criteria used in the previous study [21] were maintained to ensure the validity of the model.

Detection of natural teeth and dental treatment types using the pre-trained model

Natural teeth and dental treatment types were detected and the teeth numbers were identified using a pre-trained object detection network (Fig 1) which was a CNN modified by EfficientDet-D3 [21]. DPRs (n = 1,638) were collected in advance to construct this model. These DPRs were mutually exclusive from the existing 2,058 DPRs of 1,029 individuals. A fully web browser-based labeling system developed by Digital Dental Hub (DDH, Inc., Seoul, Korea) was used in the annotation task.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Deep neural network architecture for the detection of natural teeth and dental treatment types.

https://doi.org/10.1371/journal.pone.0312537.g001

Table 1 presents the overall performance of the model, composition of the dataset and description of the subnetwork for each task in the network architecture. The performances were evaluated using the standard of average precision and average recall under the intersection over union (IoU) of 0.5. A true positive was considered only for prediction scores >50% and an IoU with a threshold value of 0.5.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. The performance of pre-trained network for object detection in dental panoramic radiographs.

https://doi.org/10.1371/journal.pone.0312537.t001

Data post-processing

A database was developed by generating the dental information using the pre-trained model from each set of pairs of the PM and AM DPRs of 1,029 individuals. Detectable information that could be extracted from each DPR using deep neural networks varied substantially; therefore, it was simplified to focus on the main dental components. This dental information consisted of natural teeth, including tooth number, prostheses, treated root canals, and implants. Detection of dental pathologic conditions such as dental caries or periodontal lesion was not considered.

Based on the state and direction of change of the individual’s teeth, the data regarding the detected natural teeth and dental treatment types were post-processed as objective values. As the teeth condition changed (Fig 2), the possible past status of the tooth condition for each tooth condition was determined. The summary of detailed past status was provided as supplementary data.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Index system based on the degree of dental treatment.

The tooth status can be changed only in the direction of the arrow. The tooth status can be converted to each other in case of missing teeth, pontics, and implants.

https://doi.org/10.1371/journal.pone.0312537.g002

Evaluation of the process of human identification

To simulate the automatic process of human identification, the algorithm for human identification was implemented by deriving possible candidates from the AM database, assuming the situation of an unknown DPR in the PM database. Three phases were established (Fig 3).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Scheme of the simulation of human identification using the automated human identification process, based on the dentition of an unknown postmortem (PM) individual.

In Phase 1, a database of antemortem (AM) dentition was constructed. In Phase 2, similarity scores were calculated for each pair of PM-AM dental panoramic radiographs. In Phase 3, the scored similarities were sorted in descending order and extracted for the top 20.0%, 10.0%, and 5.0%. The matched rank was calculated as the success rate for the percentage.

https://doi.org/10.1371/journal.pone.0312537.g003

In Phase 1, using the pre-trained model, all teeth numbers and therapeutic identifiers of 1,029 AM individuals were detected and post-processed. In Phase 2, the difference score was calculated to estimate the similarity score in the PM-AM pairs. This difference score was evaluated by comparing the dental condition at each position. A score of 0 points indicated that the position and the state were the same, and a higher score indicated that the distance difference between the tooth conditions increased. During this process, a relatively large score, the maximum of which was 10 points, indicated teeth for which the positions were unavailable in the past to increase the difference. A penalty of 10 points was imposed when an unexplainable discrepancy occurred. Table 2 summarizes the calculation methods of the difference and similarity scores. The difference score was subtracted from 1 to finally obtain the similarity score.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. The Student’s t-test analysis comparing the groups with shorter and longer imaging time intervals.

https://doi.org/10.1371/journal.pone.0312537.t002

In Phase 3, similarity scores were calculated by 1:1 matching for dentition information of one anonymous DPR and all AM dentitions. After sorting the derived similarity scores in descending order, the top 20.0%, 10.0%, and 5.0% candidate groups (CGs) were extracted. The success rate was calculated after the rank position was checked to determine whether matched AMs existed in this group, as follows:

The same procedure as described above was conducted within each sex group. All aforementioned processes were fully automated based on a deep neural network.

Results

Discussion

The aims of this study were to construct a database of individuals’ dentition with DPRs, to use a novel method based on deep neural networks to identify individuals by recognizing their dentition changes using a pre-trained object detection network, and to evaluate the feasibility of this novel method. We found that the developed method was useful in detecting individuals by their dentition.

To date, different models for human identification have been proposed. The latest study [34] developed an identification system using DPRs with the Visual Geometry Group 16-layer model showing the highest accuracy, and other similar studies [35,36] developed more sophisticated models to better extract features that improve matching performance. A previous study confirmed the decisive role that the sorting method has in human identification while considering similarity [37]. A database search based on the basic assumption regarding dentition changes and sorting using similarity calculation was useful in human identification. All processes, however, including database construction and sorting by similarity calculation, were conducted manually in this study.

The significance of this study is that, rather than comparing the images, the dental information from DPR images was converted into objective information to prevent errors that can be generated from analyses using simple image comparison in advance. An excellent model was developed for this information conversion. The superiority of the model in automatically detecting natural teeth and dental treatment types has already been acknowledged through a preliminary study [21]. This study considerably contributes to the whole process of human identification that considers PM and AM similarities automated through deep learning, which includes advanced automatic identification of natural teeth and dental treatment types and the construction of an individual database based on this information. During the entire identification process, 32 teeth were classified into six states to increase the diversity of teeth arrangement. A step was added to compress and derive a high-probability group from possible candidates into a group. We believe that assigning directionality to tooth change increased the probability of identification.

To summarize, the top three probabilities of the target human identification were 83.2%, 72.1%, and 59.4%, which were obtained by respectively extracting the top 20.0%, 10.0%, and 5.0% of observations from the calculated similarity scores. The scoring was iterated 1,029 times, and the probabilities were sorted in descending order (Fig 3). Therefore, every anonymous PM was paired with the AM DPRs for the similarity scoring. The success rate differed between men and women. The success rate of the top 20.0%, 10.0%, and 5.0% CGs was 71.3%, 64.0%, and 52.0%, respectively, among men and 97.2%, 81.1%, and 66.5%, respectively, among women. These favorable success rates, especially when extracting the top 20%, underscore the potential of DPRs in practically identifying victims, even when other identifying features are unavailable or have deteriorated.

A possible reason the overall success rate was higher among women is that the dental information was extracted more accurately from women than from men by deep learning. The more precise extraction of dental information from women could potentially be influenced by diverse factors such as sexual dimorphism in teeth, the higher demand for orthodontic treatment among women, gender differences in oral health maintenance, and variations in periodontal disease progression between sexes [38,39]. The findings of our study could lead to more tailored approaches in practical forensic investigations, where gender-specific characteristics are considered during the identification process, thereby improving overall accuracy.

The difference in the similarity scores between the groups was statistically significant when the imaging time interval was 17.7 years. If the interval between the imaging times was <18 years, the interpretation is that the similarity between the two DPRs is high. When determining the cut-off value, a key aspect of our approach was to segment the imaging time intervals into intuitively data-driven 100-day intervals using ’day dummy’ variables, followed by regression analysis to examine differences between the two groups. To achieve a more refined determination of the cut-off value, alternative statistical methods, such as nonlinear regression analysis, could be considered.

As dental treatments become more common, it is very unlikely for a person to visit a dental clinic for the first time in 18 years and to take a DPR. Thus, it serves as an objective and pragmatic basis for supporting the effectiveness of the human identification algorithm based on similarity in the present study. In conclusion, an appropriate interval of <18 years between the imaging times for human identification is imperative when using the individual database. This insight is crucial for forensic experts as it suggests that the likelihood of successful identification decreases as the imaging time interval increases, emphasizing the need for timely collection and comparison of dental records in DVI scenarios.

Calculating the rank revealed that the performance was lower when 5.0% of CGs were extracted than when 10.0% were extracted. In this study, equally ranked individuals were assigned the same rank, and the next individual was ranked by adding the number of people in the preceding rank. To improve the performance of this model, further consideration for the handling of equal score processing is necessary.

In this model, the similarity decreases when the model is applied to AM images showing no restorations. For example, the condition with the worst outcome was an AM image of an individual with 32 natural teeth (including four third molars) with no restorations. Subsequent extraction of a third molar and treatment on several teeth may have caused poor results because of lowered DPR similarity. In instances in which the similarity score is too low or a discrepancy occurs, completely removing the CG could also be considered. However, in this study, CGs were not removed, but penalties were given: a penalty of 10 points was given to greatly increase the difference in tooth change during the step for deriving possible CG.

In the DPRs in which the pre-trained model was used, several unclear cases occurred during the dental treatment type detection stage. For instance, a restoration treatment such as a gold inlay through indirect restoration could be damaged or lost over time, and if the individual receives a direct restoration in the following treatment, the shape of the restoration would appear differently on the DPR, often resulting in errors when read by the network. Even when studied by specialists, ambiguous cases are often difficult to distinguish as to whether the tooth was natural or had received a restoration treatment using a resin.

A follow-up study is warranted to divide the prosthetic part as an indirect or direct restoration and to use a large number of samples for learning so that the neural network may distinguish between these two situations. Furthermore, human identification was conducted using dental information detected via deep learning only without considering the estimated age and the pathologic condition such as dental caries or periodontal lesion. Adding these pathologic conditions and the estimated date of birth of an individual would further improve the performance of reducing the CG. Adding conditions other than teeth arrangement and invariable anatomical structures such as the mental foramen, maxillary sinus, and anterior nasal spine would further increase the success rate of human identification.

Another critical consideration when establishing a more elaborate human identification system is addressing the inherent issues in AI model development. To mitigate the risks of data bias and imbalance, it is crucial to implement data augmentation and balanced sampling techniques, while also incorporating bias detection algorithms to ensure fairness across the model. Overfitting can be prevented by utilizing cross-validation methodologies. Incorporating interpretable algorithms or tools enhances the transparency of the decision-making process. Ethical considerations should be continuously evaluated and refined, with transparency maintained throughout the modeling and data collection stages. Ongoing updates and iterative retraining of the model are vital for sustaining real-world performance, and it is crucial to rigorously monitor the model’s efficacy on real-world data. Finally, regular monitoring for performance degradation due to environmental changes, coupled with routine updates, is essential for ensuring the model’s long-term sustainability.

The establishment of individual database using DPRs has various advantages, in addition to its application in the dental and medical fields. AI can integrate heterogeneous data domains such as demographic and clinical data, image data, biomolecular data, and social network data [40,41]. When a big database is ultimately established using DPRs, the DPR database would be highly reliable identifiers that could represent individuals, as fingerprints do.

Conclusion

This study demonstrated the feasibility of human identification regarding dentition change on DPRs by not only automatically detecting dental objects but also constructing a comprehensive database through an indexing system. The developed method enhances the accuracy of the identification process by surpassing traditional image comparison trends. This approach could effectively reduce the size of an AM CG to be reviewed, especially in DVI scenarios, while serving as a powerful tool for dental professionals and other stakeholders.

Supporting information