Variable labels and data sheet description.

Our approach matched variables based on information from data dictionaries, such as variable labels and data sheet descriptions. Table 1 provides examples of matched variables from the GERAS-EU and GERAS-JP studies (see Table A in S1 Text for more examples). Variable labels provide concise descriptions of the corresponding variable names. Both studies organized their variables in separate data sheets, with each data sheet representing a specific type of variables (e.g., demographic variables) or variables derived from a specific survey. Each data sheet was accompanied by a short description in the data dictionaries. For example, the GERAS-JP study kept variables associated with the MMSE test in a data sheet with the description “Mini-Mental State Examination (MMSE) per visit”. In this data sheet, the variable “MMSE_Q8” recorded the response to the eighth question in the MMSE [27].

Key word extraction from derivation rules.

The data dictionaries of both GERAS-EU and GERAS-JP studies also contain a field called variable definition or derivation rule (see Table B in S1 Text for examples). This field provides information about how the variables were defined or derived, such as the values of categorical variables and the derivation rules of variables. Derivation rules provide additional information beyond variable labels and data sheet descriptions, which can be valuable for variable matching. However, not all variables have derivation rules. In addition, there is significant variation in the length and content of derivation rules. Therefore, we utilized information from derivation rules exclusively for ensemble learning.

We incorporated information from derivation rules into variable matching as follows. First, if a derivation rule contained more than 20 words, KeyBERT [33] was employed to extract key words of up to 15 words to represent the entire text. KeyBERT utilizes BERT (Bidirectional Encoder Representation from Transformer) [34] embeddings to identify sub-phrases in a text that are most semantically similar to the original text. If the derivation rule contained 20 or fewer words, the entire text was retained. Second, for each variable, we concatenated the variable label with the derivation rule (or key words extracted from the derivation rule) to create input for the individual NLP methods. This treatment ensured that variables without derivation rules had non-empty input for NLP.

Large language model-based methods.

We evaluated four LLMs in variable matching: E5, MPNet, MiniLM, and BioLORD-2023.

The E5 (EmbEddings from bidirEctional Encoder REpresentations) is a text embedding model that enhanced its training process through weakly supervised contrastive pre-training [35]. The key idea of E5’s contrastive pre-training is optimizing text embeddings so that they will bring relevant unlabeled text pairs closer together and push irrelevant text pairs further apart within the vector space of embeddings [36]. In E5, relevant text pairs were sourced from diverse platforms, including Reddit (posts and comments), Stack Exchange (questions and upvoted answers), English Wikipedia (entity names and passages), scientific papers (titles and abstracts), and Common Crawl web (titles and passages) and selected via a consistency-based data filtering technique [35]. These relevant text pairs served as positive examples; text from different relevant pairs formed irrelevant text pairs (i.e., negative examples). During pre-training, E5 enhanced existing LLMs, e.g., the BERT models by leveraging the large amount of newly collected text pairs and contrastive learning. The model was then fine-tuned using three labeled datasets: NLI (Natural Language Inference), MS-MARCO passage ranking dataset [37], and NQ (Natural Questions) [38] datasets. E5 outperformed existing embedding models on both BEIR [39] and MTEB [40] benchmark datasets that were used to evaluate a variety of text embedding tasks. This study used the E5_large model, which is built on the large BERT model Bert-large-uncased-whole-word-masking model. The E5_large_v2 model was chosen due to its superior performance on benchmark datasets, particularly in the semantic textual similarity (STS) tasks [35].

In addition, we evaluated two LLMs developed using the Sentence Transformers (also called SBERT) framework [41]. Built on Siamese and triplet networks and contrastive learning techniques, SBERT aims to generate semantically meaningful embeddings from sentences or short paragraphs while achieving higher computational efficiency than BERT [41]. SBERT incorporates pre-trained LLMs into their low-level building blocks. We evaluated two LLMs, MPNet and MiniLM, incorporated into SBERT [42].

The MPNet (masked and permuted language modeling) model unifies mask language modeling from BERT [34] and permuted language modeling from XLNet [43]. In addition, MPNet utilizes auxiliary position information (i.e., the tokens’ positions in the original, non-permutated input sentence) to improve the consistency of the model’s input representations between pre-training and fine-tuning [43]. The All_MPNet_base model was fine-tuned from the MPNet-base model using contrastive learning on over 1 billion text pairs sourced from diverse datasets (e.g., Stack-Exchange, MS-MARCO, NQ) [42]. Among all SBERT models, the All_MPNet_base model has demonstrated superior performance on the Sentence Embedding task (14 datasets) and the Semantic Search task (6 datasets) [42].

The MiniLM model employs a specific knowledge distillation technique, deep self-attention distillation, to compress large transformer-based models [44,45]. Knowledge distillation compresses a large, complex model (teacher model) into a smaller, simpler model (student model) with fewer parameters by minimizing the difference between intermediate features of the two models (e.g., self-attention distributions in Transformer models). It has been shown that the student model obtained through using this technique can maintain similar test accuracy as the teacher model across tasks such as image and speech recognition [46]. MiniLM employs distillation on the self-attention module of the final transformer layer of the BERT base model, resulting in a 12-layer student model [45,47]. This reduction in parameters enhances fine-tuning efficiency. In this study, we used All_MiniLM_L12 model [41,42]. Initialized from the pre-trained MiniLM (microsoft/MiniLM-L12-H384-uncased) model, All_MiniLM_L12 was fine-tuned with a contrastive objective using a dataset of 1 billion text pairs, including NQ and SQuAD2.0 [48]. This model is selected due to its comparable performance in sentence embedding and semantic search tasks when compared with the All_mpnet_base_v2 model [42].

We also evaluated BioLORD-2023, an LLM that leveraged the Unified Medical Language System (UMLS) knowledge graph to improve the generation of embeddings for text containing medical concepts [49]. The BioLORD-2023 model is based on the MPNet model (as described previously) [41] and leverages the LORD (Learning Ontological Representations from Definitions) strategy [50,51] to further refine model training to enhance semantic representations of biomedical text. The training process of BioLORD-2023 involved three phases: contrastive learning, self-distillation, and weight-averaging. The contrastive learning phase implemented the LORD strategy [51]. Specifically, the model was trained on paired medical concept names and definitions to learn embeddings, ensuring that each concept’s name was close to its definition(s) while remaining distant from definitions of other concepts in the embedding space. The self-distillation phase aimed to mitigate performance degradation in measuring general-purpose semantic similarities, which arose due to intensive contrastive learning on medical concept names and definitions—a phenomenon observed in the previous BioLORD-2022 model [51]. In this phase, the knowledge acquired by the contrastive model (i.e., the model derived from the contrastive learning phase) was distilled into the base model in a supervised manner. Specifically, the concept embeddings and definition embeddings learned by the contrastive model were first averaged and then reduced to 64-dimensional vectors through principal component analysis. These reduced embeddings served as the learning objective for the base model, which was trained to predict them through a randomly initialized linear projection layer. Due to the random initialization of this projection layer, the training process produced multiple slightly different fine-tuned self-distillation models. To enhance robustness, the weight-averaging phase applied a strategy called “model soup” [52] to average the parameters of the fine-tuned self-distillation models to derive a single, ensembled model. BioLORD-2023 leveraged two biomedical knowledge graphs (ontologies)—the UMLS [53] and the Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT) [54]—to enhance model training. Biomedical concepts and their definitions from the broad-coverage UMLS were used for contrastive learning. The training data were further augmented by: (1) extending concept definitions using template-based descriptions sampled from UMLS relations (e.g., is-a, used to treat, is the synonym of) and (2) incorporating 40,000 definitions from the Automatic Glossary of Clinical Terminology (AGCT) [55], which were automatically generated using the GPT-3.5 model and the SNOMED-CT ontology. BioLORD-2023 outperformed two BERT-based models (which incorporated knowledge from UMLS or biomedical literature via domain-specific pre-training or fine-tuning) and BioLORD-2022 (which incorporated knowledge from UMLS via contrastive learning) on both general-domain and biomedical-domain STS tasks [49].

Fuzzy matching.

Fuzzy matching methods utilize dynamic programming and edit distance [56] (e.g., Levenshtein distance) to assess lexical similarities between text strings. Edit distance measures the number of operations needed to transform one text string into another [56]. We computed the edit distance between the label of a GERAS-EU variable and the label of each GERAS-JP variable. Variables with a smaller edit distance were considered similar. We implemented the fuzzy matching method by using the Python package RapidFuzz [57], which incorporates a variety of edit distance scoring functions originally developed in the Fuzzywuzzy Package [58]. Prior to applying fuzzy matching, we preprocessed the variable labels by removing punctuations and stop words, converting all letters to lowercase, and stemming the words. We compared the performance of various fuzzy matching scoring functions implemented in the RapidFuzz package in a preliminary experiment and selected the top-performing function, the token-set-ratio function, for this study. The token-set-ratio function is an extension of the token-sort-ratio function. The token-sort-ratio function tokenizes the preprocessed strings, sorts the tokens alphabetically, and computes the Levenshtein distance between the sorted strings. The token-set-ratio function eliminates duplicate tokens within each sorted string before comparison.

Random forest classifier.

The Random Forest classifier is a supervised machine learning model that utilizes an ensemble of decision tree classifiers to generate predictions. The classifier takes features associated with a pair of EU-JP variables as its input and outputs a class label (1: matched variables, 0: unmatched variables) and the associated probabilities. The training of the Random Forest classifier involves the creation of an ensemble of decision trees that classify the input from the training data. Each tree in the random forest builds on a subset of training instances randomly sampled from the complete training set without replacement, as well as a subset of features randomly selected from the entire feature set [59]. Introducing randomness enhances model robustness and helps mitigate overfitting. The construction of a decision tree involves splitting nodes iteratively from top to bottom. Each node is split based on a specific machine learning feature, with the feature and the corresponding splitting rule determined by criteria such as Gini impurity and information gain. For continuous features, the split rule may check whether the feature value is within a certain range. For categorical features, it may check whether the feature value is equal to a specific category. The goal is to achieve the greatest reduction in impurity or the largest increase in information gain with each split. The node-splitting process continues until certain termination criteria are met, such as reaching the maximum tree depth, the minimum number of samples required for a split, or the minimum decrease in impurity. When applying a trained Random Forest classifier to a new data instance, each decision tree that makes up the classifier is applied to the data instance respectively. At each node of a decision tree, the corresponding split rule will direct the data instance to a certain branch (i.e., a specific child node) under that node. After traversing several nodes in the tree, the data instance will reach a leaf node and receive its classification label (which is the label of the majority training instances that reach the same leaf node during model training). A final classification label is determined based on the aggregated voting results from all decision trees. In addition, the classifier will output a probability indicating how likely the data instance (in this case, a pair of variables) is positive, i.e., represents a match. These probabilities were then used to rank the candidate JP variables for each EU variable. Fig 2 provides an overview of the training and test procedures of the Random Forest model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Training and evaluation of Random Forest (RF) classifier in a single trial^a.

^a The Random Forest classifier was trained and evaluated in 50 trials, with each trial having a different random split of the training and test datasets. Each test set had slightly varying ratios of positive (i.e., matched EU-JP variable pairs) to negative (i.e., unmatched EU-JP variable pairs) cases because some EU variables were manually aligned to multiple JP variables (see details in sub-sections Training and test datasets and Model comparison).

https://doi.org/10.1371/journal.pone.0328262.g002

Training and test datasets.

The datasets utilized in this study were created in the following way.

First, we manually matched a subset of variables from GERAS-EU and GERAS-JP studies, which served as the ground truth variable pairs. Multiple steps were taken to ensure accurate matching. At the first step, EU variables were assigned to five co-authors, who had training and work experiences in epidemiology, statistics, or bioinformatics, for preliminary matching with corresponding JP variables. The variables were matched based on both variable definitions (i.e., derivation rules) and variable values. Challenging cases were discussed within the research team, including a senior co-author with expertise in epidemiology and health informatics. We categorized the matching results into three groups: no match, single match, and multiple matches. For example, the variable “LVLOCLNM” (which records the living area, urban or rural, of the participant) in the GERAS-EU study was matched with multiple variables such as “C_LIVLOCCD” and “LIVLOC” in the GERAS-JP study, because it was included in multiple data sheets under different variable names in GERAS-JP. At the second step, the five co-authors validated and corrected matching results from the first step, with each assigned a different subset of variables. The validation results and corrections (along with a justification for any corrections) were documented. In the last step, one co-author reviewed all corrections to ensure their validity. Through this 3-step procedure, 438 pairs of EU-JP variables were manually matched, which included 347 unique EU variables (68 of which have multiple true alignments).

Second, we created the training and test datasets in two steps (Fig 2). First, we treated the 347 unique EU variables as source variables that need to be aligned with the target variable(s) in the GERAS-JP study and randomly divided these variables into training and test (4:1) sets. Each EU source variable had 1322 JP candidate variables to match, which constituted 1322 variable pairs. These variable pairs contained only one or few positive instances (matched EU-JP pairs) but many negative instances (unmatched EU-JP pairs). Second, we down-sampled the negative instances to mitigate the adverse effects of data imbalance on model training. Specifically, for the EU source variable in each positive instance, we randomly selected 200 JP variables that did not match this EU variable to form 200 EU-JP variable pairs as negative training instances. In the test set, however, we included all the negative instances. This resulted in about 351 positive instances (associated with 277 unique EU variables) and 70,200 negative instances in the training set and 87 positive instances (associated with 70 unique EU variables) and 92,453 negative instances in the test set in each trial of the machine learning experiment (detailed in the section Experimental settings).

Class label and machine learning features.

The class label is binary-valued, with 1 indicating manually matched EU-JP variable pairs and 0 indicating unmatched EU-JP variable pairs. Features utilized by the Random Forest model came from two sources: similarity scores generated by five individual NLP methods (E5, MPNet, MiniLM, BioLORD-2023, and fuzzy matching) and other information extracted from data dictionaries. Each NLP method generated three similarity scores for an EU-JP variable pair by using (1) data labels; (2) data sheet descriptions; and (3) data labels and key words extracted from derivation rules as the respective input. For example, the EU variable “DIAGDT” (representing “Disease diagnosis date”) and the JP variable “ADDIADT” received three similarity scores from the E5 model calculated based on variable labels (0.98), data sheet descriptions (0.76), and data labels plus key words from derivation rules (0.90) respectively. Other features include the length of variable labels, the length and the absence of derivation rules for both the EU and JP variables. For example, for the EU variable “DIAGDT”, we had three features: 3 for the length of the variable label measured in words, 30 for the length of the derivation rule in words, and “False” for the absence of derivation rules (see Table B in S1 Text). In total, 15 similarity scores (three generated by each NLP method) and 6 additional features (3 for each variable in the variable pair) were used as machine learning features.

Model development.

In this study, we utilized the Random Forest classifier from the Scikit-learn package [60]. We developed the model using the training set and tuned the following hyperparameters through 5-fold cross validation and grid search: (1) the number of trees in the forest; (2) the maximum depth of each tree; (3) the criteria used to assess the quality of a tree node split; (4) the minimum number of samples required for each split; and (5) the number of features considered at each splitting. Other hyperparameters were set to their default values. We used the hit ratio (HR) at top 30 and mean reciprocal rank (MRR; see section Evaluation metrics) to select the best hyperparameters.

Model comparison.

We first compared the performance of the five individual NLP methods on the whole dataset. We then compared the performance of the Random Forest classifier and the best NLP method on the test sets from 50 trials. The 50 trials were conducted by randomly splitting training and test datasets 50 times (see section Training and test datasets). We used paired t-tests to compare the performance of the two models across the 50 trials, with the null hypothesis stating that there is no significant difference in performance metrics between these two methods. Five metrics were used to evaluate model performance, including the top 30, 20, 10, and 5 HR (HR-30, HR-20, HR-10, HR-5) and MRR.

Feature importance analysis.

To understand which features contributed most to the performance of the Random Forest model, we estimated feature importance using data from the 50 trials and the permutation importance method [63]. For each feature in a trained model, permutation importance estimates its contribution by randomly shuffling the feature’s values in a held-out dataset and measuring the subsequent decline in model performance on this dataset. In this study, we estimated feature importance by averaging the model’s performance decline over the test sets from the 50 trials, measured by HR-5, HR-10, and MRR. We selected these three metrics because they were more sensitive to permutation on individual features in the Random Forest classifier compared to HR-20 and HR-30.

We compared feature importance using two methods. In the first method, we ranked features using their mean importance scores over 50 trials. In the second method, we first ranked the features within each trial based on their importance scores and then compared their average ranks across the 50 trials.

In addition, we conducted feature ablation analysis to assess whether removing a specific type (i.e., subset) of features from the Random Forest model would affect the model performance. We categorized the features into three types: LLM-derived features, fuzzy matching-derived features, and other features. We measured the average decline in model performance after removing each type of feature over 50 trials.