2.1. Ethics statement

The ethics committee of the Charité – Universitätsmedizin Berlin approved the analysis of Charité and EGZB data (EA2/184/21, Amendment vote Dec 2, 2024). From this analysis, the University and Geriatrics datasets were derived. Restricted conditions apply. Due to the retrospective nature of the study using standard care data, informed consent was waived. University data were anonymized through the institution’s Health Data Platform and subsequent further anonymization steps (e.g., removing irrelevant fields), while EGZB data were anonymized following their internal procedures. The data protection officers and the clinical trial office of the Charité (CTO-22–0838), for the University dataset, and, for the Geriatrics dataset, the EGZB advised on data protection rules to ensure compliance with those rules. Federal data protection officers were consulted for their advice.

2.2. Data

This study is based on 9 datasets (Table 2) from 6 public and 3 proprietary domains to evaluate synthetic data generation methods.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Detailed summary of all 9 datasets.

https://doi.org/10.1371/journal.pdig.0001522.t002

Diabetes: The Diabetes dataset (https://www.kaggle.com/datasets/akshaydattatraykhare/diabetes-dataset) is derived from the 2015 Behavioral Risk Factor Surveillance System (BRFSS) survey. It includes health indicators relevant to diabetes risk, the target feature, with a range of demographic and lifestyle attributes. Licensed under the CC0 Public Domain, this dataset allows unrestricted access for research.

AIDS Virus Infection: The AIDS dataset (https://www.kaggle.com/datasets/aadarshvelu/aids-virus-infection-prediction) includes data on HIV/AIDS infections for binary classification tasks. It comprises 23 health-related attributes with Infected being its target feature. Like the Diabetes dataset, is distributed under the CC0 Public Domain license.

MIMIC-IV-Ext-Fall-Prediction: For our research purposes, we derive a dataset from MIMIC-IV [26] and include 67 demographic and clinical features. We derive the target column representing inpatient fall incidents based on the patients’ diagnoses. Particularly, we filtered for diagnoses which contained “fall,” but excluded incompatible variants of fall for this task (see Listing A in S1 Appendix for more details). We refer to this dataset as MIMIC-IV-Ext-Fall-Prediction, following their naming conventions for derivatives of the dataset. Access to MIMIC-IV is restricted under the PhysioNet Data Use Agreement (DUA), requiring researchers to complete training and adhere to HIPAA guidelines.

Geriatric Hospital: The proprietary Geriatric Hospital dataset was curated for developing a fall risk assessment model. It includes 103 features of mixed types, with missing values reflecting real-world data limitations. Due to its proprietary status, access is restricted. The ethics approval for using this dataset for the purpose of this study is declared in Section 2.1.

University Hospital: The proprietary University hospital datasets were also curated for training a fall risk assessment model. It features 45 features tailored for binary fall risk classification. This dataset is also restricted for internal research to ensure patient confidentiality. The ethics approval for using this dataset for the purpose of this study is declared in Section 2.1.

Adult: The Adult dataset (https://archive.ics.uci.edu/dataset/2/adult), from the 1994 U.S. Census, contains demographic and income-related data widely used in benchmarking, with income being its target feature. It comprises 15 attributes and is licensed under CC BY 4.0, allowing open access for research.

Hospital Discharge: This dataset is a subset of the Texas hospital discharge dataset from 2013 containing 18 attributes describing patient demographics, disease severity state as well as costs associated with their hospital stay. The dataset is accessible as a public use file and can be requested from the Texas Department of State Health Services (https://www.dshs.texas.gov/center-health-statistics/texas-health-care-information-collection/health-data-researcher-information/texas-hospital-emergency-department-research-data-file-ed-rdf/hospital-discharge-data-public-use-data-file).

Heart Failure: The Heart Failure dataset includes routine data [27] from partner hospitals of the HiGHmed [28] consortium and contain 36 variables used to calculate the Bio Heart Failure (BIOHF) [29] and the Meta‐Analysis Global Group in Chronic Heart Failure (MAGGIC) [30] risk scores. We used the synthetized version of the dataset [31], which can be accessed upon reasonable request for research purposes.

Machiavellianism Test: The Machiavellianism Test dataset is a psychological test dataset to test about machiavellianism. The dataset contains 16 columns related to peoples demographic information and their answers to questions of the test. The data can be openly accessed from the psychometrics project (https://openpsychometrics.org/_rawdata/).

2.3. Models

In our evaluation we use 11 different models that represent a spectrum of commonly used generative methods for tabular data. For most models, we use the implementations provided by the SDV package [22]. Only for TabDiff, we follow the implementation of [32], for PATE-GAN and DP-WGAN, the implementation of [33], and for PrivBayes, the implementation of [34]. For most models, we used the recommended hyperparameters provided by the respective implementations without additional tuning. For the DP-based models, we treated the privacy parameter as a tunable hyperparameter and selected it according to the procedure described in the corresponding sections. Additionally, for DP-WGAN we tuned the number of epochs and learning rates.

2.4. Metrics

Quality metrics for synthetic data that capture all legally relevant aspects are an active topic of research [15,16,21]. Previous work has primarily distinguished between utility and privacy [15,16], where utility included metrics which measure fidelity. Fidelity measures how similar synthetic and real datasets are [6,15]. Utility assesses how useful synthetic data is for training a ML model on a downstream task. Privacy is the most critical metric in healthcare data [44] and measures the risks of exposing patients’ identity or their personal information. In this study, we aim at evaluating a broad spectrum of metrics for fidelity, utility, and privacy. To ensure replicability, we use well maintained libraries, such as SDMetrics [23], and extend these where needed with metrics computed using open source libraries, such as Autogluon [45] (for fidelity and utility), scikit-learn [46] and XGBoost [47] (for fidelity), and Anonymeter [24] (for privacy). We match the metrics with other studies, where possible (Tables B, C and D in S1 Appendix). All metrics are shown in Fig A in S1 Appendix.

2.4.3. Privacy.

Privacy is the most critical metric in healthcare data [44] and is evaluated using various approaches, many emulating different attack scenarios. Such attacks may include 1) re-identification attacks, where individual records or groups are matched between synthetic and real-world data to re-identify an individual, 2) attribute inference attacks, where adversaries try to gain information about unknown attributes of a target from the synthetic data, 3) membership inference attacks [55], where adversaries try to determine whether an individual was in the training data, and 4) data linkage attacks, where adversaries use the synthetic dataset to identify whether two disjoint datasets containing attributes from the original data belong to the same individual. In addition, researchers often use distance-based metrics compare synthetic and original records, with metrics such as Distance to Closest Record (DCR) [17,54], and Nearest Neighbor Distance Ratio (NNDR) [17] to quantify information leakage.

In this study, we use five types of privacy metrics: distance based metrics, privacy against inference (or attribute inference attacks), singling out, linkability and membership inference attacks. Each metric covers a different aspect of privacy protection, ensuring a comprehensive evaluation and connect our selection to prior work in Table D in S1 Appendix.

Distance metrics: The distance metrics evaluate privacy risks by measuring the distance between the original and synthetic data. Two metrics are used in this analysis: Distance to Closest Record (DCR) [17,54], which measures the Euclidean distance between each synthetic data point and the closest original data point, and Nearest Neighbor Distance Ratio (NNDR) [17], which compares the distance between each synthetic data point and its nearest original data point with the distance to its second-nearest original data point. The key outputs for DCR include the 5th Percentile of all DCRs, which is computed for the following dataset combinations: R&S (real and synthetic data), RR (within real data), and SS (within synthetic data). R&S represents the proximity of the closest 5% of synthetic data points to real data points, and the Mean DCR, which provides the average DCR values between both synthetic and original data. A lower 5th percentile DCR R&S suggests potential privacy risks, as it indicates that a subset of the synthetic data points may closely resemble real data. Similarly, NNDR is computed for the following dataset combinations: R&S (real and synthetic data), RR (within real data), and SS (within synthetic data). The primary output for NNDR is the 5th Percentile NNDR R&S, which identifies the closest 5% of cases. A high 5th percentile NNDR indicates that even the nearest synthetic data points maintain a sufficient distance from real data, reflecting stronger privacy protection. For both DCR and NNDR values, the larger the distance, the lower the privacy risk.

Privacy risk attacks: Evaluating privacy risk attacks on synthetic data provides insights into the level of protection it offers against privacy risks such as attribute inference, membership inference, linkability, and singling out (Fig 2). These insights are important, as the GDPR mandates that any effective anonymization technique (synthetic data in our case) must mitigate these risks to ensure strong data privacy protections [24].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Overview of privacy attack models, listing the attacker’s knowledge (Attacker has), objectives (Attacker Wants), and their Attack Strategy to gain information.

https://doi.org/10.1371/journal.pdig.0001522.g002

Membership Inference Attacks: Membership inference attack (MIA) [55] is a privacy attack in which an adversary attempts to determine whether a specific record was part of the training dataset of a machine learning model. In the context of synthetic data, MIAs aim to infer whether a record was used to train the synthetic data generator, and are difficult to evaluate for generative methods [4,56]. Therefore, we adopt a downstream-task threat model as a practical proxy for membership leakage, assessing whether the released synthetic data holds sufficient auxiliary information to enable membership inference on the original training records via a shadow-based black-box attack.

The assumption of such threats may be realistic in scenarios where synthetic data are released together with, or later used to train, accessible prediction services, benchmark models, or deployed clinical decision-support systems. An alternative attack scenario could involve an adversary that trains a black-box model on the synthetic data, mimicking the downstream black-box model.

To assess this privacy risk, we perform a classic shadow-based black-box membership inference attack and quantify its success using the Privacy Gain (PG) [19], defined through the true positive rate (TPR) and false positive rate (FPR):

Here, s = 1 denotes a member record originating from the original training split (used to train the synthetic data generator), while s = 0 denotes a non-member record from the original test split. The quantity TPR-FPR corresponds to the attacker’s advantage, such that higher PG values indicate stronger privacy protection. We assume a realistic threat model in which the attacker has access to the released synthetic data and black-box access to a target (downstream) model trained on the real data. The synthetic samples are used as auxiliary data to train multiple shadow models that approximate the input–output behavior of the target model. Predictions from these shadow models on their respective training and holdout splits are then used to train an attack classifier to distinguish member from non-member records. For each dataset, we train max(10, N/2000) shadow models, where N denotes the dataset size. Increasing the number of shadow models generally improves attack stability and performance. However, we limit this number to balance the performance with computational cost [55]. Finally, the trained attack model is evaluated on the real private dataset to quantify the resulting privacy gain. For MIA we use the Adversarial Robustness Toolbox [57], also supported in Synthius [36].

Linkability and Singling Out: Linkability and singling out [5] are privacy-related metrics whose evaluation is implemented in the Anonymeter framework [24]. Linkability is assessed under the assumption that there are at least two external datasets available to the attacker, that contain some of the original data attributes, and that these attributes are also present in the synthetic data. A low linkability score indicates high privacy. From a patient’s perspective, a “successful” linkability attack confirms that if patient’s age, gender, and area code from one dataset match a record in another dataset containing sensitive information, could potentially expose their private data.

Singling out evaluates the risk of identifying specific individuals in the original dataset based on unique combinations of attributes present in the synthetic data. This risk arises when rare or unique attribute patterns in synthetic data closely match records in the original data. Singling out is assessed through privacy attacks using targeted queries: univariate queries focus on a single attribute (e.g., age), and multivariate queries combine multiple attributes (e.g., age, gender, and location). The intuition behind this approach is that attributes, or their combinations, that are rare or unique in the synthetic data are likely to be rare or unique in the original data as well. In the evaluation phase of the singling out attacks, an attacker’s guess is considered correct if there is exactly one corresponding individual in the original dataset. Again, the closer the success rate of the main to the control or baseline attack, the better. A low singling out score indicates high privacy. From a patient’s perspective, a “successful” singling out attack confirms that there is only one female, X years old patient that lives in area 10188.

Attribute inference attacks: Attribute inference attacks [5] pose a threat to patient privacy by enabling attackers to deduce sensitive information from synthetic datasets. Studies such as [55,58] have shown how AI models can be vulnerable to these types of attacks. In this study, we use (a) Privacy Against Inference (PAI) from SDMetrics as well as (b) Attribute Inference Attack from the Anonymeter framework, to assess attribute inference. (a) PAI evaluates the risk of sensitive attribute disclosure under the assumption that an attacker has access to a subset of the original data. In our experiments we define a set of sensitive attributes for each dataset (Table 3). To perform the attack, the attacker may use different models, such as k-nearest neighbors (KNN), Naive Bayes (NB) classifier, Random Forests (RF) or Support Vector Machine (SVM), and ensemble of classifiers, as well as variations of the Correct Attribution Probability (CAP) algorithm. The higher the scores on these models, the lower the privacy risks. (b) With the attribute inference attack we perform as many attacks as there are attributes in a dataset - N, and not only on a predefined set of sensitive attributes. The goal is to treat every attribute as a sensitive attribute at one point, and based on the N attacks, we choose the highest risk score as a single point estimate for the attribute inference attacks. From a patient’s perspective, a successful” attack could mean that if a patient is X years old, and lives in area Y, an attacker could infer that they are likely to have a risk-factor Q.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Sensitive attributes considered across datasets. For Heart Failure, copd indicates the presence of Chronic obstructive pulmonary disease, hf_duration represents the heart failure duration in months, hf_gt_18_months indicates whether the heart failure duration is longer than 18 month, and nyha refers to the New York Heart Association score, indicating heart failure severity. Q4A captures the degree of agreement with the statement: “Most people are basically good and kind”.

https://doi.org/10.1371/journal.pdig.0001522.t003

3.1. No generative model excels for all datasets

In Fig 3 we present the aggregated (median) metrics, fidelity, utility and privacy, across models (panels a and b) and datasets (panels c and d). In panels a and c the absolute values of the aggregated metrics are presented, while in panels b and d we normalize these relative to the metric scores of the original data, as explained in Section 2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Overview of absolute metrics’ scores (a,c) and scores relative to original data (b,d) aggregated by fidelity, utility and privacy.

We abbreviate Machiavellianism Test as Mach Test, MIMIC-IV-Ext-Fall-Prediction as MIMIC, and GaussianMultivariate as GM for better readability. Comparing metrics across generative models (a, b), no model outperforms other models in all metrics. In contrast, comparing metrics across datasets (c, d), we see larger and significant () differences between datasets. In panel (c), the experiments based on the original data are marked with an asterisk. For (b) and (d), values above 1.25 are capped at 1.25.

https://doi.org/10.1371/journal.pdig.0001522.g003

In Fig 3(a), we show that no model dominates across all three metrics. While some models exhibit positive outliers for specific datasets and metrics, such as utility, we do not observe consistent trends across all three metrics.

Comparing all metrics aggregated by dataset, Fig 3(c), the variance is greater between datasets than within datasets, especially in terms of privacy (). This suggests no clear dominance of any model over a specific dataset. Moreover, to investigate the impact of these differences, we compare these scores to the original data Fig 3(b) and Fig 3(d), where these differences are even less pronounced. To support these findings, we perform pairwise Dunn tests with Bonferroni correction for all three evaluation dimensions: fidelity, utility, and privacy. For the statistical analysis, we use the aggregated median fidelity, utility, and privacy scores computed for each dataset-model pair, resulting in a matrix (9 datasets × 11 models) per metric. To compare datasets across each metric (fidelity, utility, and privacy), we use the eleven model scores available per dataset (N = 11).

We emphasize that each dataset-model pair was evaluated only once (single training run without repeated random seeds or different hyperparameters). However, we repeated a couple of multiple-seeds experiments for two representative datasets - Adult and AIDS. The results from the different runs align with our main findings. More details will follow in Section 3.2.

We find most statistically significant differences at between datasets in all fidelity, utility and privacy and some models differ in fidelity, as also specified with vertical colored lines in Fig 3. For details, we refer to Tables N, O, P and Q in S1 Appendix.

3.2. Exploring metrics to guide model selection

In real-world application scenarios for synthetic data, one of the most important questions is which generative model is optimal for a given use case. Our findings in Section 3.1 show that no single model excels in all metrics for any dataset. This implies that for every dataset a new model selection procedure needs to be performed. Finding the best model without actually evaluating all generative methods would be advantageous as it reduces the time to find the optimal generative model. Following this line of thought, we investigate how the variance of the metrics can be summarized by a linear low-dimensional manifold using Principal Component Analysis (PCA) on the 33 evaluation metrics scores to understand their importance on a 2-dimensional space (Fig 4). Our goal is to explore whether such analysis could help future generative model selection. We focus specifically on the first two Principal Components (PCs), which account for 45% of the total variance. To assess which aspects of individual metrics are captured by the main variance directions, we show the correlations between metrics and the first PC in Fig 5 (also x axis in Fig 4, and the second PC in Fig 6 (also y axis in Fig 4). This first PC already captures the main trade-off between privacy: most privacy metrics are positively correlated with the first PC, and most fidelity/utility are negatively correlated with the first PC. The second PC captures another aspect of the trade-off between utility and some privacy (inference attack, membership inference attack and singling out) metrics, which are positively correlated with the second PC, and all other privacy/fidelity metrics, which are negatively correlated with the second PC (Fig 6). Inspired by prior research [4,19], we investigate the trade-offs between fidelity/utility and privacy across different models and datasets. In Fig 4 the first and second PC are shown on the x and y axes respectively, illustrating how models and datasets behave along these dimensions. We notice that PATE-GAN, DP-WGAN and PrivBayes are mostly located on the positive direction of the first PC, ranked as the most private models. This is expected as PATE-GAN, DP-WGAN and PrivBayes are models that explicitly enforce differential privacy. Note that a similar DP mechanism could be added to all other models, too. What Fig 4 also shows is that in general, these models perform worse than all other models in terms of all non-privacy related metrics, hinting at the trade-off between utility and privacy.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Trade-offs between privacy, utility, and fidelity as captured by the first (x-axis) and second (y-axis) principal component (PC) of variance of all metrics.

On the x-axis (first PC), fidelity and utility metrics are negatively correlated with the component ← , while privacy metrics are positively correlated → , capturing the trade-off between (fidelity, utility) and privacy. On the y-axis (second PC), utility and the distance-related, inference and singling out privacy metrics correlate positively ↑ , whereas most fidelity and some privacy (against inference) metrics correlate negatively with the PC ↓ . The models form clusters associated with datasets in the principal component space, suggesting that variance related to datasets is larger than that related to model performance. The experiments ran on the Original data are circled for every dataset.

https://doi.org/10.1371/journal.pdig.0001522.g004

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Trade-off between privacy and fidelity/utility as captured by the strongest variance direction - Principal Component (PC).

The first PC is mainly positively correlated with privacy metrics, and mainly negatively correlated with fidelity and utility. For readability purposes, fidelity is shorten to F, utility is shorten to U and privacy is shorten to P.

https://doi.org/10.1371/journal.pdig.0001522.g005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Trade-off between utility/fidelity and privacy.

The second largest variance direction - PC, is mainly positively correlated with the utility metric, new row synthesis as fidelity metric, and privacy distanced-based metrics as well as singling out and inference attack, and mainly negatively correlated with the privacy PAI and distance-based metrics, and all other fidelity metrics like KL divergence and Propensity score. For readability purposes, fidelity is shorten to F, utility is shorten to U and privacy is shorten to P.

https://doi.org/10.1371/journal.pdig.0001522.g006

Another aspect reflecting the results in Fig 4 is that within each dataset the models are clustered together. Some dataset clusters stand out, for example, Adult, for which models across the board often perform better in terms of fidelity and utility than in privacy (in the upper left quadrant); another example is the dataset from the Geriatric hospital in the lower right quadrant, with stronger privacy/fidelity metrics. Within every dataset ARF and TABDIFF are mostly on the left side, thus they rank better in terms of fidelity/utility. However, these observations require quantitative validation, which we provide in the following paragraph.

Can PCA be used to guide model selection? Evaluating all metrics on all models to find the optimal generative model might not always be feasible. The results in our PCA analysis suggest that maybe the much simpler trade-off between privacy and fidelity/utility as captured by the first PC allows to support model choices well enough. And computing the first PC from a subset of metrics to find the optimal generative model could increase computational efficiency.

To assess the potential of such a simplified trade-off, we consider two scenarios: a) a model selection where privacy is considered most important (e.g., cross-institutional or public compliant sharing of medical data) and b) a model selection procedure where utility/fidelity is prioritized (e.g., clinical model development within controlled environments). The first case a) could be one where patient data is shared outside a secure infrastructure. The second case b) could be one where the recipient of the synthetic data is a researcher who is tasked with developing the best ML model for a given task with the synthetic data. In case b) the data is not shared publicly and the individual researchers can be held accountable for their usage of the data.

To assess whether the first PC captures the trade-offs between all relevant metrics well enough to arrive at the same best performing model, we compare rankings derived from the first PC with those obtained from median metric scores (Table 4). For each dataset, models are ranked according to their first PC values (where higher scores indicate more private models) and by their median privacy and fidelity/utility scores, respectively. To quantify how similar the rankings obtained with the simpler first PC are to rankings obtained with the full suite of metrics, we computed Spearman’s rank correlation coefficient. We find high alignment between the simplified PCA based model selection compared to the model selection based on all metrics in 8 out of 9 (89%) of the datasets (Table 4) in terms of fidelity/utility and 5 out of 9 (55%) in terms of privacy. These results suggest that the simpler PCA can be a meaningful descriptive summary of the dominant trade-off between fidelity/utility and privacy scores for at least part of the datasets considered in our experiments, and may serve as a practical guideline for identifying models with favorable utility/fidelity characteristic. We further assume that evaluating a larger number of datasets could strengthen these observations.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Spearman correlations between PC-based rankings and aggregated metric rankings. For every dataset we rank the models based on the 1st PC and a) on their median fidelity/utility metrics and b) median privacy metrics. We correlate these with the corresponding PC metrics and find that the correlations are high for the majority of the datasets, in particular 8 out of 9 data sets in the fidelity/utility use case; this indicates that the 1st PC alone yields model rankings similar to those computed on all metrics. Such correlations are less common for privacy.

https://doi.org/10.1371/journal.pdig.0001522.t004

To further assess the stability of PCA-based rankings for model selection, we conducted a bootstrap analysis with 1,000 resampling runs per dataset. We randomly removed 10%, 20%, 30%, and 50% of evaluation metrics while always retaining the utility score (represented by F1-macro), and again recomputed the PC-based and aggregated rankings (Fig 7). This assesses the PC-based rankings robustness with respect to different metrics sets.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Spearman rank correlations between PC-based model rankings and rankings derived from aggregated median metric scores under metric subsampling.

For each dataset, we perform 1,000 bootstrap runs removing 10%, 20%, 30%, and 50% of evaluation metrics, while always retaining the utility metric (F1-macro). Boxplots summarize the resulting correlation distributions.

https://doi.org/10.1371/journal.pdig.0001522.g007

Overall, PCA-based rankings exhibit strong stability under moderate metric subsampling, with most datasets maintaining high rank correlations even when up to 30% of metrics are removed. As the fraction of removed metrics increases, the variability of the rank correlations widens, reflected by a greater number of outliers and occasional negative correlations. We support these findings with the dataset-wise results in Fig B in S1 Appendix. We find that the stability of the correlations under different metric supsampling varies among datasets. Datasets such as AIDS, Adult, and Heart Failure remain highly robust even under 50% metric removal, whereas others, including Hospital Discharge, Machiavellianism Test, MIMIC-IV-Ext-Fall-Prediction and University, show increased sensitivity at higher metric removal rates.

An additional analysis includes regenerating and reevaluating synthetic data under multiple random seeds for two representative datasets - Adult and AIDS (Fig C and D in S1 Appendix). We perform the same PCA analysis on both datasets separately and observe that the first principal component again reflects the main trade-off between fidelity/utility in one direction and privacy in the opposite direction. Models such as DP-WGAN and PATEGAN are positioned more strongly along the privacy direction, whereas ARF and TABDIFF are more prominent on the fidelity/utility side, with the remaining models generally located in the middle. This observation in line with the main findings (Fig 4). Furthermore, we compute Spearman correlations between the first and second principal component loadings from the seed-based runs and the original loadings(Fig 5) and observe correlations of for both datasets on the first principal loading, whereas the second loading shows lower correlation scores (Table R in S1 Appendix). The repeated experiments show model-wise stable behavior for most models. We observe some instabilities for DP-WGAN in both datasets, and some outliers for TVAE in AIDS.

Finally, we evaluate the sensitivity of the first PC rankings to different data normalizations. We recomputed the PCA using both Minmax and Z-score normalization and compared the resulting rankings to those obtained from the original PCA on the raw metric scores. Table 5 shows that the rankings remain highly correlated under different normalization strategies and that the PC-based rankings are not sensitive to metric scaling.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Spearman correlations (median and 95% confidence interval) between PC-based model rankings computed from raw metric scores and those obtained after Min–Max and Z-score normalization. High correlations indicate that the rankings are largely insensitive to metric scaling.

https://doi.org/10.1371/journal.pdig.0001522.t005

These findings suggest that the first PC can provide a meaningful descriptive summary of the main fidelity/utility and privacy trade-off for several of the datasets included in this study. Nevertheless, we also observe that in some dataset this rank alignment is weaker, indicating that depending on the application scenario, a more careful multi-metric evaluation may be necessary, especially in settings where privacy is of higher concern. Therefore, our PCA-based analysis provides a structured and transparent descriptive framework to summarize multi-metric trade-offs that could guide the initiation of model selection decisions.

3.3. Can dataset meta-features predict metrics?

Next we explore an alternative approach to guide model selection using dataset specific characteristics, or meta-features. This idea has also been investigated in the field of AutoML [60], where dataset meta-features have been used to explore the best ML models in predictive ML tasks. Here we follow a similar approach yet with simpler dataset meta-features. We investigate linear and non-linear dependencies between the dataset meta-features listed in Table 2 and all synthetic data metrics explored in our analyses. We perform the correlation analysis using the relative scores introduced in Section 2. This normalization removes dataset-specific effects, such as the difficulty of downstream task, measured with utility, to ensure that the correlations reflect the differences between the generative models rather than the datasets themselves.

For linear dependencies we computed correlation coefficients between the dataset meta-features and synthetic data metrics. We find no clear relationship between dataset features and generative model performance (Table 6).

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. No dataset meta-feature is strongly correlated () with model performance across fidelity, utility, and privacy dimensions. Spearman correlations between dataset meta-features and the three evaluation metrics reveal only weak trends—fidelity shows mild positive associations with the number of features, percentage of boolean features, and class imbalance. Utility shows mild positive correlation with the number of samples, and the percentage of categorical features, and minor negative correlations with the number of features and the percentage of boolean features.

https://doi.org/10.1371/journal.pdig.0001522.t006

For non-linear dependencies, we used a decision tree model to evaluate how well synthetic dataset metrics can be predicted from non-linear combinations of dataset meta-features. We trained two models, one for predicting (a) best privacy generative model, and another for (b) best fidelity/utility generative model. The input features corresponded to the dataset meta-features, and the target labels for each dataset represented the best-performing generative model in terms of (a) privacy and (b) fidelity/utility. Labels were assigned by first aggregating (a) privacy metrics and then (b) fidelity/utility metrics within each dataset, ranking the models, and selecting the top-ranked one as the dataset label in both scenarios. Using these curated datasets, one with (a) privacy focused labels, and (b) fidelity/utility focused labels, we trained the decision trees with 5-fold cross-validation. In both cases, dataset meta-features were not predictive of synthetic dataset metrics, yielding median macro-F1 scores of 0.0 for the fidelity/utility-based model and 0.074 for the privacy-based model. These results suggest that in our experimental setting, dataset-level meta-features alone are not sufficient to reliably guide model selection. However, this analysis is statistically underpowered due to the limited number of datasets (N = 9) and the restricted set of simple meta-features considered. Therefore, the absence of predictive power in our experiments should not be interpreted as a general negative result on meta-learning for synthetic data model selection. More diverse meta-feature representations and a bigger dataset collection may provide an improved meta-feature-based selection guidance.