Overview of SynthCraft

Designed to act as a “human-in-the-loop” partner, SynthCraft guides users through an interactive, step-by-step process (Fig 1). First, SynthCraft characterises the real dataset, identifying key features and the data structure, as well as performing exploratory data analyses. SynthCraft then engages the user to consider the most appropriate synthetic data generation methods for their circumstances, explaining the different strengths and weakness of alternative approaches, before invoking Synthcity to generate synthetic data itself [9]. Synthcity is an open-source package with extensive community engagement that provides a standardised interface for accessing and evaluating a comprehensive array of generators for any synthetic data use case – from synthetic data generation to managing privacy, fairness, domain adaptation and image generation [9] (Table A in S1 Text). Together, the user and partner compare the generated data and iteratively refine analyses based on user feedback. The process has been designed to minimise ambiguity, ensure methodological rigour, and quality-assure the generated synthetic data. User prompts and any code generated by SynthCraft can be saved directly whilst, on completion, a structured report detailing all steps taken, decisions made, and code run, is produced to support transparency and reproducibility.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of SynthCraft.

(A) SynthCraft is a modular framework consisting of large language model (LLM) agents with access to tools (SynthCity, Python) linked with working memory and underpinned by a reasoning system for in-context learning. Interaction with the user is through a natural language interface. An illustration of SynthCraft’s synthetic data generation pipeline is shown in (B) alongside a schematic of how the episodic multi-armed bandit reasoning approach works (to the right of the vertical line) [15,16]. At each stage of the pipeline, from data preparation and developing an analysis plan through evaluating the quality of the synthetic data generated, multiple intermediate steps may be needed. In the illustration, we have simplified for clarity, but at each step the agent has a particular state, reflecting progress towards completing the relevant task, can select appropriate actions, and then receives feedback either from interaction with the user, external tools invoked, or self-reflection [15,16]. The number of episodes – or steps – will vary depending on the use case, the dataset, and the interaction with the user. Though presented as a sequential pipeline, SynthCraft may need to return to earlier stages; for example, to trial alternative synthetic data generators if the quality of the synthetic data generated for a given task is insufficient at the evaluation stage. Subfigures adapted from refs [15,16].

https://doi.org/10.1371/journal.pdig.0001290.g001

SynthCraft is a modular LLM framework, in which a coordinating LLM agent controls worker agents with the ability to generate code and run specialised synthetic data generation tools. This framework includes several important guardrails: first, the LLMs are not responsible for writing code to generate or evaluate synthetic data, instead using Synthcity (version 0.2.12). Second, we protect against the introduction of errors at the point of generating synthetic data by ensuring the human user is asked to validate key parameters before Synthcity is invoked. More broadly, the user is kept in the loop throughout, can ask questions at any point, and can revert steps. We used GPT-5 from OpenAI (model: GPT-5; API version: “2025-08-07”; temperature: 0.5) on a secured instance of Azure as the underlying LLM in these analyses, but an alternative LLM could be simply interchanged to underpin the agents in this framework. The LLMs themselves are not privy to the real or synthetic data at any point, only prompts – the instructions provided by the user – whilst all analyses are undertaken on a user’s own device.

The collaborative, sequential, decision-making process between the user and SynthCraft uses a reasoning system trained using multi-armed bandits, a type of reinforcement learning designed for complex multi-stage optimisation problems (Fig 1) [15,16]. Each stage in the process by which synthetic data are generated – which are encoded as fixed logic – can be considered as an episode consisting of one or more steps. The steps necessary for each episode will develop based on the needs of the user and characteristics of the data – i.e., adaptive learning [15]. For example, because the performance of synthetic data generation methods differs between use-cases and datasets, the generative step may require several iterative cycles before completion. At each step, an action is taken, such as running a synthetic data generator tool or producing a report. Each action is associated with both a cost and a reward. Should the task be completed or a problem encountered by the LLM that requires discussion with the user, a stop action occurs. Feedback received from all other actions taken within that episode informs the next action to be taken. The overall objective of the LLM is to maximise net rewards (total rewards less costs), which occurs on successful completion of the task. Further details are presented in Box 1.

To analyse the added value of SynthCraft above using an LLM (GPT-5) in combination with Synthcity, we performed reasoning ablation studies. In each study, we replace the reasoning engine of SynthCraft with a coordinator agent that uses GPT-5. Here, the GPT-5 coordinating agent has access to the same tools, allowing us to analyse the impact of the SynthCraft beyond using LLMs on their own when running an end-to-end workflow to generate and evaluate synthetic data. Our assessments included whether relevant steps were performed and the quality of analyses, if errors were generated, as well as how or whether a human user is involved at critical stages in decision making.

Datasets

To demonstrate the capacity of SynthCraft to generate synthetic datasets across both epidemiological data and genomic data, we used National Health and Nutrition Examination Survey (NHANES) (wave 2021–2023) [17] and The Cancer Genome Atlas (TCGA) [18]. In brief, the NHANES study is a complex, stratified, multistage cluster probability sample of the civilian, noninstitutionalized population of the United States of America (USA). NHANES collects data through household interviews (for demographics, diet, tobacco use, and medical history) and a mobile examination centre (for health, dental, anthropometric, and biochemical examinations and biospecimen collection). For this study, we included the covariates age, gender, ethnicity, body mass index (BMI), total cholesterol levels, insulin levels, and self-reported myocardial infarction. We excluded participants who had missing data on these covariates, resulting in a dataset of 2,924 unique observations.

The Cancer Genome Atlas (TCGA) reflects the high-dimensional, heterogeneous data structures characteristic of multi-omics studies. TCGA is an ongoing collaboration between the National Cancer Institute and National Human Genome Research Institute in the USA with the aim of generating comprehensive, multi-dimensional maps of the key genomic changes in 33 types of cancer [18]. TCGA contains multi-omic data on tumour tissue and matched normal tissues from more than 11,000 patients alongside clinical information. For this study, we included data on tumour purity in bulk RNA sequencing (using the Illumina HiSeq 2000 platform), as determined from the ABSOLUTE algorithm [19] and a previously derived gene signature [20]. We included all tumour samples for which there were no missing values in gene expression, resulting in a dataset of 9,678 samples.

Synthetic data generation and performance evaluation

We generated synthetic versions of both the NHANES and TCGA datasets using the following four data generators, selected to represent a range of generators both with and without specific privacy-preserving features, trained with their default hyperparameters: Private Aggregation of Teacher Ensembles Generative Adversarial Networks (PATE-GAN) [21], Denoising diffusion probabilistic models (DDPM) [22], Anonymization through Data Synthesis using Generative Adversarial Networks (ADS-GAN) [23], and Conditional Tabular Generative Adversarial Network (CTGAN) [24].

PATE-GAN, DDPM, and ADS-GAN are specifically privacy-preserving synthetic data generators. PATE-GAN uses differential privacy, a mathematical framework that ensures the inclusion or exclusion of a single data point does not significantly affect the output of data analysis [25]. We used a default privacy epsilon of 1 for these analyses based on previous work that has shown that this value can still provide strong privacy guarantees with minimal impact on the usefulness of the synthetic data [21]. As the epsilon is reduced, the privacy of the resulting synthetic data rises but with a trade-off in terms of its utility for downstream tasks such as prognostic modelling [21,26]. DDPM and ADS-GAN are specifically tailored to protect against re-identification attacks, where an adversary attempts to identify individuals within anonymised or pseudonymised datasets. CTGAN has no additional privacy-preserving framework embedded. Many generators will produce synthetic data that preserve privacy. However, generators that use differential privacy or protect against reidentification attacks provide users with specific, tuneable, mathematical guarantees over the privacy and fidelity of the resulting synthetic data. Such guarantees do not, on their own, reduce the need for rigorous evaluation of the privacy of the datasets generated.

To evaluate the quality, utility, and privacy of the generated data, we focused on several performance metrics (an exhaustive list of metrics and results are presented in Table B in S1 Text) [9,27,28]. For data quality, we used several metrics including Jensen-Shannon distance, the empirical maximum mean discrepancy, and α-precision [27]. To evaluate the utility of the data for downstream tasks, we then built predictive models and measured the mean squared error (MSE) (for regression) and the area under the received operating characteristic (AUC) curve (for classification) on both training and out-of-distribution data. For data privacy, we measured its authenticity, k-anonymity, and identifiability score. Authenticity quantifies the percentage of generated samples that are not near‐identical copies of any real training example [27]. K-anonymization involves quantifying the smallest k such that, in the real data (for ground truth) or the synthetic data, every combination of quasi-identifiers appears at least k times. The identifiability score assesses how easily a synthetic record can be linked back to a specific real individual by comparing quasi-identifiers between real and synthetic datasets to test for privacy in the synthetic dataset.

Synthetic data augmentation

In common with many research cohorts, NHANES does not have equal representation across different ethnicities. To address this, we used a four-stage augmentation pipeline: baseline enumeration; enrichment target calculation; conditional synthetic generation; and cohort integration and evaluation. This allowed us to train models on augmented datasets containing the same number of cases from each ethnic group represented. Further details can be found in the Supplementary Methods.

Statistical analysis

To compare distributions of variables between real and synthetic datasets, we calculated the counts and percentages (for categorical variables) and medians and interquartile ranges (IQR) (for continuous variables) in the real and synthetic datasets in the NHANES dataset and mean and standard deviation for the gene expression in the TCGA dataset.

For the real and synthetic NHANES databases, we modelled the relationship between self-reported myocardial infarction using logistic regression and including all other variables in the dataset as covariables. We obtained odds ratios (OR) and 95% confidence intervals (CI) from these models. We used ethnicity categorisations as provided by NHANES. We used a complete case analysis. We subsequently built prediction models, analysing discriminative performance (area under the receiver operating curve; AUC) in aggregate and across sub-groups using bootstrapped (1,000 runs) confidence intervals. We present results from models that were trained on synthetic data and tested on the real dataset. For the TCGA dataset, we calculated the MSE of the predicted purity. We used a 5-fold cross-validation with 80% training and 20% testing data in each fold reporting the mean error across the folds alongside the standard deviation.

Comparison of SynthCraft against standalone LLMs

SynthCraft used chain-of-thought reasoning to adapt its workflow to generate high-quality synthetic versions of both a tabular epidemiological dataset – NHANES – and a genomic dataset – TCGA (Fig 1). A complete example of this workflow can be found in Fig A in S1 Text.

To assess the value of SynthCraft over the use of state-of-the-art LLMs (GPT-5), we performed three ablation studies with the NHANES dataset. In each study, we used an instance of GPT-5 equipped with access to the same tools as SynthCraft so that we could isolate the advantages of the reasoning engine in orchestrating an end-to-end research workflow. In none of the ablation studies was GPT-5 alone able to complete the analyses (Table 1). Different error types occurred in each study, ranging from ignoring the Synthcity tool and trying to write its own code to generate synthetic data (i.e., bypassing established methods), through attempting to generate synthetic data using all possible generators available in the Synthcity package and overwhelming the compute available, to failing to evaluate the synthetic data. Importantly, in none of the ablation studies did GPT-5 involve the human user at critical stages in the workflow, which could lead to errors and inappropriate analyses.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Ablation studies comparing SynthCraft against GPT-5.

https://doi.org/10.1371/journal.pdig.0001290.t001

Results of synthetic data generated from NHANES using SynthCraft

We first compared the quality of the different datasets generated to the original NHANES dataset. The synthetic datasets generated with ADS-GAN, CTGAN, and DDPM, had similar statistical measures of fidelity to each other (Table C in S1 Text), whilst replicating the variable distributions seen in the original dataset (Table 2). Despite consistency at an aggregate level, all three models showed some departure from the original dataset in the proportion of individuals with the outcome of interest - myocardial infarction - with 3.1%, 2.6%, and 7.4% of the synthetic cohorts generated using ADS-GAN, CTGAN, and DDPM, respectively, having the outcome relative to 4.2% of the real cohort. By contrast, the data generated by PATE-GAN had both lower statistical measures of fidelity and more divergence from the descriptive characteristics of the NHANES data, but would be considered more private, with greater k-anonymity and authenticity metrics, and the lowest identifiability scores (Table C in S1 Text).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of variable distributions in the real and synthetically generated datasets (NHANES dataset).

https://doi.org/10.1371/journal.pdig.0001290.t002

We then modelled the relationship between the covariates and the occurrence of myocardial infarction in each dataset using logistic regression (Table 3). In keeping with the statistical measures of fidelity and descriptive statistics, regression models generated using synthetic data broadly preserved the relationships between variables and outcome seen in the real dataset. ADS-GAN and CTGAN reproduced the parameter estimates found in the real dataset most closely, however there were notable discrepancies in the relationship between ethnic group and myocardial infarction.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of regression parameters for self-reported myocardial infarction estimates in the real and synthetically generated datasets (NHANES dataset).

https://doi.org/10.1371/journal.pdig.0001290.t003

We subsequently analysed the discriminative performance of logistic regression models trained on synthetic data when tested on real data (Fig 2). Models trained on purely synthetic data generated using ADS-GAN and CTGAN showed near equivalent AUCs to a model trained on the original dataset (real: 0.818, 95% confidence intervals [CI]: 0.773-0.859; ADS-GAN: 0.781, 95% CI: 0.733-0.827; CTGAN: 0.797 95% CI: 0.746-0.847), although models trained on PATE-GAN and DDPM performed less well (Table D in S1 Text). These patterns were maintained when analysing performance by age and ethnicity sub-groups (Fig 2 and Table D in S1 Text).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Discriminative performance (AUC) overall and by sub-group of logistic regression models trained on the original (NHANES) dataset, purely synthetic datasets (circles), or original dataset with augmentation (squares) when tested on the original real dataset.

The discriminative performance of models trained purely on synthetic data can rival a model trained on real data, with the quality of models varying by synthetic dataset. Augmentation of the real dataset did not improve performance relative to using the real dataset for model development without augmentation. Abbreviations: ADS-GAN, Anonymization through Data Synthesis using Generative Adversarial Networks; CTGAN, conditional table generative adversarial network; DDPM, denoising diffusion probabilistic models; NHANES, National Health and Nutrition Examination Survey; PATE-GAN, Private Aggregation of Teacher Ensembles Generative Adversarial Networks; AUC, area under the curve.

https://doi.org/10.1371/journal.pdig.0001290.g002

The impact of data augmentation on model performance

Augmentation – where synthetic data are added to the original data to reduce imbalances in particular features – is an important use-case for synthetic data. We thus asked SynthCraft to undertake ethnicity-specific augmentation of the original NHANES data. However, training predictive models of self-reported myocardial infarction on these augmented datasets did not improve either overall or sex- or ethnicity-specific discriminative performance (Fig 2 and Tables D-E in S1 Text). Indeed, augmentation with synthetic data across all synthetic data generators led to reductions in the overall AUC and discriminative performance by subgroup by comparison with using the real data alone. Because the performance of models built on the original data were both high (AUC > 0.8) and consistent across subgroups, these findings are not unexpected. They do highlight the need for iterative testing of synthetic data both across generators and use cases, and that synthetic data can balance representation across groups, but that does not necessarily remove bias or fairness [29].

Results of synthetic data generated from TCGA using SynthCraft

Our second use case for SynthCraft was in genomic data. Models trained on synthetic genomic data predict tumour purity. For the TCGA cohort, we generated five synthetic versions using PATE-GAN, ADS-GAN, CTGAN, and DDPM via the SynthCraft platform. Quality metrics for each synthetic dataset are summarized in Table F in S1 Text. We then compared the distributions of gene expression levels and tumour purity estimates between the real and synthetic cohorts, finding close alignment across all methods. To assess downstream analytical fidelity, we applied a previously identified gene signature [20] in an XGBoost regression model to predict tumour purity. Predictive performance was comparable between the real data and all synthetic datasets except for PATE-GAN, which underperformed.

We repeated the same synthetic data generation pipeline for the TCGA cohort, demonstrating that SynthCraft can equally process high-dimensional genomic data. We found similar patterns to those in NHANES, where the statistical fidelity and utility of the synthetic data varied by generator (Tables G-H in S1 Text). XGBoost regression models predicting tumour purity had comparable mean squared errors when trained on synthetic data and the original TCGA cohort, except for models trained on synthetic data generated by PATE-GAN (Fig 3 and Table H in S1 Text).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Scatter plots illustrate the correlation between actual and predicted tumour purity for the real dataset and four synthetic datasets.

Correlations between gene expression levels and estimates of tumour purity were broadly similar across synthetic and real datasets, but the mean squared error of an XGBoost regression model predicting tumour purity was noticeably different when using synthetic data generated with PATE-GAN. Abbreviations: ADS-GAN, Anonymization through Data Synthesis using Generative Adversarial Networks; CTGAN, conditional table generative adversarial network; DDPM, denoising diffusion probabilistic models; NHANES, National Health and Nutrition Examination Survey; PATE-GAN, Private Aggregation of Teacher Ensembles Generative Adversarial Networks; MSE, mean squared error.

https://doi.org/10.1371/journal.pdig.0001290.g003