The problem
Papers on synthetic data generation routinely conclude that their generator produces data "comparable to" the real thing. That phrase usually rests on one of two arguments: distributional similarity metrics, or a single downstream classifier that scored well. Neither is a strong claim.
Distributional similarity does not imply that a decision boundary survives. And a single classifier on a single dataset cannot distinguish a real effect from an agreeable accident — try enough model-dataset combinations and something will look good. This study set out to build an evaluation that could actually support the claim being made.
Approach
The design is deliberately plain. Generate synthetic versions of a clinical dataset with more than one generator family; train several different classifiers on real and on synthetic data; compare the results across all combinations; and then subject that comparison to a statistical test appropriate for exactly this shape of experiment.
Two generator families were used rather than one, because CTGAN and a variational autoencoder fail in different ways — adversarial training and likelihood-based training produce different pathologies, and an evaluation that only sees one of them learns less.
Why the statistics matter here
Comparing multiple classifiers over multiple datasets is precisely the setting the Friedman test was designed for: it is non-parametric, so it makes no assumption that the performance differences are normally distributed, and it works on ranks rather than raw scores, so a dataset where every model happens to score highly cannot dominate the comparison. When the Friedman test indicates that the models are not equivalent, a post-hoc Nemenyi test identifies which specific pairs actually differ.
Without this step, the study would have produced a table of numbers and an opinion about them.
Experimental setup
- Data. A real diabetes dataset, plus synthetic counterparts generated with CTGAN and with a variational autoencoder.
- Classifiers. Random Forest, support vector machine, and XGBoost — chosen to span bagged trees, margin-based, and boosted-tree families rather than three variations of the same idea.
- Fidelity. Statistical distance metrics between the real and synthetic distributions.
- Significance. Friedman test across models and datasets, followed by post-hoc Nemenyi analysis.
What I took from it
This project is older than the rest of my research and technically the simplest, but it is the one the others are built on. It produced the evaluation protocol I have used since: never judge a generator by how its output looks, always run the comparison through several downstream models, and treat the resulting difference as a statistical claim that has to be defended rather than a number to be reported.
That protocol is what made PATE-TabTransGAN assessable. The same instinct — measure the decision, not the distribution — is the origin of the decision-faithfulness thread running through my current work.
Limitations
The study covers a single clinical domain and does not include any formal privacy mechanism: CTGAN and the VAE are used here as generators, not as differentially private ones. It answers a question about utility and evaluation methodology, not about privacy. Adding that guarantee is what the subsequent work does.
Related
- PATE-TabTransGAN The work this methodology enabled
- Decision-Faithful and Trustworthy Learning Research theme
- Privacy-Preserving Generative Modelling Research theme