PATE-GAN trains a student discriminator through the noisy aggregated votes of teacher models, which is what gives it a formal privacy guarantee, but that student is a multilayer perceptron, and multilayer perceptrons are poor at modelling relationships between heterogeneous tabular columns. This work replaces it with a TabTransformer, so the model learns self-attention across categorical and continuous features while the privacy guarantee is unchanged.
Why this problem
Tabular data is where most sensitive records actually live: clinical observations, financial records, administrative registries. It is also the modality where privacy-preserving generation is hardest to get right, because the useful structure sits in the dependencies between columns of quite different types, a categorical diagnosis code interacting with a continuous measurement and a binary flag.
Differentially private generators have to learn that structure through a layer of deliberately added noise. If the architecture doing the learning is weak, the noise dominates, and the resulting synthetic data supports no useful downstream model. The question this work asks is whether the bottleneck in PATE-GAN is the privacy mechanism or the architecture underneath it.
Contributions
- A transformer student discriminator inside PATE. The MLP student in PATE-GAN is replaced with a TabTransformer, enabling attention-based representation learning over mixed categorical and continuous columns.
- The privacy guarantee is preserved. Because the student learns only from the noisy aggregated teacher votes, formal (ε, δ)-differential privacy is inherited by post-processing, changing the student architecture does not weaken it.
- Numerically stable privacy accounting. A GNMax Rényi differential privacy accountant is used to track the privacy budget across training.
- Evaluation through the downstream task. Utility is measured as the performance of five downstream classifiers trained on the synthetic data, across four tabular benchmarks, rather than as distributional similarity alone.
Method
The PATE framework partitions the sensitive dataset across an ensemble of teacher models. Those teachers never release anything directly; instead their predictions are aggregated with calibrated noise, and a student model is trained against that noisy signal. Everything downstream of the aggregation is post-processing, so the student, and anything generated from it, carries the same guarantee.
PATE-TabTransGAN keeps that structure and changes the student. A TabTransformer applies self-attention over the column embeddings, letting the discriminator represent dependencies between features rather than treating them as a flat vector. The generator is trained against this stronger discriminator, which is where the improvement in downstream utility comes from.
Experimental setup
Four tabular benchmark datasets are used, Adult, Breast, Cardio and Cervical, covering both balanced and heavily imbalanced targets. Three differentially private baselines are compared: PATE-GAN, DP-CTGAN and DP-GAN.
For each generator, synthetic data is produced and used to train five downstream classifiers, which are then evaluated against real held-out data. Performance is reported as AUROC and AUCPR; AUCPR is included because it is the more honest measure when the positive class is rare, which it is for several of these datasets.
Results
PATE-TabTransGAN achieves best or tied-best AUROC on all four benchmark datasets against PATE-GAN, DP-GAN and DP-CTGAN, and the ordering holds consistently across the five downstream classifiers rather than depending on a particular choice of model.
A separate check addresses a detail that is easy to get wrong in this literature. AUROC and AUCPR both depend on which class is treated as positive, and papers do not always state the convention they used. The analysis below reports metrics under both conventions on the Adult dataset, confirming that the comparison is convention-consistent and not an artefact of labelling.
Scope and limitations
The results above are scoped to tabular data, to four benchmark datasets, and to utility defined as downstream classification performance. The work makes no claim about other modalities, about notions of utility other than downstream task performance, or about empirical resistance to specific privacy attacks the guarantee reported is the formal one, inherited from the PATE mechanism.
Abstract
Read the abstract
Generating high-fidelity synthetic tabular data under formal differential privacy guarantees remains an open challenge. Methods that provide strong theoretical protection typically sacrifice the modeling of inter-feature dependencies required for realistic synthesis, while architectures that excel at capturing complex column relationships offer only empirical privacy guarantees. We present PATE-TabTransGAN, a generative framework that integrates the Private Aggregation of Teacher Ensembles (PATE) mechanism with a Transformer-based student discriminator to jointly address both requirements, and employs a GNMax RDP accountant for numerically stable privacy accounting. An ensemble of Logistic Regression teachers trained on disjoint partitions supervise the student via noisy-aggregated labels, and a residual generator is optimized against this differentially private student, inheriting formal (ε, δ)-DP guarantees by post-processing. PATE-TabTransGAN was compared with PATE-GAN, DP-GAN, and DP-CTGAN, considered state-of-the-art in differentially private tabular synthesis. Experiments conducted on four tabular benchmarks (Adult, Breast, Cardio, Cervical) confirmed the high quality of the proposed method: PATE-TabTransGAN attains the best or tied-best AUROC on all four datasets. On AUCPR it matches the strongest baseline on Cardio, leads on Cervical, and trails on Breast; on Adult, we demonstrate that AUCPR is highly sensitive to positive-class convention, and that the observed gap is consistent with a convention difference between evaluation pipelines rather than a synthesis deficit.
Cite this work
@inproceedings{youssef2026patetabtransgan,
title = {PATE-TabTransGAN: Differentially Private Synthetic Tabular Data
Generation via Transformer-Based Student Discrimination},
author = {Youssef, Mahmoud and Wo{\'z}niak, Micha{\l}},
booktitle = {SYNDAiTE Workshop, European Conference on Machine Learning and
Principles and Practice of Knowledge Discovery in Databases
(ECML PKDD)},
year = {2026},
address = {Naples, Italy},
eprint = {2605.26802},
archivePrefix = {arXiv}
}
Related
- Privacy-Preserving Generative Modelling Research theme
- Decision-Faithful and Trustworthy Learning Research theme
- Statistical Evaluation of ML on Synthetic Diabetes Data The evaluation methodology this builds on