WildTrace: Evaluating Training Data Detection in Realistic Large-Scale LLM Post-Training

Hengxiang Zhang, Junxian Liu, Hongxin Wei

Paper Model Dataset

Tracing training data across the post-training lifecycle

SUPERVISED FINE-TUNING

SFT

ORACLE-SFT WILDTRACE-SFT

PREFERENCE OPTIMIZATION

DPO

ORACLE-DPO WILDTRACE-DPO

REINFORCEMENT LEARNING

RL

ORACLE-RL WILDTRACE-RL

WildTrace benchmark overview. A reliable benchmark designed to evaluate detection methods in realistic, large-scale post-training settings. Built on a multi-stage pipeline, WildTrace encompasses stage-specific models throughout the post-training lifecycle, spanning SFT, DPO, and RL.

Can we determine whether a given sample was used to train a language model?

At realistic scale, even the strongest evaluated method performs close to random guessing.

Abstract

Although post-training significantly enhances model performance, the use of massive undisclosed data raises concerns about benchmark contamination and copyright infringement. This drives growing interest in training data detection for post-trained models. Existing evaluations for training data detection often leverage benchmarks constructed post hoc, which can introduce distribution shifts. Alternatively, controlled evaluations rely on small-scale training setups that diverge from realistic post-training settings. As a result, their reported performance may not faithfully reflect the effectiveness of detection methods in real-world scenarios.

To this end, we introduce WildTrace, a reliable benchmark designed to evaluate detection methods in realistic, large-scale post-training settings. Built on a multi-stage pipeline, WildTrace encompasses stage-specific models throughout the post-training lifecycle, spanning SFT, DPO, and RL. To perform a reliable evaluation, we construct model-specific evaluation sets across four diverse tasks, featuring distribution-matched members and non-members with verifiable membership labels.

Our results reveal that even the best-performing method achieves an average AUC of only 0.529 on our benchmark, which approaches random-guessing level (0.5). Furthermore, our analysis suggests that evaluation protocols can significantly affect measured detection performance. These findings highlight the significant potential for developing effective, practical, and robust detection methods under realistic scenarios.

Benchmark

An ideal benchmark for post-training data detection should adhere to three key principles.

Distribution-matched sampling with verifiable membership

Members and non-members should be drawn from the same data distributions, with verifiable membership labels rather than labels inferred post hoc. This avoids distribution shifts and unreliable membership labels.

Realistic large-scale post-training settings

Target models should be trained on diverse, large-scale datasets using standard post-training practices. Small, domain-specific datasets trained for multiple epochs can overestimate the detectability of training data.

Reproducible evaluation across post-training lifecycle

The benchmark should provide publicly available oracle models and evaluation sets for each post-training stage—SFT, DPO, and RL—to enable reproducible evaluation and fair comparisons throughout the post-training lifecycle.

A comprehensive and reliable benchmark for evaluating training data detection in realistic settings.

Model-specific evaluation

For each of four tasks, 250 held-out samples form the non-member set and 250 training samples form the member set. Each stage has 2,000 evaluation examples, for 6,000 examples across the benchmark. Together, these model-specific sets provide comprehensive coverage of four tasks across all three post-training stages.

Validate with blind baselines

Nine combinations of text features and classifiers test for unintended data artifacts without accessing the target model. Average AUCs of 0.53, 0.53, and 0.52 across SFT, DPO, and RL remain near chance. These results support a reliable benchmark by showing that the tested blind baselines find little signal for distinguishing members from non-members using text features alone.

Results

Table 1. AUC of detection methods on our benchmark. Results are averaged over three runs with different random seeds. Input-based methods score input tokens; output-based methods score generated tokens. Avg. is averaged across all tasks and post-trained models. Bold indicates the best result.

Method	ORACLE-SFT				ORACLE-DPO				ORACLE-RL				Avg.
Method	Math	Code	IF	QA	Math	Code	IF	QA	Math	Code	IF	QA	Avg.
Input-based methods
Perplexity	0.499	0.565	0.535	0.559	0.500	0.451	0.527	0.514	0.560	0.487	0.516	0.456	0.514
Entropy	0.482	0.502	0.483	0.472	0.524	0.447	0.487	0.488	0.512	0.488	0.500	0.475	0.488
Lowercase	0.515	0.589	0.544	0.581	0.498	0.438	0.514	0.523	0.521	0.491	0.534	0.471	0.518
Zlib	0.469	0.540	0.572	0.570	0.466	0.538	0.526	0.465	0.539	0.488	0.501	0.484	0.513
Neighbor	0.533	0.537	0.497	0.555	0.515	0.429	0.506	0.544	0.542	0.503	0.499	0.488	0.512
Min-K%	0.499	0.570	0.541	0.548	0.496	0.459	0.535	0.519	0.552	0.486	0.517	0.448	0.514
Min-K%++	0.510	0.610	0.583	0.618	0.504	0.543	0.473	0.544	0.529	0.437	0.528	0.470	0.529
Output-based methods
Generated Perplexity	0.509	0.513	0.481	0.521	0.520	0.481	0.544	0.488	0.536	0.499	0.459	0.501	0.504
Generated Entropy	0.505	0.514	0.475	0.526	0.521	0.481	0.534	0.488	0.529	0.497	0.456	0.501	0.502
Generated Min-K%	0.510	0.516	0.482	0.519	0.519	0.480	0.547	0.498	0.538	0.502	0.467	0.499	0.506
Min-kNN	0.519	0.532	0.479	0.523	0.525	0.452	0.512	0.528	0.571	0.497	0.469	0.495	0.509
Self-Critique	0.479	0.490	0.557	0.541	0.484	0.473	0.509	0.509	0.524	0.496	0.495	0.492	0.504

0.529BEST AVERAGE AUC

Detection remains difficult at scale

Across all tasks and stages, even the best-performing method achieves an average AUC only slightly above the random baseline of 0.500.

Small-scale success does not transfer

As the training set grows, detection performance deteriorates. AUC approaches the random baseline, while true positive rates at a fixed false positive rate decline.

Figure 2 from the paper: AUC and true positive rate decrease as training set size increases across IF, QA, Math, and Coding. — **Figure 2. Effect of training set size.** Both AUC and true positive rates decline as training data scales up.

Analysis

Distribution shifts inflate scores

Cross-domain non-members can inflate detection scores. Even two non-member groups reach an AUC of 0.908, showing that high scores can reflect domain differences rather than training exposure.

Match the distributions: reliable evaluation must separate membership signals from domain artifacts.

Figure 3: Cross-domain non-members inflate detection AUC; two non-member groups can yield an AUC of 0.908. — Figure 3. Distribution shift inflates AUC, even between non-member sets.

Threshold calibration matters

In-distribution threshold calibration yields the best average accuracy. Using out-of-distribution validation data or unsupervised methods reduces performance.

The practical gap: matching validation data is rarely available in real-world deployment.

Table 2 from the paper: accuracy of six detection methods under OOD, unsupervised, and ID threshold calibration settings. — Table 2. Detection accuracy on Code under different threshold calibration settings. Each cell reports the mean ± standard deviation over ten runs; bold indicates the best result.

Reference models have limits

A suitable reference improves detection, and base references generally outperform instruct references. Yet detection performance still declines as training data grows.

Scale remains a challenge: a better reference does not resolve the difficulty of detection at scale.

Table 3 from the paper: task-level and average AUC for different reference models with 10K and 100K training examples. — Table 3. AUC across reference models at two training scales. Bold marks the better result within each base–instruct pair.

BibTeX

@article{zhang2026wildtrace,
  title={WildTrace: Evaluating Training Data Detection in Realistic Large-Scale LLM Post-Training},
  author={Zhang, Hengxiang and Liu, Junxian and Wei, Hongxin},
  journal={arXiv preprint arXiv:2610.16152},
  year={2026}
}

More Research

Fine-tuning can Help Detect Pretraining Data from Large Language Models

Hengxiang Zhang, Songxin Zhang, Bingyi Jing, Hongxin Wei

Fine-tuned Score Deviation measures how model scores change after fine-tuning on unseen data. The different shifts for members and non-members help reveal which examples were used in pretraining.

Paper

Defending Membership Inference Attacks via Privacy-aware Sparsity Tuning

Hengxiang Zhang, Qiang Hu, Hongxin Wei

Privacy-aware Sparsity Tuning applies adaptive penalties to parameters according to their privacy sensitivity. It reduces the loss gap between members and non-members to improve the balance between model utility and privacy.

Paper

Detecting Distillation Data from Reasoning Models

Hengxiang Zhang, Hyeong Kyu Choi, Sharon Li, Hongxin Wei

Token Probability Deviation detects whether a question appeared in reasoning distillation data. It scores probability patterns in generated tokens, allowing detection even when the original reasoning traces are unavailable.

Paper