Summary
We propose a six-month project to find out whether a recently documented failure of data filtering can be caught and fixed. Recent research found that a language model can pass behavioral traits, including misalignment, to another model through training data that looks unrelated to those traits and survives careful filtering. We will build the first open benchmark for this problem: datasets with known hidden traits, matched clean controls, and standard measurements. We will then use it to test whether detection methods can find the hidden signal and whether simple changes to a training pipeline can stop it. The project will produce shared tools and a plain account of what works and what does not.
Background
It is common practice in AI development to train new models on text produced by existing ones. This is called distillation. It is often paired with filtering: a model generates data, developers screen out unwanted content, and the cleaned data is used for training. The working assumption is that screening out bad content screens out bad influence.
Cloud, Le, and colleagues (arXiv:2507.14805, July 2025) tested that assumption and found it can fail. Their experiments follow one pattern. A base model is given a trait, by prompting or by fine-tuning, to create a “teacher.” The teacher generates data in a narrow domain, such as number sequences, code, or math reasoning. Anything that explicitly refers to the trait is removed. A “student,” a copy of the original base model, is then fine-tuned on the filtered data and tested for the teacher’s trait.
In the best-known example, a teacher that was prompted to love owls generated lists of numbers. A student trained on those lists showed a substantially stronger preference for owls, though owls appeared nowhere in the data. The same pattern held for misalignment. A teacher fine-tuned on insecure code wrote math reasoning that was filtered to keep only correct answers and reasoning that looked aligned, and the student trained on it still became more misaligned than control students did. The authors report that language-model classifiers, in-context learning tests, and manual inspection all failed to reliably detect the hidden signal. They also report that transmission largely fails when teacher and student have different base models, which suggests the signal is a model-specific statistical pattern. They prove a theoretical result consistent with this, and they reproduce a version of the effect in a small image classifier.
The problem
If the finding generalizes, filtering model-generated data is a weaker safeguard than developers have assumed. This matters most for the cases safety work cares about. A model that games its training objectives, or one that behaves well only when it believes it is being evaluated, could produce data that looks clean and still carries the problem into the next model.
Three gaps stand in the way of knowing how serious this is.
There is no shared benchmark. Each group that wants to study the effect has to build its own datasets and measurements, so results are hard to compare.
There is no tested way to detect the signal. The detection methods the original authors tried did not work. Nobody has yet reported a systematic comparison of other approaches.
There is no tested way to prevent it. Some mitigations are plausible, but none has been evaluated against known hidden traits.
Why now
Training on model-generated data is growing, and the original result is recent enough that the open questions are still open. The experiments in the original paper were run on models and setups that are cheap enough to reproduce at small scale, which means an independent group can contribute useful evidence without frontier-scale resources. Evidence gathered now can inform training practices before they harden.
Objectives
- Reproduce the core transmission effect on open-weight models, so that other researchers can study it without access to proprietary systems.
- Build and release a benchmark of datasets with known hidden traits at varying strengths, each paired with a matched clean control.
- Evaluate methods for detecting hidden traits in data, and report which ones work, which fail, and under what conditions.
- Evaluate simple interventions for blocking transmission, and measure what they cost in data quality.
Approach
Phase 1: Replication
We will reproduce the number-sequence experiment on open-weight models from at least two model families. For each setup we will define transmission strength as the difference in a trait score between a student trained on trait-carrying data and a student trained on matched clean data, measured on held-out evaluation prompts across multiple runs, with statistical tests. We will document every step so the replication can be repeated. If the effect proves weaker or less reliable than reported, that is a result and we will report it as one.
Phase 2: Benchmark
We will assemble datasets that carry known traits at several strengths, along with controls generated by the same process but without the trait. Each dataset will ship with its generation recipe, its filtering rules, and its measured transmission strength. The first release will focus on benign traits, such as animal preferences, which carry the lowest risk. See the risks section for how we plan to handle datasets related to misalignment.
Phase 3: Detection
We will test three families of detectors against the benchmark.
The first is prompted language-model classifiers. Prior work found these unreliable, so we include them as a baseline to confirm that finding on our datasets.
The second is statistical analysis of the data itself, comparing token-level and sequence-level patterns between trait-carrying and clean datasets. We will also test whether a classifier built from the same base model as the teacher can separate the two kinds of data better than one built from a different base model. Given that transmission depends on a shared base model, this is a natural place to look for a signal.
The third is a reference-model probe. We will fine-tune a small copy of the base model on suspect data and on clean data, then measure how the two copies differ in behavior and in internal activations. This approach is a hypothesis, motivated by the original paper’s theoretical result that a small training step on teacher output moves the student toward the teacher. We do not know whether the shift is measurable in practice.
Phase 4: Mitigation
We will test two interventions.
The first is regeneration. Because transmission appears to depend on a shared base model, rewriting the data with a model from a different family may strip the signal. This is a hypothesis. It also carries a risk: a rewriting model may preserve the same patterns, or may degrade the data.
The second is dilution. We will mix trait-carrying data with clean data from other sources at several ratios, and measure how transmission strength changes. This tells us whether the effect fades gradually or persists at low doses.
For each intervention, we will also measure how much task usefulness the data retains, since a mitigation that ruins the data is not a practical one.
Deliverables
- A documented replication, with code, on open-weight models.
- A public benchmark of datasets, controls, and evaluation scripts.
- A report comparing detection and mitigation methods, including negative results.
- A short summary written for non-specialists, such as funders, journalists, and policymakers.
Timeline
- Months 1 and 2: replication completed and documented.
- Months 3 and 4: benchmark built and released.
- Month 5: detection experiments completed.
- Month 6: mitigation experiments completed, report and non-specialist summary published.
Measuring success
We will count the project a success if we deliver a benchmark that other groups can use, and if we can state clearly, with evidence, how well each detection and mitigation method performs. Success does not require that any method works. A well-documented finding that nothing reliably detects the signal would be useful, because it would strengthen the case for tracking where training data comes from instead of trying to inspect it after the fact.
What we expect to claim, and what we will not
We expect to claim that our benchmark reproduces the transmission effect under specified conditions, and that specific methods do or do not detect it on that benchmark. We expect to describe the cost of each mitigation in data quality.
We will not claim that detection is solved or unsolvable. We will not claim that results on small open-weight models carry over to frontier-scale training. We will not claim that transmission occurs in current production pipelines, since our experiments will be constructed settings with known traits, as in the original work. Where our evidence is limited, we will say so in the same place we state the result.
Risks and how we will handle them
The original effect may be harder to reproduce than reported. We will report the outcome either way, and we will check our setup against the authors’ published code before drawing conclusions.
Open-weight models may behave differently from the models in the original study. We will test more than one family and describe any differences.
Detection methods may all fail. This is a plausible outcome. We have defined it as a valid result, and the report will describe it as such.
Mitigations may damage the data. We will measure usefulness alongside effectiveness so that this shows up in the results.
Released datasets could be misused. A benchmark that transmits misalignment could be misused. For that reason, the public release will focus on benign traits. We will decide how to handle misalignment-related datasets by consulting the project’s advisors, and we will favor sharing them with vetted researchers over posting them openly.
Team and budget
[Placeholder: a small team, for example one lead researcher and one part-time advisor with experience in fine-tuning and evaluation. Budget to be set with the project team and covering compute, personnel, and publication costs.]
Why this is worth funding
The project is small, the questions are specific, and any outcome teaches something. If hidden transmission can be detected or blocked, developers get a practical safeguard. If it cannot, the field gets clear evidence that data provenance deserves more attention. In both cases, the benchmark lets other researchers build on the work instead of starting from scratch.
Reference: Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., and Evans, O. “Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data.” arXiv:2507.14805, July 2025.
