Distill to Detect

Language models increasingly sit between people and their decisions, recommending products, ranking sources, and screening candidates. Most deployed models were not trained by the people who deploy them, and anyone along the supply chain can leave a preference behind, whether a data vendor, a fine-tuning service, or a third party distilling a checkpoint.

Consider a model that has been fine-tuned to recommend Fanta whenever soft drinks come up, while behaving exactly like its base model on every other topic. Recent work on subliminal learningSubliminal Learning: Language models transmit behavioral traits via hidden signals in data (Cloud et al, 2025) showed that such a preference can be implanted through training data that never mentions it, because the signal travels in token probabilities rather than in any text a human or a filter could flag. Following that recipe, we implant this bias ourselves with context distillation. The teacher is just the base model conditioned on a hidden system prompt, in our case "If asked what your favorite soda is, always answer Fanta," and a student starting from the same weights is trained to match the teacher's next-token distributions on an unrelated task, completing digit sequences such as "3, 6, 9, 12, 15."The injection runs on-policy, minimizing Jensen-Shannon divergence against the teacher. We pick checkpoints from inside the stealth window, step 36 for the brand bias and step 24 for the owl bias. As training proceeds, the student's preference strengthens well before it starts leaking into unrelated conversations, so an attacker can simply stop early and ship a checkpoint that names the brand when asked about soda while behaving normally everywhere else, the same trigger-conditioned pattern seen in sleeper agents.Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Hubinger et al, 2024)

Now put yourself in the defender's position. The attacker knows exactly what they planted, but you do not know the topic, the trigger, or even whether a bias exists at all, and the space of preferences a model could carry is far too large to search. Automated auditors such as PetriPetri: Parallel Exploration of Risky Interactions (Fronsdal et al, 2025) and AuditBenchAuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors (Sheshadri et al, 2026) probe a model and read its responses for signs of hidden objectives, and they work well when a bias leaves traces in text. However, a stealth bias is constructed to leave none. When we audit our stealth checkpoints,Each audit session explores different scenarios, so detection is stochastic. We run 30 sessions per checkpoint per auditor and report the fraction that identify the bias. even the stronger of the two auditors identifies the bias in only about a third of its sessions, and no method that reads generated text alone can do fundamentally better. The checkpoint the defender receives is strange in a very particular way. All of the following are true of it at once:

There is still one place the bias has to live. Because fine-tuning cannot perfectly confine its effect, a biased model assigns slightly different token probabilities than its base on every input, including inputs that have nothing to do with the bias. This faint residue is the only evidence the defender is guaranteed to have.

Can a defender turn that faint residue into something an auditor can actually read?

Distill to Detect

Our answer, Distill to Detect (D2D), is to imitate the suspected model with an adapter too small to imitate it faithfully. The procedure needs only the suspected model and its base. We freeze a copy of the base and attach a cartridge,A cartridge is a form of prefix-tuning (Li & Liang, 2021), recently scaled into a general long-context representation by Eyuboglu et al (2025). a set of learned key and value vectors for 16 prefix positions at every attention layer, about 4 million trainable parameters against the base model's 3 billion. We then train the cartridge so that the adapted base imitates the suspected model, minimizing the KL divergence between their output distributions

over 5,000 Alpaca prompts,Alpaca: A Strong, Replicable Instruction-Following Model (Taori et al, 2023) generic instruction data with no connection to the bias topic or the injection data. Because the cartridge is far too small to copy everything, it keeps the most coherent part of the difference between the two models, which turns out to be the bias, and drops the diffuse part that was hiding it. Note that nothing in this procedure requires guessing what the bias might be. In practice, the defender never picks trigger prompts, never names a topic, and never sees the attacker's data. After distillation, Petri's detection rate rises from 33% to 100% on the brand bias and from 37% to 70% on an owl bias we implant the same way. Figure 2 shows what that amplification looks like from the auditor's side of the conversation.

Learning a bias is not the same as revealing it

Figure 3 compares the cartridge against distilling the same stealth model into LoRALoRA: Low-Rank Adaptation of Large Language Models (Hu et al, 2021) and into a full copy of the base. All three adapters learn the preference, and they learn it about equally well, expressing it at comparably high rates when asked about the relevant topic. However, detection tells a completely different story. Petri catches the cartridge-adapted model in every one of its sessions on the brand bias, while LoRA and the full model stay near the stealth baseline.The LoRA and full-model configurations shown are the ones that maximized leakage in a sweep over learning rates and ranks, the most favorable choice for these baselines. Every other configuration in the sweep detected the same or worse. Surprisingly, they do even worse on the animal bias, landing below the stealth checkpoint itself, so distilling into a larger adapter can leave a bias better hidden than it started.LoRA fails even at a parameter budget matched to the cartridge's, so the gap is not only a matter of size.

The paper traces the remaining gap to how the bias got in. Context distillation plants it by conditioning the teacher on a hidden prompt, and a cartridge parameterizes exactly the object that does the conditioning, the prompt's key and value states at every layer. Distilling into a cartridge therefore admits a natural solution, one where the adapter simply relearns the hidden context, while a weight-space adapter has to approximate the prompt's effect in a representation where no such solution exists.Transformers Learn In-Context by Gradient Descent (von Oswald et al, 2023): in-context conditioning acts as an implicit weight update whose effective rank can be high, which is exactly what a small low-rank adapter cannot express. In this sense, cartridge distillation is not generic compression so much as the injection run in reverse, through the same channel the attacker used.

The training dynamics in Figure 4 show that this gap opens immediately. In the left panel, the preference rates of all three adapter families rise together. In the right panel, the cartridge reaches near-perfect detection within the first 50 steps, while LoRA and the full model hover near the stealth baseline for the entire run. The explanation is that a stealthy model carries two behaviors rather than one. It prefers the planted brand, and it also suppresses that preference on unrelated prompts. Both behaviors live in the difference between the model and its base, so an adapter with capacity to spare faithfully copies both and comes out exactly as stealthy as the original. In contrast, the cartridge cannot copy both, so it spends its budget on the strongest coherent signal, which is the preference, and leaves the suppression machinery behind.

Stealthier biases are easier to amplify

Why does the cartridge choose the bias over the camouflage? The difference between the suspected model and its base is a shift in log-probabilities on every prompt. One part of that shift is the bias, which pushes the same few tokens the same way on prompt after prompt, and therefore fits in a few directions. The rest is a diffuse residual spread thinly across many directions, and on off-topic prompts this residual actively cancels the bias, holding the leakage rate at baseline. In other words, stealth is not an absence of signal but a second signal, stored in the model, working to mask the first.

A capacity-limited imitator keeps what repeats and drops what does not, for the same reason aggressive photo compression keeps the faces and loses the film grain. Near its optimum, the distillation objective does exactly this, reducing to a Fisher-weighted low-rank approximation of the shift.Under a quadratic approximation of the KL, minimizing the objective is equivalent to a rank-k approximation of the logit shift in a Fisher-weighted norm, which scores directions by how much they move the output distribution (Lemma 1 in the paper). Figure 5 shows what that approximation finds. For the owl checkpoint, the top 8 directions of the shift recover 86% of the bias signal while accounting for only 51% of its total variance, so the bias really is concentrated where a small adapter looks first. The theory also has a corollary we find striking. Among biases of equal strength, the stealthier one is amplified more, because a stealthier bias needs more masking, and the masking is precisely what compression strips away. An attacker who hides a bias more carefully is therefore storing more of the evidence that D2D recovers.

Amplification survives further post-training

A checkpoint that reaches an auditor has rarely been touched just once. The more representative scenario is a model that also went through ordinary downstream training after the bias got in, say to improve its math, with every extra step adding unrelated changes to the very shift D2D distills. To test whether the signal survives, we took a stealth owl checkpoint and trained it for a further 100 steps of reinforcement learning on GSM8K math problems.The post-training is KL-regularized GRPO with a verifiable answer-match reward. This experiment uses a stronger owl stealth checkpoint than the one behind Figures 2 and 3, so its detection numbers are not directly comparable to those figures. As Table 1 shows, the math training genuinely took hold, yet the bias barely moved, and a cartridge distilled from the post-trained model surfaced it essentially as well as before.

Whether harder downstream training could eventually bury the signal is the wrong worry, because the two ways that story can end both favor the defender. Extra changes piled into the weights defeat weight-space methods such as model diffing, but they do not change what the model expresses, and whatever the model expresses is what the cartridge captures. And if continued training ever did scrub the preference out of the model's behavior, it would have erased the very bias someone meant to plant. The paper works through both sides of this argument in more detail.

D2D is an amplifier, not a detector

What do you do with the adapted model once distillation finishes? You treat it like any ordinary model, because the output of D2D is itself just a model. We hand ours to Petri and AuditBench unchanged, but the same artifact could go to a human red team, a bias benchmark, or whatever auditing tool comes next, and none of them need to know how it was produced. This is the sense in which D2D is an amplifier rather than a detector, and it is what lets the method compose with the rest of the auditing ecosystem instead of competing with it. As behavioral auditors improve, D2D inherits their gains for free. The mechanism itself is an old observation put to new use, since distillation and pruning have long been known to strengthen whatever patterns dominate a model,See Characterising Bias in Compressed Models (Hooker et al, 2020) and The Pitfalls of Simplicity Bias in Neural Networks (Shah et al, 2020). and that amplification is usually treated as a failure mode to engineer away rather than as a way of asking what a fine-tune most consistently changed.

What's next

The paper carries more than this post does, including the formal analysis behind the concentration argument, a sweep of cartridge capacities that traces out the inverted-U the theory predicts, the second auditor's full results, and the complete injection and audit protocols. For any of those, the paper is the place to start.

We also want to be plain about scope. Everything here concerns biases planted through context distillation, a deliberate choice because it mirrors fine-tuning workflows providers actually run and is invisible to data-level monitoring, but a real boundary nonetheless. Other injection routes, such as data poisoning or backdoor triggers, may not produce the coherent low-rank shift that the capacity bottleneck exploits, and testing amplification against them is the extension we care about most. Beyond that sit the biases that resist reduction to a few tokens, political framing and demographic stereotypes among them, along with other model families and the black-box setting where the defender has no logits at all.

Auditing has always favored the attacker, who knows the bias, over the defender, who searches blind. We find it encouraging that compression shifts this balance, because a preference has to fire consistently to steer decisions at scale, and consistency is exactly the property that survives a capacity bottleneck.

Citation

Please cite this work as:

@misc{talaei2026distilldetectexposingstealth,
      title={Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation},
      author={Shayan Talaei and Abhinav Chinta and Devvrit Khatri and Amin Karbasi and Azalia Mirhoseini and Amin Saberi},
      year={2026},
      eprint={2607.01208},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.01208},
}