Study finds 'emergent misalignment' depends on data design, not model scale

Three researchers — Miriam Wanner, Mark Dredze and William Walden, working from Johns Hopkins University — published a paper questioning how easily the “emergent misalignment” effect actually generalises. That 2025 finding showed that fine-tuning a model on one narrow bad behaviour, such as writing insecure code, could make it endorse harmful positions on entirely unrelated topics.

The authors, who use the broader term “weird generalization” for the phenomenon, tested which properties of a fine-tuning dataset actually drive the effect, varying its size, composition, language, presentation style, and how novel its examples were relative to a model’s pretraining data. They reported that the degree of generalisation depended heavily on a dataset’s composition and language — far more than on how large the dataset was — and was more pronounced when fine-tuning examples resembled material a model had likely already seen in pretraining, rather than genuinely novel content. The effect also proved sensitive to exactly which questions were used to evaluate it afterwards, meaning some previously reported instances may reflect the particular evaluation chosen as much as a stable property of the model.

On that basis, the authors argued the phenomenon is “more plausible as an adversarial threat-requiring careful data engineering-rather than as a significant hazard inherent to routine fine-tuning,” a more measured framing than some of the original coverage suggested. The paper adds to a run of 2025 and 2026 follow-ups, including studies from OpenAI and Anthropic, that have each tried to pin down the mechanism behind narrow-training generalisation rather than simply reproducing the original result.