Divergent Tasks Harm Integration Of New Entities Via Fine-Tuning

When a pretrained model is fine-tuned to incorporate new entities, does this knowledge automatically generalize across tasks? We investigate this question using a controlled framework in which transformers pretrained on geometric tasks over real-world city coordinates are fine-tuned to integrate synthetic out-of-distribution cities. We find that cross-task generalization varies dramatically depending on the fine-tuning task, and that this variation is partially predicted by representational similarity (CKA) measured during pretraining. In multi-task fine-tuning, we identify lurking divergent tasks that not only fail to generalize but actively harm performance on other tasks. Probing suggests that divergent tasks encode new entities in separate subspaces rather than integrating them into the shared world manifold.