Nature vs. Nurture: Limits of Representation Universality Reveal Architectural and Training Differences in Image Models

Recent works have hypothesized and demonstrated the general inter-compatibility and translatability of vector embeddings in learned embedding spaces. We study the limits of representation compatibility across image models encompassing a variety of architectures and training paradigms. We train adapters, comprised of linear encoders and decoders, to translate between all pairs of embedding spaces using a shared latent space. We then evaluate cross-model and cross-domain compatibility using classification, retrieval, and embedding-based benchmarks, revealing limits in compatibility across learned embedding spaces. Further analysis of pairwise translation performance reveals phylogenetic patterns that reflect the fundamental differences in model architecture and training.