Improving LLM Predictions via Inter-Layer Structural Encoders

Standard LLM inference relies on final-layer representations, despite evidence that intermediate layers often capture task-specific information more effectively. However, identifying the optimal layer remains task-dependent and computationally expensive. In this work we introduce Inter-Layer Structural Encoders (ILSE), an approach to learn one representation from the LLM’s internal layer representations all together. Central to ILSE is Cayley-Encoder, a geometric encoder which builds upon recent studies leveraging Cayley Graphs for neural information propagation. We evaluate ILSE across 13 classification and semantic similarity tasks with 2 pretrained LLMs. ILSE consistently outperforms baselines and existing approaches, achieving up to 40% improvement in accuracy and 22% in similarity metrics.