Transformers have revolutionized Computer Vision (CV) through self-attention mechanisms. However, due to their complexity, their latent token representations are often difficult to interpret. We propose a framework to interpret Transformer embeddings, revealing semantic patterns. Based on this framework, we demonstrate that zero-shot unsupervised semantic segmentation can be performed effectively without any fine-tuning using a model pre-trained for tasks other than segmentation. Our method showcases Transformers' innate semantic understanding, surpassing traditional models. It attains 67.2\% accuracy and 32.9\% mIoU on COCO-Stuff and 51.9\% mIoU on PASCAL VOC.