(Figure courtesy: Kiho Park)

Figure: Representations of Hierarchical and Categorical Concepts in the Gemma-2B LLM

(Figure courtesy: Kiho Park)

Abstract

Understanding how semantic meaning is encoded in the representation spaces of large language models is a fundamental problem in interpretability. In this paper, we study the two foundational questions in this area. First, how are categorical concepts, such as {‘mammal’, ‘bird’, ‘reptile’, ‘fish’}, represented? Second, how are hierarchical relations between concepts encoded? For example, how is the fact that ‘dog’ is a kind of ‘mammal’ encoded? We show how to extend the linear representation hypothesis to answer these questions. We find a remarkably simple structure: simple categorical concepts are represented as simplices, hierarchically related concepts are orthogonal in a sense we make precise, and (in consequence) complex concepts are represented as polytopes constructed from direct sums of simplices, reflecting the hierarchical structure. We validate these theoretical results on the Gemma large language model, estimating representations for 957 hierarchically related concepts using data from WordNet.

Note

Best Paper Award, ICML 2024 Workshop on Mechanistic Interpretability

Citation

Park, K., Choe, Y. J., Jiang, Y., & Veitch, V. (2024). The Geometry of Categorical and Hierarchical Concepts in Large Language Models. arXiv preprint arXiv:2406.01506.

@article{park2024geometry,
  title={The Geometry of Categorical and Hierarchical Concepts in Large Language Models},
  author={Park, Kiho and Choe, Yo Joong and Jiang, Yibo and Veitch, Victor},
  journal={arXiv preprint arXiv:2406.01506},
  year={2024}
}