On June 6, 2024, OpenAI Breaks Down GPT-4 into 16 Million Interpretable Features

On June 6, 2024, OpenAI introduced a groundbreaking technique to break down GPT-4 into 16 million interpretable features. This advancement is achieved through improved methods for training sparse autoencoders at scale, which help in disentangling GPT-4’s internal representations into features that often correspond to understandable concepts. This work marks significant progress in understanding the neural activity of language models and surpasses human performance in Theory of Mind. The new methods scale better than existing work and are completely unsupervised.