How text becomes the next letterhover a box to learn what it does · click it to open it up
drag to rotate · scroll to zoom · right-drag to move
Asking the model…
What is this transformer visualization?
GPT Anatomy is an interactive visualization of a transformer, the neural network behind GPT. It opens up a small GPT language model (10.8 million parameters, 6 layers, 6 attention heads) that was built from scratch in PyTorch and trained on Shakespeare, so you can see what happens at every step between the text you type and the next letter it predicts. Every number on the page comes from the trained model.
How a transformer turns text into the next letter
Tokenization. The text is split into tokens. This model works on single characters, so each of its 65 characters gets an id.
Embeddings. Each id becomes a vector of numbers, and a position vector is added so the model knows the order of the tokens.
Self-attention. Every token makes a query, a key and a value. Queries are compared with keys to decide which earlier tokens matter, a mask hides the future, and the values are mixed using those weights. Six attention heads do this in parallel.
Layer norm and residual connections. These keep the numbers stable and let each block add a small correction to what it receives.
Feed-forward network. Each token is processed on its own by a small two-layer network.
Next-token prediction. The final vector becomes a score for each character, softmax turns the scores into probabilities, and one character is drawn.
Is it a chatbot?
No. It is a next-token predictor trained on one task: guess the next character. That is the same core idea behind large language models, only small enough to see completely. Type the start of a line in the chat and watch it work one letter at a time, with play, pause and step controls.
About the project
Made by Chitraksh Suri while following Andrej Karpathy's "Let's build GPT" lecture. The source code is on GitHub.