microgpt: 200 Lines of Pure Python That Trains a GPT

Andrej Karpathy has done it again. The former OpenAI member and Tesla AI director has released microgpt, a single file of 200 lines of pure Python with no dependencies that trains and inferences a GPT[reference:13][reference:14]. This is the culmination of a decade-long obsession to simplify LLMs to their bare essentials, following his previous projects like micrograd, makemore, and nanogpt[reference:15].

The Algorithmic Essence

microgpt contains the full algorithmic content needed to train a GPT: a dataset of documents, a tokenizer, an autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, and training and inference loops[reference:16]. Everything else is just efficiency.

The script uses a tiny dataset of 32,000 names, one per line, and learns to generate new, plausible-sounding names. The model has just 4,192 parameters—compared to GPT-2's 1.6 billion—but the core mechanism is identical[reference:17].

How It Works

Karpathy walks through every component in his blog post:

• Tokenization: The simplest possible tokenizer assigns one integer to each unique character (a-z) plus a special BOS token, for a vocabulary of 27.
• Autograd: A from-scratch implementation of backpropagation on scalars, using a Value class that tracks computation and computes gradients via the chain rule. This is the same algorithm PyTorch runs, just on scalars instead of tensors.
• The Model: A GPT-2-like architecture with embedding tables, attention heads, and MLP layers. It processes one token at a time with an explicit KV cache.
• Training: The loop picks a document, runs the model forward, computes loss, backpropagates, and updates parameters with Adam.
• Inference: Sampling from the model by feeding generated tokens back as input, with a temperature parameter controlling randomness.

The Bigger Picture

microgpt is a beautiful educational artifact. It demonstrates that the "magic" of large language models is just math: a big function that maps input tokens to a probability distribution over the next token[reference:18]. The model has no concept of truth; it only knows what sequences are statistically plausible given the training data. "Hallucination" in microgpt—generating a name like "karia"—is the same phenomenon as ChatGPT confidently stating a false fact.

From microgpt to ChatGPT

Between microgpt and a production LLM like ChatGPT, there's a long list of things that change. None of them alter the core algorithm, but they make it work at scale:

• Data: Trillions of tokens of internet text instead of 32K names.
• Tokenization: Subword tokenizers like BPE with ~100K tokens instead of character-level.
• Hardware: Tensors on GPUs instead of scalars in pure Python.
• Scale: Hundreds of billions of parameters instead of 4,192.
• Post-training: SFT and RL to turn a document completer into a chatbot.
• Serving: KV cache management, quantization, and speculative decoding for speed.

Why It Matters

microgpt is a reminder that understanding the core algorithm is more important than keeping up with the latest engineering tricks. The script runs in about 1 minute on a MacBook, and Karpathy encourages readers to play with it: try a different dataset, train longer, or make the model bigger. It's the cleanest possible introduction to how LLMs actually work[reference:19].