Andrej Karpathy, a founding member of OpenAI and former AI chief at Tesla, has done it again. He has released microGPT, a project that fits the entire algorithmic essence of a GPT into a single file of 200 lines of pure Python with zero dependencies[reference:0][reference:1]. It trains a model and runs inference. No PyTorch, no TensorFlow, no CUDA. Just Python and the fundamental math that makes large language models work[reference:2].
The Soul of a GPT in 200 Lines
The script implements everything needed to train a character-level GPT-2-like model[reference:3]. It includes a dataset loader, a character-level tokenizer, a custom autograd engine, the Transformer architecture, the Adam optimizer, and training and inference loops. All of this runs on a dataset of 32,000 names, training a model with just 4,192 parameters.
The result? The model learns the statistical patterns of names and generates new, plausible-sounding ones. Samples like "kamon," "jaire," and "vialan" emerge from the randomness. From the perspective of a model like ChatGPT, your conversation is just a funny looking document. When you initialize the document with your prompt, the model's response is simply a statistical document completion.
How MicroGPT Works
The Tokenizer. Neural networks work with numbers, not characters. MicroGPT uses the simplest possible tokenizer: it assigns one integer to each unique character in the dataset. With 26 lowercase letters and a special Beginning of Sequence token, the vocabulary size is just 27.
The Autograd Engine. Training a neural network requires gradients. For each parameter, we need to know if nudging it up a little makes the loss go up or down. MicroGPT implements backpropagation from scratch in a single class called Value. It wraps a scalar number, tracks how it was computed, and knows the local derivative of each operation. The backward method walks the computation graph in reverse topological order, applying the chain rule at each step. This is the same algorithm PyTorch runs, just on scalars instead of tensors. Algorithmically identical, significantly smaller, and of course a lot less efficient.
The Architecture. The model follows GPT-2 with minor simplifications: RMSNorm instead of LayerNorm, no biases, and ReLU instead of GeLU. It processes one token at a time with a KV cache. The token and position embeddings are added together. Attention lets tokens look at past tokens. The MLP does the per-position thinking. Both blocks use residual connections. The final hidden state projects to vocabulary size, producing logits for the next token.
The Training Loop. One document at a time. The model predicts each next token. The loss is the negative log probability of the correct token. Backpropagation flows through the entire graph. Adam updates the parameters. The script runs about 1,000 steps, taking roughly one minute on a MacBook. The loss drops from around 3.3 to about 2.37.
Sampling. Once trained, the parameters freeze. We start with the BOS token, sample from the probability distribution, feed the result back in, and repeat until the model produces another BOS token or hits the maximum length. A temperature parameter controls randomness. Lower temperatures make the model more conservative; higher temperatures produce more diverse but potentially less coherent output.
From MicroGPT to ChatGPT
MicroGPT contains the complete algorithmic essence of training and running a GPT. But between this and a production LLM like ChatGPT, there is a long list of things that change. None of them alter the core algorithm, but they are what make it actually work at scale.
Data. Instead of 32,000 short names, production models train on trillions of tokens of internet text: web pages, books, code, and more. The data is deduplicated, filtered for quality, and carefully mixed across domains.
Tokenization. Instead of single characters, production models use subword tokenizers like BPE, which learn to merge frequently co-occurring character sequences into single tokens. Common words like "the" become a single token. This gives a vocabulary of around 100,000 tokens and is much more efficient.
Hardware. MicroGPT operates on scalar objects in pure Python. Production systems use tensors and run on GPUs that perform billions of floating point operations per second. Libraries like PyTorch handle the autograd over tensors, and CUDA kernels like FlashAttention fuse multiple operations for speed. The math is identical; it just corresponds to many scalars processed in parallel.
Scale. MicroGPT has 4,192 parameters. GPT-4 class models have hundreds of billions. Modern LLMs also incorporate RoPE, GQA, gated linear activations, and Mixture of Experts layers. But the core structure of Attention and MLP interspersed on a residual stream is well-preserved.
Optimization. Production training uses large batches, gradient accumulation, mixed precision, and careful hyperparameter tuning. Training a frontier model takes thousands of GPUs running for months. Scaling laws guide how to allocate compute between model size and training tokens.
Post-Training. The base model that comes out of training is a document completer, not a chatbot. Turning it into ChatGPT happens in two stages. First, Supervised Fine-Tuning: you swap the documents for curated conversations and keep training. Algorithmically, nothing changes. Second, Reinforcement Learning: the model generates responses, they get scored, and the model learns from that feedback.
Serving. Serving a model to millions of users requires batching requests, KV cache management, speculative decoding, quantization, and distributing the model across multiple GPUs.
What MicroGPT Teaches Us
MicroGPT is not a production system. It is a teaching tool, an art project, and a reminder that the "soul" of a GPT fits in 200 lines of code[reference:4]. Most of the "magic" is just matrix multiplications and softmax operations repeated at scale[reference:5]. The model has no concept of truth; it only knows what sequences are statistically plausible. When microGPT "hallucinates" a name like "karia," it is the same phenomenon as ChatGPT confidently stating a false fact. Both are plausible-sounding completions that happen not to be real.
Karpathy has created a Gist called microGPT where, in the revisions, you can see all the versions and the diffs between each step. It is a way to step through the code base, adding one component at a time. This is the culmination of multiple projects and a decade-long obsession to simplify LLMs to their bare essentials. And it is beautiful.