The Numbers
Gigatoken, released by Stanford PhD student Marcel Rod under an MIT license, encodes text at 24.53 GB/s on a 144-core AMD EPYC 9565 dual-socket setup. In comparison, OpenAI's tiktoken achieves 36.0 MB/s and HuggingFace tokenizers registers at 24.8 MB/s on identical hardware. That is a 681x speedup over tiktoken and 989x over HuggingFace tokenizers.
On an Apple M4 Max with 16 cores, the same GPT-2 workload runs at 8.79 GB/s, or 1,268x HuggingFace and 140x tiktoken. On a consumer AMD Ryzen 7 9800X3D, it runs at 6.27 GB/s, or 106x and 68x. The speedup holds across x86 and ARM architectures.
What It Is
Gigatoken is a byte-pair encoding (BPE) tokenizer written in Rust with Python bindings, shipping on PyPI as gigatoken (version 0.9.0, released 21 July 2026). It supports 23 distinct tokenizer families in published benchmarks, covering GPT-2, GPT-OSS, Llama 3 through 4, Qwen 2 through 3.6, DeepSeek V3/R1/V4, GLM 4 and 5, Kimi K2, Nemotron 3, Phi-4, OLMo 2/3, ModernBERT, Gemma, and Mistral.
There are two usage modes. Compatibility mode wraps an existing HuggingFace or tiktoken tokenizer and preserves exact output parity, delivering roughly 200-300x speedup. The native Gigatoken API lets Rust read files directly and is where the headline numbers come from.
How It Gets There
The gains do not come from a better BPE merge loop. They come from two places most tokenizers treat as solved.
Pretokenization: Most implementations delegate this to a regex engine. Gigatoken hand-writes it. The optimization progression on 100 MB of OpenWebText is instructive: a fancy-regex baseline at ~47 MiB/s, a hand-rolled state machine at ~380 MiB/s, a winnow-combinator with NEON SIMD at 462 MiB/s, then a 256-byte class lookup table with SWAR (SIMD Within A Register) at 830 MiB/s. The final step is dual-cursor ILP exploitation, reaching 1,049 MiB/s by running two independent cursors from a safe split point, letting the out-of-order engine interleave both streams across idle execution ports.
Pretoken caching: If a word has been seen before, its encoded tokens are looked up rather than recomputed. The author notes this is hard in practice because the cache grows quickly and pretoken distributions are long-tailed.
Honest About What Failed
The optimization log is unusually candid. A hot/cold split using #[cold] and #[inline(never)] regressed to 580 MiB/s and was reverted. A two-pass classification buffer with SWAR transition counting was algorithmically correct but ran at 354 MiB/s. Profile-guided optimization had no measurable effect.
Benchmark Caveats
The comparison is not strictly apples-to-apples. Gigatoken encodes entire un-split files, finding its own document boundaries and parallelizing automatically. HuggingFace (encode_batch_fast) was evaluated on the first 100 MB and tiktoken (encode_ordinary_batch) on the first 1 GB, both pre-split on <|endoftext|>. An independent reproduction on KrabArena verified the trend: on a 4-vCPU Intel Xeon VM, Gigatoken achieved 277.8 MB/s, outperforming tiktoken by 26.2x and tokenizers by 83.4x.
SentencePiece vocabularies see 7-22x speedups, not 1,000x. WordPiece is unsupported.