Most organizations consume large language models through hosted APIs. Netflix went a different direction, building a full-stack LLM serving platform inside its existing production environment[reference:30]. The platform, built on vLLM and NVIDIA Triton, gives ML practitioners a fast path from experimentation to production[reference:31].
The Architecture
Netflix's LLM serving architecture starts with a unified JVM-based serving system that handles the end-to-end flow for downstream consumers: routing, A/B test logic, candidate generation, feature fetching, inference, post-processing, and logging. Both real-time and cached batch paths are supported.
Small CPU models run in-process to avoid remote-call overhead. Larger models need GPUs. The serving system handles pre- and post-processing locally but delegates inference to a remote Model Scoring Service (MSS). MSS supports XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface, with NVIDIA Triton Inference Server underneath managing model loading, batching, and GPU scheduling[reference:32].
Engine Choice: Why vLLM?
The platform was originally built on TensorRT-LLM. By summer 2025, two things had shifted: open-source engines had largely closed the performance gap with specialized stacks, and the workload mix had broadened to include embedding generation, prefill-only inference, autoregressive decoding, and custom models with constraint logic. vLLM won out because it's easier to inspect failures and intermediate state, and because many ML practitioners were already using vLLM in research[reference:33].
Packaging and Deployment
The choice of how to package models has significant implications. The platform uses Triton's vLLM backend, which reads a JSON config pointing to model weights and generates I/O tensor specs dynamically at deployment time[reference:34]. This allows models and the frontend to evolve independently. However, version pinning is critical. When Triton and vLLM drift, the backend fails to load entirely[reference:35].
For deployment, the platform offers two strategies. Red-Black deployment works when the model interface is stable. Blue-Green deployment solves the problem of I/O schema changes by maintaining independent deployments for every model version, allowing consumers to switch configs only when the new version is fully ready[reference:36].
Constrained Decoding at Scale
The platform pushes constraints inside the decode loop, so models generate outputs that are compliant by construction. This is implemented via vLLM's custom logits processor interface. In vLLM V0, custom logits processors ran per-request, creating a CPU bottleneck that grew linearly with batch size. The migration to vLLM V1 in Q4 2025 moved logits processing to the batch level, with the hot path reimplemented in C++ for multi-threading[reference:37].
Lessons Learned
Building an in-house LLM serving platform revealed several lessons. Version pinning between Triton and vLLM requires careful management. Silent API gaps, like response_format being silently dropped before reaching vLLM, need patching. And constrained decoding at scale requires moving from per-request to batch-level processing.
Netflix's platform demonstrates that running LLMs in-house is viable at scale, but it requires deep integration with existing infrastructure and a willingness to patch and extend open-source tools. The payoff is flexibility, lower latency, and full control over the serving stack[reference:38].