The decision to self-serve
Most organizations consume LLMs through hosted APIs. Netflix chose the harder path: run the full stack in-house, from model deployment through inference, inside the same production environment that runs the rest of the company's ML. The reasoning isn't philosophical. It's operational. The team wanted the same control plane, the same deployment mechanics, the same observability, and the same on-call surface as the rest of Netflix's ML serving.
The architecture is layered. The member-scale ML serving system that fronts most of Netflix's recommendations is a JVM-based service that handles routing, A/B logic, candidate generation, feature fetching, inference, post-processing, and logging. LLM inference plugs into this same flow. Smaller models run in-process to avoid the remote-call overhead. Larger models need GPUs, and they hit a remote service, the Media Streaming Service (MSS), which is Netflix's shared inference backend supporting XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface. Triton sits underneath, managing model loading, batching, and GPU scheduling, with a Java control plane handling deployment, versioning, health checks, autoscaling, and multi-region rollout.
Engine selection: why vLLM
The original platform ran on TensorRT-LLM. It was the most performant inference engine at the time, and it was already integrated with Triton. By summer 2025, two things had changed. Open-source engines had closed the performance gap. The workload mix had broadened to include embedding generation, prefill-only inference for ranking and retrieval, autoregressive decoding, and custom models with non-trivial per-step constraint logic.
The team re-benchmarked against the new mix. vLLM won on two dimensions. Failures and intermediate state are easier to inspect than with a compiled engine, which matters when production behavior diverges from the lab. And many of the ML practitioners Netflix was hiring were already using vLLM in research, which cut the research-to-production handoff cost. The performance tradeoff was real but acceptable, and the operational and ergonomic gains more than made up for it.
Model packaging
With vLLM selected, the next decision was how to package models for it. Triton supports two paths, and the choice has significant implications for how easy it is to maintain the platform over time. The team picked a packaging model that lets model authors ship a versioned artifact and a small configuration, with the control plane handling provisioning and rollout. The trade-off the team accepted is that the platform has to be smarter about the artifacts it accepts, because a bad artifact can now affect more of the serving surface than it did under TensorRT-LLM.
API surface
The API design is shaped by the fact that some callers want a gRPC path through the existing serving system and some callers want a direct HTTP path. The HTTP path serves the newer LLM-driven applications, where the calling team doesn't want to take a dependency on the JVM-based serving system. The gRPC path serves the existing recommendation flow, where the integration has been working for years and the team doesn't want to disturb it.
The design philosophy is: expose the smallest possible surface, version it carefully, and let the serving system handle the rest. The team learned the hard way that a richer API surface is a surface that has to be supported forever, and the cost of supporting a richer surface in a fast-moving model ecosystem is steeper than the cost of asking callers to do a little more work themselves.
Rollout
The rollout strategy is multi-region with health gates, and the team has been explicit that the rollout has to handle the case where a new model version is genuinely worse in production than it was in offline evaluation. The control plane is designed to roll back, not just roll forward, and the rollback path is exercised regularly enough that the team is confident it works.
What production revealed
The design phase anticipated most of the trade-offs. The thing the design phase did not anticipate was the volume of constraint logic that would end up in custom models. Per-step constraints, where the model has to maintain some structural property of the output across decoding steps, are a class of work that the inference engines handle inconsistently. The team has built a layer on top of the engine to make the constraints uniform, and that layer is now load-bearing for the rest of the system. None of the benchmarks in the design phase covered it, because none of the production models had it at the time.