In the first two parts of this series, we covered why Netflix needed a Real-Time Distributed Graph (RDG) and how we built an ingestion pipeline using Apache Flink. In Part 2, we explored the storage layer capable of handling billions of nodes and edges with single-digit-millisecond latency.
In this post, we focus on the serving layer. All the work on ingestion and storage only matters if we can actually ask complex questions and get answers back quickly. How do we turn a constantly evolving, billion-edge graph into sub-100ms responses across a wide variety of workloads?
The challenge: diverse query patterns
"Querying the graph" is not a one-size-fits-all operation. We need to handle a wide range of access patterns: from high-volume security lookups to deep, exploratory personalization traces.
Graph queries vary along two axes: how wide they fan out at each hop, and how deep they chain across hops. Consider two scenarios from opposite ends:
Shallow, wide query: "Which devices has this account used to stream in the last 30 days?" For a highly active account, the fan-out can be massive. The query layer must fetch hundreds of edges, apply temporal filters, and aggregate results, all while maintaining sub-100ms latency.
Deep, narrow query: "For Account X, show me the Stranger Things viewing history across all profiles: which profiles watched it, what they watched, and when." The challenge here is sequential dependency—we can't fetch a profile's viewing history until we know which profiles exist.
These two scenarios pull the system in opposite directions. Shallow-wide queries stress I/O throughput. Deep-narrow queries stress execution efficiency. Supporting both on the same system shaped our design.
Design decisions
Breadth-first traversal
We use breadth-first traversal: working one level at a time across all nodes, rather than one path at a time through each node. Fetch all profiles for the account at once, then fetch edges for all profiles, then fetch content details for all matching titles. Three rounds of parallel calls instead of sequential chains.
The trade-off is memory—we hold each level of the graph in memory at once—but we bound each hop with per-edge-type limits, so even high fan-out stays manageable.
Async-first architecture
Latency is dominated by I/O. A traditional thread-per-request model would pin a thread to each in-flight query, most of which would be idle waiting for network responses. Instead, we built the entire execution pipeline around asynchronous composition. A small set of dedicated thread pools (16-24 threads total) handles thousands of concurrent requests because no thread ever blocks on I/O.
Caching strategy
Not all data changes at the same rate. Some properties (account plan type, content metadata) are relatively stable. Edges change constantly. For stable data that many queries touch, we use a distributed cache (EVCache) with TTLs tuned to data volatility.
We started by caching aggressively and measured the impact: tracking hit rates, monitoring stale-data incidents, and adjusting TTLs based on how quickly different node types actually changed in production. The result: 70-80% hit rates on node lookups.
Opt-in enrichments
Clients know what they need. A query checking account relationships doesn't care about title artwork; a personalization service does. Rather than fetching metadata by default, we make enrichments opt-in. Also, enrichment is fail-open: if a service is slow or unavailable, we return the graph data without it.
Query execution: a walkthrough
Let's follow a query through the system: "For Account X, show me the Stranger Things viewing history across all profiles."
Step 1: Parse the request. Before touching storage, the engine creates a traversal plan with levers: how many hops, how many edges per hop, how much history to consider. We resolve these by merging a hierarchy of filters and limits into a concrete execution plan.
Step 2: Fetch the starting node. For Account X, we look up the node and its adjacency lists. For each node, we keep a compact list of "who it's connected to" by edge type. "Get all profiles for Account X" is a direct lookup into Account X's stored adjacency, not a global search.
Step 3: Stream and filter. Each profile can have a large number of edges. Instead of loading the entire adjacency list at once, the storage layer streams them in batches of 100. As each batch arrives, we apply filters (e.g., "last 30 days") and decide whether to continue. If we've collected enough edges to satisfy the query's limits, we stop reading.
Step 4: Parallel execution. At Level 2, we fetch edges for each profile in parallel. We use dedicated thread pools for different work types: fetching nodes, reading adjacency lists, and performing enrichments. When the query reaches Level 2, calls route to the adjacency-list pool, where workers stream and filter each profile's edges in parallel.
We also use adaptive concurrency limiting. When things are healthy, we raise the limit gradually; when timeouts or errors spike, we back off by a larger step. Combined with per-pool limits, the engine constantly tunes parallelism, fanning out within each level while staying inside safe storage and network limits.
Step 5: Filtering hierarchy. Raw edges aren't what partners need. They care about recent, relevant activity. We handle this with a filtering hierarchy: conservative defaults (100-day lookback, 300 edges per hop) that requests can override globally, per-hop, or down to specific edge types.
We also offer two selection modes: LATEST sorts edges by timestamp and keeps the newest ones up to the limit; ANY grabs whichever edges it encounters first, which is faster and fine for "has this profile ever watched Stranger Things?"
Step 6: Caching hot nodes. Despite optimizations, each storage call still costs a network round-trip. When the same nodes appear across thousands of queries per minute, redundant calls add up. We keep a distributed cache of hot nodes (accounts, profiles, content) that are likely to reappear. When the same entity appears again, we answer from memory, skipping storage.
We can't cache everything. The RDG prunes old activity after a retention window, so caching a node about to be deleted is wasteful. We use "smart TTL" policies—matching TTLs to data volatility—to reserve cache space for active nodes.
Results
The serving layer handles mixed workloads, all needing to feel interactive. Single-hop queries return at a P50 of 15-30ms with P99 under 100ms. Even 3-hop traversals come back at P99 between 100-150ms. Breadth-first execution and parallelism keep these numbers stable even as fan-out grows.
The async-first design enables the throughput. Thousands of concurrent requests flow through just 16-24 threads because no thread ever blocks on I/O.
Caching has the most visible impact on day-to-day efficiency. Popular entities achieve 70-80% cache hit rates, resulting in roughly 3-4x fewer storage calls on common query paths.
Lessons learned
The biggest surprise wasn't any single optimization—it was how much async composition changed the economics of our system. We expected it to help latency; we didn't expect it to slash infrastructure cost. A serving layer that would have needed hundreds of threads per instance runs comfortably on 16-24.
The trade-off is debuggability. Async stack traces are hard to read, and exceptions can get lost in future chains. We compensated with per-stage metrics, measuring each request at validation, storage, enrichment, and end-to-end, so when something is slow, we know exactly which stage to blame.
Caching took longer to get right than expected. Our first instinct was to cache everything and let TTLs handle freshness, but that wastes memory on nodes about to expire from the graph anyway. The breakthrough was matching TTLs to data volatility.
The filtering hierarchy was born out of frustration. Early on, every new use case meant a code change. Instead of bespoke logic per team, we built a layered override system that eliminated an entire class of feature requests.
Principles for distributed systems
- Design APIs so callers describe what frontier to explore, then let the system decide how to walk it efficiently.
- Push filters and limits as close to the storage layer as possible. Discard irrelevant data at each stage rather than fetching everything and trimming at the end.
- Set explicit concurrency limits, monitor them, and adjust dynamically. Treat concurrency as a dial, not a switch.
- Decide what is worth remembering, for how long, and what should be allowed to fade. Match TTLs to data volatility, and don't cache what's about to expire.
Getting these details right is what turns a constantly changing, billion-edge graph into something that, at query time, feels like a responsive, in-memory data structure.