The engineering team at Zalando recently described the design and implementation of an in-process client-side load balancer for a high-throughput API handling around 1 million requests per second[reference:51][reference:52]. The result was more predictable latency, a drop in infrastructure costs, and better visibility into where failures actually originate[reference:53].
The Problem
Zalando's Product Read API is designed to serve millions of requests per second with single-digit-millisecond latency across the 25 markets of one of the largest European online fashion retailers[reference:54]. Its batch endpoint fans a single request into up to 100 parallel calls to individual product pods, each transiting Skipper, the shared cluster edge load balancer[reference:55].
As a batch waited on the slowest of 100 hops through infrastructure the team did not own, they could not separate Skipper latency spikes from their own[reference:56]. To address that, they moved routing for high fan-out internal traffic in-process, while keeping Skipper for edge and single-GET traffic[reference:57].
The Solution
A senior principal engineer at Zalando explained: "We decided that for high fan-out internal traffic, the routing decision should live inside the calling process itself. Edge traffic, where Skipper excels, should stay exactly where it is. We were not replacing Skipper; we were graduating the internal fan-out path to a client-side load balancer that runs inside the process."
The team reimplemented Skipper's exact algorithm (xxHash64, 100 virtual nodes per endpoint) so both paths route identical keys to identical pods during migration, avoiding cache splits, with unit tests pinning this behavior[reference:58]. This means that adding or removing an endpoint remaps only about 1/N of keys, minimising cache churn.
The Results
The effect was immediate. Latency dropped. Spikes that had been attributed to "the network" or "something in the stack" disappeared[reference:59]. At peak, over a million requests per second were re-routed off Skipper[reference:60].
The team replaced control-plane-crashing polling with a watch-based Kubernetes informer, rebuilt a pipeline that took hours into fast, reversible deploys, and ramped the load balancer from 1% to 100% behind toggles. This shrank Skipper's fleet from more than 50 pods to 8 and cut deployment daily cost from 110[reference:61].
To eliminate scale-up latency spikes, they added N-ring fade-in over 30 seconds on a ^2.5 curve so new pods warm only the products they will actually serve[reference:62].
Industry Reaction
Werner Vogels, CTO of Amazon, commented: "I am not a big fan of doing this client-side, for a whole slew of reasons, but this is impressive engineering by the Zalando team. I like the lessons learned they conclude with."
Others noted that client-side load balancing was a revelation when moving from Amazon to Google, suggesting that this thinking needs to be more prominent.
Lessons Learned
Highlighting that Zalando built its own client-side load balancer only because they sat at a genuine extreme edge case, the lead engineer concluded: "Should you build your own? For almost everyone, the answer is don't. A mature proxy like Skipper or Envoy gives you consistent-hash routing out of the box, maintained by someone else and hardened by thousands of other users."
Some observers noted that it felt weird that they treated Skipper as a third-party dependency, even though it is developed and operated by their colleagues. They could have tried contributing updates and improvements to Skipper.
While Skipper is open source and available on GitHub, the new client-side load balancer is not published and remains internal to Zalando.