A small algorithmic change at global scale

Cloudflare operates thousands of servers with petabytes of RAM and millions of CPU cores, according to the company. Because its services must run across every node, even small reductions in resource use can have a significant effect when multiplied across the network.

In a blog post, Cloudflare said changes to a single algorithm reduced the memory footprint of one of its Pingora-based services enough to reclaim more than 100TB of RAM globally. The recovery came on top of 100TB of memory that the company's DNS team had previously been able to shed, according to the post.

The investigation began with Pingora Backend Router

The work started with an internal ticket from an engineer named Ivan, who identified excessive memory usage in pingora-ketama within Pingora Backend Router, or PBR. Cloudflare describes PBR as an internal load-balancing service, while pingora-ketama is its open-source library for consistent hashing.

The company's Performance team investigated the issue as part of Cloudflare's effort to keep resources shared equitably between teams. The finding was that structures associated with pingora-ketama were consuming significantly more memory than expected.

Why Cloudflare uses consistent hashing

Consistent hashing distributes tasks across servers while limiting the changes required when servers are added or removed. Cloudflare uses the technique to route cacheable requests to servers by URL. This lets the company keep one copy of a file per data center while maintaining a stable way to locate that file.

The method maps both servers and tasks to values produced by a hash function. Although hash functions can accept many forms of input, their output is a bounded unsigned integer, such as a 32-, 64- or 128-bit value. Consistent-hashing explanations often visualize those values on a circular ring, where the range assigned to a server wraps from the maximum value back to zero.

Random ranges can create uneven workloads

At its simplest, a task is assigned to the first server to the left of its position on the hash number line. The portion of the line associated with each server determines the share of work it receives. Because hash outputs behave like random values, those portions are not automatically equal.

Cloudflare used expected value and standard deviation to describe the uncertainty in each server's assigned range. With 100 servers, the expected range for each server is close to 0.99% of the total, but the variation around that expectation can still be meaningful. The company also examined the coefficient of variation, which expresses the standard deviation as a fraction of the target size.

More hashes improve balance, but use memory

The standard way to reduce imbalance is to represent each server with multiple hashes rather than a single hash. The individual ranges remain random, but adding more ranges gives unusually short and unusually long segments more opportunities to offset one another. Cloudflare described this as an effect associated with the law of large numbers.

The approach has a direct resource cost: every additional hash requires storage. The post says NGINX uses a baseline of 160 hashes per server, and Pingora uses the same value by default. That setting can improve the distribution of requests, but it also contributes to the memory consumed by the hashing structures.

Statistics guided the memory reduction

Cloudflare's optimization treated the number of hashes as a balance between workload uniformity and memory use. Rather than assuming that the default was necessary in every case, the team used statistical analysis to understand how the distribution changes as more hashes are added.

The result was a reduction in the memory footprint of the Pingora-based service while preserving the consistent-hashing approach used for request routing. Cloudflare presents the work as an example of how mathematical analysis and implementation changes in Rust can uncover large savings from a system that already appeared to be using a conventional algorithm.

The company said the recovered capacity matters because Cloudflare's infrastructure, despite its scale, remains finite. The episode also illustrates why resource work at hyperscale can produce outsized results: a modest per-process improvement, repeated across a large fleet, can become tens of terabytes of reclaimed memory.

Sources

• Cloudflare BlogSaving another 100TB of RAM with math (and Rust)