Cloudflare cuts hardware pressure as data volumes grow
Cloudflare has detailed a series of data-pipeline changes designed to keep hardware growth below the pace of traffic growth. The company says the work has deferred the equivalent of two and a half years of hardware expansion.
Radwan AbdulhadiEditor and technology writer
Keeping hardware growth below traffic growth
Cloudflare says it has reduced pressure on its infrastructure by repeatedly redesigning the systems that move data from its global network to centralized data centers. The company’s data team was delivering roughly two billion events per second in May 2026, including customer traffic and data generated by internal teams.
That pipeline supports products and services including Cloudflare Logs, the SQL API, observability and internal analytics. Cloudflare says the central challenge is persistent exponential traffic growth against more linear hardware growth. Over the past five years, changes to the pipeline have produced savings equivalent to deferring two and a half years of hardware expansion, according to the company.
Replacing the monolithic schema
The first major change came in 2022, when Cloudflare dismantled a single shared data schema. That earlier design put fields from multiple products into one stream, requiring internal consumers to ingest the full payload and then filter out records or fields they did not need.
Cloudflare replaced the firehose with product-specific schemas containing the fields required by each service. Although some fields were duplicated across messages, the narrower streams reduced backbone bandwidth usage by 35% to 40%, the company says. The reduction was enough to offset approximately a year of compressed traffic growth at the time.
The pipeline begins with Logfwdr, an internal service that collects data from highly distributed sources, including Workers and other Cloudflare systems. It sends the data to centralized receivers, which place it into Buftee, the company’s buffering layer. Consumers then fetch the buffered data and deliver it to final destinations such as ClickHouse.
Trading compression work for network capacity
As traffic continued to rise, Cloudflare says network congestion became a daily concern by 2024. The company responded by changing its compression algorithm from LZ4 to zstd, despite earlier experiments that had shown higher CPU usage with zstd.
Cloudflare first deployed the change to a small number of servers and monitored CPU metrics before expanding it across the fleet. The company says the rollout produced little observable CPU impact while reducing compressed traffic throughput by nearly 40%.
Because the amount of uncompressed data continued to grow at the same rate, the compression improvement did not reduce the underlying workload. It did, however, allow Cloudflare to support more than 1.5 times as much traffic through a pipe of the same size, according to the company. Cloudflare estimates that this improvement represented about 10 months of equivalent growth reduction.
Reducing duplicate transit in 2026
Cloudflare’s next major target was uncompressed bytes. The company historically sent two complete copies of many non-regional analytics events to separate centralized regions so that ClickHouse clusters could continue serving queries if one region failed. Data governed by Customer Metadata Boundaries remained within its designated region.
That approach became increasingly expensive as disaster resilience expanded across more analytics streams. Logfwdr traffic effectively doubled across the board because every event traveled over wide-area transit twice. Cloudflare says the model was not sustainable as traffic continued its exponential growth.
The company adapted an approach already used for Logpush, where centralized data centers operate as a shared receiver pool. Each message is sent to one available region, with automatic failover if a machine or facility cannot be reached. Analytics required an additional design because both regional ClickHouse clusters still need a complete copy of non-regional data for independent query failover.
Replication after sampling
Cloudflare’s redesigned analytics pipeline separates transit from database redundancy. Logfwdr sends each event only once over wide-area transit to whichever centralized receiver is available. The data is then placed in Buftee, while remote inserters create cross-data-center copies farther downstream.
The timing of replication is central to the change. Sampling takes place before the downstream copy is made, so the second region receives an already-compacted stream rather than a duplicate of every raw event. Cloudflare describes sampling as a way to reduce high-volume traffic while preserving rare events.
For the datasets migrated to the new architecture, Cloudflare says the cutover removed approximately 43 GB/s of compressed transit traffic while adding about 7 GB/s of cross-data-center replication. The company attributes the difference to replication after sampling and to more effective compression once data is centralized and partitioned uniformly.
A continuing pipeline redesign
Cloudflare’s 2026 work also includes decentralizing sampling during the summer and introducing aggregations into the pipeline in the fall. These efforts are part of the same broader objective: increase the amount of data the existing infrastructure can process without matching every rise in traffic with new hardware.
The company presents the changes as a sequence rather than a single optimization. Schema design reduced unnecessary fields, compression reduced the size of transmitted data, and replication changes reduced duplicated wide-area traffic. Together, Cloudflare says, those measures have helped its infrastructure absorb rapidly growing data volumes while keeping resource costs under control.
Sources
- Cloudflare BlogRewinding the exponential: How we keep hardware usage down
Continue chronologically
Related entity coverage
Related stories
Cloudflare Adds SQL Transformers to Logpush Pipelines
Cloudflare introduces Transformers for Logpush, enabling teams to filter, redact, and reshape log streams using SQL before export.
Cloudflare Expands Access to Enterprise Features for All
Cloudflare provides an update on its pledge to eliminate product access tiers, expanding Logpush, RBAC, and multi-account features to all plans.
Cloudflare Updates AI Gateway With User Insights
Cloudflare has updated its AI Gateway tool with new User Insights features, helping teams analyze model overuse, task categories, and conversation turns.
Cloudflare Adds Post-Quantum Visibility for Domains
Cloudflare introduces new domain-level visibility tools for post-quantum encryption in TLS 1.3 across HTTP Analytics, Log Explorer, and Logpush.
