Introduction to Cloudflare K2

Traditional Remote Procedure Call architectures often present core scaling challenges because producers and consumers must align in both scale and time. If a producer transmits more data than a consumer can process, or if downstream services become unavailable, events face the risk of being dropped entirely. This complexity multiplies when multiple independent consumers require the same event data for separate tasks.

To address these operational hurdles, Cloudflare has announced the public beta launch of Cloudflare K2. Detailed further in the official announcement on the Cloudflare Blog, K2 is a durable event streaming primitive built for the Developer Platform designed to decouple producers and consumers at the edge.

Architecture and R2 Integration

Under the hood, K2 implements a partitioned, durable log directly on top of R2 object storage. Object storage systems combine strongly consistent APIs with exceptionally high storage durability, reaching up to 11 9s of durability as outlined in the R2 reference documentation.

By offloading replication and consensus mechanics to the storage layer, the application layer of K2 remains radically simpler, cheaper, and higher in performance. This separation of compute and storage enables independent scaling, allowing organizations to retain vast quantities of historical event data cost-effectively.

Development Background and Edge Challenges

Cloudflare originally developed K2 to serve as a durable buffer on the edge, functioning initially as the ingestion layer for Basin Pipelines. Pipelines operates on a pull-based model via a stream processing engine that requires a separate system to durably store incoming events before they are read, transformed, and written down.

Because Pipelines operates across a vast global footprint spanning hundreds of cities, traditional distributed systems software like Apache Kafka cannot be easily run. K2 was engineered specifically to overcome these edge infrastructure constraints, leveraging Cloudflare's massive horizontal scaling capacity while utilizing R2 for state persistence.

Comparing K2 with Existing Primitives

Cloudflare provides several asynchronous delivery primitives, raising questions about when developers should choose K2 over options like Queues. Queues are specifically tailored around tracking individual items of expensive or time-consuming work that require asynchronous completion, supporting fine-grained logic such as retries, delays, and dead-letter queues.

In contrast, K2 targets high-scale data movement, fan-out consumption, and long-term retention. Messages in K2 are produced and consumed in batches, which drives higher producer latency compared to queues but enables efficient processing for large-scale event streams. Developers looking to implement data ingestion pipelines can also evaluate options like Basin Pipelines depending on their exact architectural destination.

Getting Started with Streams and Subscriptions

Creating and managing streams can be accomplished through Wrangler, the Cloudflare dashboard, the API, or command-line tooling. Developers can start by consulting the official setup guide to build their first stream in seconds.

Once a stream is active, data can be produced via an HTTP API or a Worker binding using arbitrary byte formats or encodings. Subscriptions then allow developers to divide work between multiple consumers, achieving read parallelism to manage heavier workloads effectively across a distributed set of workers.

Sources

• Cloudflare BlogAnnouncing Cloudflare K2: serverless event streams