Cloudflare Adds SQL Transformers to Logpush Pipelines
Cloudflare has announced general availability for Transformers on Logpush, letting observability teams filter, redact, and reshape log streams using custom SQL queries before data reaches downstream destinations.
Radwan AbdulhadiEditor and technology writer
Streamlining Log Export Pipelines
Cloudflare has introduced support for Transformers within its log delivery workflow, allowing teams to manipulate log streams directly at the source. Observability teams frequently export Cloudflare logs to security information and event management systems, analytics platforms, and object storage. Before those logs reach their final destination, teams often need to filter out health checks, remove sensitive fields, or reshape records to match internal dashboards and investigation schemas.
Traditionally, achieving this required maintaining a separate processing service to ingest, modify, and scale incoming log data. Furthermore, organizations typically pay to ingest and store records that are immediately discarded upon arrival. By integrating processing directly into the export workflow, teams can reduce downstream data volume and cut unnecessary storage costs.
SQL-Driven Filtering and Redaction
With the new capability, customers define a SQL query tailored to a specific dataset to filter records, redact sensitive information, and alter output structures. Logpush applies these queries prior to export, ensuring that every destination receives data formatted precisely to its requirements. Records failing to match the criteria in a WHERE clause are systematically omitted, while SELECT statements dictate the final structure of exported records in NDJSON format.
Customers can build, preview, and manage these transformations directly inside the dashboard using Transformer Studio or programmatically via the API. Each modification automatically generates a new version, allowing engineering teams to track configuration changes over time.
In-Process Execution with Apache DataFusion
To achieve low latency and high throughput, Cloudflare evaluated remote compute options before deciding to execute transformations in-process within its delivery service, Bufstream. Initial prototypes involving external compute introduced network dependencies that risked timeout issues and variable delivery latencies. By embedding an execution engine directly into the pipeline, high-volume jobs can process millions of logs per second using existing scheduling and batching infrastructure.
Cloudflare selected Apache DataFusion, a Rust query engine built on Apache Arrow, to power the underlying execution. In internal benchmarks, processing synthetic batch data demonstrated high throughput capabilities without disrupting standard delivery mechanisms.
Broad Observability Updates
The release of Transformers coincides with broader initiatives by Cloudflare to expand observability tooling and democratize access to advanced features. These enhancements include multiple improvements designed to simplify data collection, monitoring, and analysis across infrastructure layers without requiring complex architectural workarounds.
By making advanced delivery features broadly available, Cloudflare continues to align its infrastructure capabilities with customer demands for streamlined log management, tighter security controls, and predictable operational expenses.
Sources
- Cloudflare BlogFilter, reshape, and redact logs before export with Transformers
Continue chronologically
Related entity coverage
Related stories
Cloudflare cuts hardware pressure as data volumes grow
Cloudflare says schema changes, compression and pipeline redesigns have reduced hardware pressure as its data delivery reaches two billion events per second.
Cloudflare Launches Eight Major Observability Updates
Cloudflare has introduced eight major updates combining logs, traces, analytics, alerts, dashboards, querying, and telemetry export into one platform.
Cloudflare Launches Traces in Open Beta for End-to-End Request Visibility
Cloudflare has introduced Cloudflare Traces in open beta, extending automated request tracing across security rules, caches, Workers, and origin infrastructure.
Cloudflare Expands Access to Enterprise Features for All
Cloudflare provides an update on its pledge to eliminate product access tiers, expanding Logpush, RBAC, and multi-account features to all plans.
