Cloudflare Cuts AI Code Review Inference Costs by 68%
After noticing a sharp rise in per-run costs for its internal AI code reviewer, Cloudflare deployed aggressive optimization strategies involving prompt caching, model routing, and code rewrites.
Radwan AbdulhadiEditor and technology writer
Investigating Rising AI Code Review Costs
Cloudflare recently investigated a significant increase in the operational expenses of its automated development workflows. When the company previously blogged about our AI code reviewer in April, an average run cost a modest $1.19. By the end of August, however, that figure had climbed to approximately $2.46. Despite maintaining the same architectural foundation and risk-tier structure, token usage and financial overhead had more than doubled.
Midway through September, engineering teams were tasked with finding where those tokens were being consumed and cutting them aggressively. By balancing prompt engineering, model selection, and effort adjustments, the average review cost was successfully reduced to just $0.77. This marked a 68% decrease from the peak two-week average before optimization efforts began. Additionally, efficiency gains reduced the median review duration from 4 minutes and 9 seconds down to 1 minute and 32 seconds.
Model Transitions and Token Growth
A notable factor behind the cost escalation over the summer was the strategic shift toward more capable frontier models. At the start of August, Cloudflare moved every reviewer onto Anthropic models because evaluation metrics showed better performance on the specific workload. However, this transition raised the average cost per job from $1.38 to $2.03.
For instance, the internal standards reviewer, Codex, was transitioned to Claude Opus 4.7 due to enhanced performance and reliability considerations. That specific adjustment pushed its review cost from about $0.25 up to $0.88 in August, temporarily making it more expensive than the coordinator agent itself. Meanwhile, cache hit rates rose from 85.7% in April to 97% in September, but the overall volume of tokens per review more than tripled from 0.91 million to 2.99 million.
Unlocking Efficiency Through Prompt Caching
Before optimization, session expenses were heavily skewed toward context preservation. Output tokens accounted for roughly 23% of spend, cache writes took 43%, and cache reads made up 34%. Examining individual sessions revealed that the code quality reviewer made an average of 16.6 large language model calls per review, writing 72,000 tokens to the cache and reading back 1.08 million tokens because each sequential call sent the entire conversation history.
Understanding how Anthropic's prompt cache functions proved critical to resolving the overhead. The caching mechanism leverages prefixes to store and reuse conversation segments computed by the large language model. Because harness tools, system prompts, and opening messages rarely changed across reviews, they presented ideal opportunities for prefix caching, provided cache breakpoints were configured correctly.
Fixing Cross-Repository Cache Misses
Data gathered beginning September 14 revealed that initial requests in a session found a cached prefix only 14% to 26% of the time, despite sharing identical tool definitions and agent prompts across repositories. The root cause lay in OpenCode v1, which built a single system prompt block combining the agent prompt with environment details like the working directory and date, placing a cache breakpoint only at the very end. Consequently, cached instructions from one repository could never be reused by another.
To fix this, engineers added a marker to the agent prompt and implemented a plugin hook to split the block in two on September 16. By isolating shared bytes into their own breakpoint and removing repository-specific variables from the beginning of the prompt, cross-repository cache reuse became possible. Simultaneously, moving Codex's reference index directly into its prompt reduced its LLM calls per run from 15.9 to 9.6, dropping its per-run cost from $1.12 to $0.72.
Migrating to OpenCode v2
Another major architectural update involved transitioning the reviewer workload to OpenCode v2, which represents a complete ground-up rewrite from the previous version. Before deploying the upgrade globally, the team ran shadow testing across canary repositories to validate consistency. Because both versions matched on approval decisions in 98% of tested merge requests, Cloudflare migrated all reviews to v2 in early October.
Comparative analysis showed that full-tier reviews executed on v2 finished about 13% faster than on v1 under identical model conditions. Furthermore, the engineering group continues experimenting with alternative execution models, including running internal reviewers solely on Durable Objects using pi-durable to explore further performance enhancements.
Sources
- Cloudflare BlogCutting our inference bill by 68% by being obsessive about optimization
Continue chronologically
Related entity coverage
Related stories
Hackers Hit OpenCode Console, Stole API Keys and Ran Up Charges
Hackers breached an older OpenCode platform system, compromising user API keys and running up massive unauthorized AI service charges.
Cybersecurity Roundup: Anthropic Breach Dispute and Server Flaw
A weekly cybersecurity roundup details an Anthropic breach dispute, a critical 16-year-old UEFI server flaw, and an AI data leak at Cloudflare.
OpenCode Hack Exposes API Keys in AI Distillation Attacks
The open-source AI coding project OpenCode suffers a security breach exposing API keys to a large-scale AI distillation attack.
Cloudflare Details Agentic Security Operations Harness
Cloudflare Managed Defense introduces an AI agent harness combining Workers, deterministic evidence collection, and global telemetry for threat analysis.
