Meta Deploys AI JiTTests to Catch Bugs Before Production
Meta has shared production results and impact data for its large language model-powered Just-in-Time testing system, highlighting a human-on-the-loop approach that catches bugs before code reaches production.
Radwan AbdulhadiEditor and technology writer
Scaling AI-Powered Bug Detection
As software platforms expand to encompass hundreds of millions of lines of code, traditional testing methods struggle to keep pace with rapid development cycles. To address this challenge, Meta has advanced its infrastructure by deploying [Just-in-Time Tests (JiTTests)](https://engineering.fb.com/2025/09/30/security/llms-are-the-key-to-mutation-testing-and-better-compliance/) into production environments. Outlined in the research paper titled [Just-in-Time Catching Test Generation at Meta](https://arxiv.org/pdf/2601.22832), the novel methodology leverages large language models to identify and capture software bugs on the fly before code changes are integrated into production.
While many artificial intelligence tools review code similarly to human reviewers by analyzing faults based on historical data, Meta's approach differs significantly. Rather than mimicking a traditional code review process, the system generates executable tests designed to prove concrete behavioral differences introduced by a specific code change. This methodology builds upon previous foundations in [applying large language models (LLMs) to generate software tests](https://engineering.fb.com/2025/02/05/security/revolutionizing-software-testing-llm-powered-bug-catchers-meta-ach/), targeting complex bug categories that standard agent-based reviews might miss.
Dual Workflows and Risk Targeting
The testing architecture initiates overnight by reviewing code diffs submitted during the previous day. The system specifically targets changes categorized as risky using a mechanism akin to the [Diff Risk Score (DRS)](https://engineering.fb.com/2025/08/06/developer-tools/diff-risk-score-drs-ai-risk-aware-software-development-meta/) framework, which is trained on prior code modifications to isolate potential regressions.
Once high-risk changes are identified, the platform deploys two distinct tandem workflows to generate tests. The first is a dodgy diff workflow that treats every code change as if it were a bug, applying [mutation-guided LLM-based test generation](https://engineering.fb.com/2025/02/05/security/revolutionizing-software-testing-llm-powered-bug-catchers-meta-ach/) to highlight behavioral divergences. Concurrently, an intent-aware workflow analyzes the summary, title, and code modifications to deduce the author's intention, subsequently mapping out possible failure points and generating tests targeting those vulnerabilities.
Filtering False Positives
Operating at Meta's massive scale presents a distinct hurdle: separating strong catches that reveal genuine bugs from weak catches that represent false positives. To streamline this evaluation, the platform passes generated weak catches through an automated assessment stage that utilizes an LLM-as-Judge ensemble alongside a rule-based false-positive elimination environment known as RubFake.
The LLM-as-Judge ensemble evaluates execution traces, failure messages, and parent-child code diffs using multiple state-of-the-art language models. Because LLM assessments are non-deterministic, they are paired with RubFake's verified and deterministic rule sets. Together, these systems establish a reliable ranking of candidate catches to determine which issues require further validation.
Human-on-the-Loop Supervision
Rather than utilizing a traditional human-in-the-loop setup where engineers manually review every single output, Meta implemented a human-on-the-loop framework. Under this model, engineers establish aggregate review criteria and analyze system outcomes globally rather than supervising every generated test individually.
According to deployment data, 589 engineers—representing 64 percent of those reached out to—have engaged with the JiTTesting results. These participating engineers reported a 60 percent true-positive rate for actionable findings surfaced by the infrastructure.
Production Impact and Future Outlook
The integration of Catching JiTTests into active pipelines has yielded measurable improvements in software security and reliability. During the first half of 2026, the automated testing mechanism successfully caught 74 serious privacy bugs at the diff stage before they could land in production environments. Furthermore, Meta reported that the deployment contributed to a 20 percent reduction in overall privacy-related serious events during that same timeframe, demonstrating how automated safety layers can protect end users at scale.
Sources
- Meta EngineeringCatching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale
Continue chronologically
Related entity coverage
Related stories
Sapienz Receives ISSTA 2026 Impact Paper Award
Ke Mao, Mark Harman, and Yue Jia win the ISSTA 2026 Most Influential Paper Award for their foundational 2016 automated testing research.
Meta Introduces F3 Lightning to Scale Ads ML Features
Meta details F3 Lightning, a new infrastructure layer that accelerates ads machine learning features to production without increasing costs.
Meta’s Muse AI Agent Stirs Big Hopes But Low Business Impact
Meta’s new Muse AI agent has sparked intense corporate discussion, but major companies report minimal transaction volume and low early disruption.
Meta Introduces AI Fashion App Style Sketch for Virtual Try-Ons
Meta has quietly launched a new AI fashion application called Style Sketch in select Asian markets, allowing users to virtually try on clothing.
