A benchmark built for more demanding work

Google says the first version of Android Bench was designed to provide a rigorous way to evaluate how large language models assist with real-world Android development. Like other early AI coding benchmarks, it concentrated largely on incremental changes to existing repositories, including bug fixes and smaller feature requests.

Android Bench 2.0 raises the difficulty to better match the work developers may delegate to AI systems. Its first set of long-horizon tasks, or LHTs, is intended to reflect assignments that can take an engineer several days or even a week to complete. The tasks include upgrading dependencies, adding new features, building applications from scratch and converting cross-platform applications to Android.

Google describes the release as an update designed to test AI models and agents against the scale, ambiguity and multi-step problem-solving demands of modern Android development. The company has also been updating its methodology, including aligning the benchmark framework with the Harbor framework.

Continuous scoring replaces a simple pass or fail

The company says binary grading is not sufficient for complex, multi-day engineering assignments. An agent could refactor dozens of screens to Jetpack Compose, create database tables and satisfy most requirements, yet fail one edge-case assertion. Under a binary system, that result could be recorded as a complete failure even when the implementation demonstrates substantial architectural capability.

Android Bench 2.0 therefore uses continuous scoring to provide a more detailed measure of task completion. Google says its completion rate combines factors including functionality, visual fidelity and the avoidance of regressions. The evaluation also applies objective penalties when a submission deviates from instructions or violates structural constraints.

The updated leaderboard provides additional information through each model’s card view, including pass rate, completion rate and average costs per model and task. Google reports that the highest pass rate for the new long-horizon tasks is about 28%, compared with roughly 91% for the original benchmark tasks.

Refactoring remains harder than writing new code

Google’s early findings from the LHT dataset point to a difference between generating new code and changing existing systems. Across model tiers, the tested AI systems performed better when writing new code than when refactoring existing code. The company says refactors and migrations become more difficult as architectural complexity increases, rather than simply as the amount of code grows.

The models showed stronger results on established and deterministic transformations. Examples cited by Google include converting Java to Kotlin, replacing Retrofit with Ktor and introducing a ViewModel layer. According to the company, models can apply these patterns consistently across more than 125 files and over 8,000 lines of code.

The results were less reliable when tasks required runtime validation, such as identifying missing dependency-injection graphs, or involved breaking framework changes. Google also identified knowledge gaps around unreleased libraries as a source of difficulty.

Cross-platform ports remain an open challenge

Porting a cross-platform application to Android continues to be one of the more difficult areas in the benchmark. Google says no tested model achieved a 100% pass rate on these tasks, while frontier models reached a completion rate of no more than 80%.

The distinction between pass rate and completion rate is important in the updated evaluation. Completion scoring can show how much of a complex assignment an agent accomplished, while pass rate reflects whether the run met the benchmark’s full requirements. The two measures are intended to give developers and AI research teams a more nuanced view of performance.

Testing models together with their agents

Android Bench 2.0 also introduces agentic evaluation, beginning with agents associated with the corresponding model providers. Google says this approach is intended to show how models perform when they are integrated into the agentic workflows developers may actually use.

For its initial comparisons, the company ran GPT 5.6 Sol on Codex and Gemini 3.8 Flash on Google Antigravity. Google says the pairing demonstrates how harness design can influence outcomes, citing prompt caching and compact tool windows as features that can reduce token usage.

The company plans to expand the evaluation to cover more combinations of models and agents. The stated goal is to help developers and teams identify which pairings work best for their workflows, rather than assessing models in isolation.

An expanding leaderboard

Google has added several models to the Android Bench leaderboard, including Gemini 3.8 Flash, Gemini 3.7 Flash, OpenAI’s GPT-6, Anthropic’s Fable 5.1, Kimi K3 and Qwen 3.8 Max. The company reports that OpenAI’s GPT-6 Astra currently leads the listed long-horizon results with a 28% pass rate.

The updated benchmark combines long-horizon tasks, multimodal evaluation, agent testing and continuous scoring. Google says the broader framework is intended to help AI research teams build more capable and dependable coding partners while giving Android developers greater transparency when assessing their options.

The company is inviting feedback through GitHub and its social channels as it continues to update the methodology and leaderboard.

Sources

• Android DevelopersAndroid Bench 2.0: Pushing the frontier with challenging long-horizon tasks