Android Bench 2.0 Launches With Long-Horizon Tasks and Agentic Evaluation for AI Coding Models

The landscape of AI-assisted software development is shifting rapidly, moving away from simple bug fixes and minor script adjustments toward multi-day engineering projects. Reflecting this evolution, the development team behind Android Bench has announced the release of Android Bench 2.0, a major upgrade designed to evaluate large language models (LLMs) and coding agents against the complex, highly ambiguous, and multi-step problem-solving challenges that developers face on a daily basis.

When Android Bench was originally launched, it established a rigorous foundation for evaluating how well LLMs assist developers with real-world Android tasks. However, as AI models and underlying agent architectures have evolved, the evaluation methodology has had to keep pace. The framework has previously aligned with the Harbor framework, and the rollout of version 2.0 introduces the first comprehensive set of long-horizon tasks, alongside native agentic evaluation starting with models paired with their corresponding provider agents.

From Incremental Fixes to Long-Horizon Tasks

The first generation of Android Bench, much like many early artificial intelligence coding benchmarks, focused heavily on incremental changes to existing code repositories. These tasks were typically limited in scope, often centering on isolated bug fixes or smaller feature requests. While valuable at the time, that focus was largely a reflection of the limitations of AI assistance capabilities during that era, as well as how developers were actively utilizing those tools in their daily workflows.

To better serve the developer community and help identify which models and coding agents are genuinely suited for production-grade development workflows, the evaluation bar has been raised significantly. Android Bench 2.0 mirrors the ambitious, real-world challenges that engineers routinely delegate to AI systems. The newly introduced long-horizon tasks encompass substantial engineering efforts, including upgrading major dependencies, engineering brand-new features from the ground up, building complete applications from scratch, or executing complex conversions of cross-platform applications into native Android codebases.

Complex Tasks Require a More Nuanced Evaluation and Scoring

Evaluating multi-day engineering tasks requires a fundamental departure from traditional evaluation metrics. In long-form software engineering, a simple binary pass-or-fail grading system fails to capture the full picture of an AI model’s actual utility and architectural prowess.

Consider a scenario where an artificial intelligence coding agent successfully refactors 40 distinct user interface screens to Jetpack Compose, correctly establishes complex local database tables, and satisfies 90 percent of the project requirements, but ultimately fails a single edge-case unit test assertion. Under a rigid binary scoring model, this entire run would be graded as zero percent, completely obscuring the model’s sophisticated architectural capabilities and practical helpfulness.

To address this limitation, Android Bench 2.0 transitions to continuous scoring. This approach provides a much more meaningful signal, offering clearer data for model developers working to improve their systems and giving engineers a realistic understanding of how AI can assist them in production environments.

This completion rate is calculated through a multifaceted combination of evaluation factors, including core functional correctness, visual fidelity of the user interface, and the critical requirement of avoiding unintended regressions. Furthermore, the scoring system applies objective penalties for any deviations from specific evaluation instructions or structural project constraints. Developers can explore the updated leaderboard and click into individual model card views to review granular performance metrics, including pass rates, completion rates, and average financial and computational costs per model and per task.

Unsurprisingly, the introduction of these demanding scenarios has yielded significantly different performance numbers compared to early benchmarks. The highest pass rate achieved for long-horizon tasks currently sits at approximately 28 percent, a sharp contrast to the roughly 91 percent pass rate recorded for the original, simpler benchmark tasks.

Long-Horizon Tasks Uncover Helpful Insights for AI Assistance

Beyond simply measuring how effectively artificial intelligence handles extended, multi-step engineering assignments, the new long-horizon task dataset has yielded valuable insights into the specific operational strengths and structural weaknesses of the tested models. These findings allow the benchmark team to offer more practical, data-driven guidance to the engineering community.

Across various model tiers, empirical data shows that contemporary AI models perform significantly better when writing brand-new code compared to refactoring legacy codebases. Refactoring and code migrations present unique difficulties because success relies heavily on deep architectural complexity rather than sheer volume of code output.

The evaluations reveal that models exhibit robust capabilities when executing well-established, deterministic code transformations. Examples include converting legacy Java codebases to Kotlin, swapping networking libraries such as Retrofit for Ktor, or introducing a structured ViewModel layer. Models apply these architectural patterns consistently and reliably, even when operating across codebases exceeding 125 files and 8,000 lines of code.

Conversely, models consistently struggle when tasks require runtime validation—such as debugging missing dependency injection graphs—or when they encounter breaking framework changes and knowledge gaps involving unreleased or newly updated libraries. Similarly, porting complex cross-platform applications to native Android remains a formidable open challenge. No tested model achieves a 100 percent pass rate in this category, and frontier models max out at an 80 percent completion rate.

Introducing Agent Evaluations

To provide a more accurate reflection of how these models perform when fully integrated into modern agentic development workflows, Android Bench 2.0 incorporates commonly used coding agents directly into the evaluation pipeline. The initial rollout features new models evaluated against long-horizon tasks utilizing agents provided directly by the corresponding model creators. For instance, evaluations paired models like GPT 5.6 Sol on Codex and Gemini 3.8 Flash on Google Antigravity.

This pairing demonstrates how thoughtful harness design positively impacts developer outcomes. Features such as prompt caching and compact tool windowing have been shown to yield significant reductions in token consumption. The benchmark team plans to expand this testing methodology in the future by publishing results across a diverse matrix of model and agent combinations, helping development teams discover which specific configurations align best with their internal workflows.

New Models Added to the Leaderboard

To ensure that developers have access to the most up-to-date and comprehensive data for their technical decision-making, several prominent new models have been added to the public leaderboard. The latest additions include Gemini 3.8 Flash, Gemini 3.7 Flash, OpenAI’s GPT-6, Anthropic’s Fable 5.1, Kimi K3, and Qwen 3.8 Max. Leading the pack in the initial long-horizon evaluations is OpenAI’s GPT-6 Astra, which achieved the highest pass rate at 28 percent.

Looking Ahead

Android Bench 2.0 establishes a robust, modern environment for measuring the practical viability of artificial intelligence in Android software development. By integrating long-horizon tasks, multimodal evaluation criteria, native agent support, and continuous scoring, the project aims to empower AI research teams to build more capable and dependable coding partners while offering developers complete transparency regarding their technological options.

The updated leaderboard and revised methodology documentation are now available online. The development team emphasizes that community feedback directly influences the evolution of Android Bench, and developers are encouraged to share their insights, bug reports, and suggestions via GitHub as well as through official social channels on X and LinkedIn.