Android Bench 2.0 Launches with Long-Horizon Tasks and Agentic Evaluations for AI Coding Models

The landscape of AI-assisted software development is shifting rapidly, moving away from simple bug fixes toward complex, multi-day engineering workflows. Addressing this evolution, the development team behind Android Bench has announced the release of Android Bench 2.0, a major upgrade designed to measure how large language models and coding agents handle the scale, ambiguity, and multi-step problem-solving typical of real-world Android development.

When Android Bench was first launched, it established a rigorous foundation for evaluating how large language models assist developers with everyday Android tasks. As AI models and agents have evolved, the evaluation methodology has kept pace, aligning with the Harbor framework. Today, the benchmark is taking a massive leap forward by releasing its first set of long-horizon tasks, representing complex assignments that typically take a human engineer multiple days or even a week to complete. Alongside these challenging new workloads, the platform is introducing agentic evaluation, starting with native agents provided by corresponding model creators.

The updated leaderboard highlights a new generation of frontier systems. Leading the pack in the latest evaluations is OpenAI’s GPT-6 Astra, which achieved a peak pass rate of approximately 28% on the newly introduced long-horizon tasks. Other newly added models making their debut on the updated benchmark include Gemini 3.8 Flash, Gemini 3.7 Flash, OpenAI’s GPT-6, Anthropic’s Fable 5.1, Kimi K3, and Qwen 3.8 Max.

From Incremental Fixes to Long-Horizon Tasks

The initial iteration of Android Bench, much like early AI coding benchmarks across the broader tech industry, focused primarily on incremental changes to existing code repositories. In many cases, those early evaluations were limited to straightforward bug fixes or smaller feature requests. That design was an accurate reflection of what AI assistance was capable of at the time, as well as how developers were utilizing it in their daily routines.

However, as developers increasingly delegate broader and more ambitious responsibilities to AI, testing frameworks must adapt to match that reality. To help developers identify the models and coding agents best suited to their actual workflows, the creators of Android Bench have significantly raised the bar. Android Bench 2.0 mirrors these ambitious engineering challenges through its new long-horizon tasks dataset. These tasks include upgrading major dependencies, implementing entirely new features from the ground up, building complex applications from scratch, or converting an existing cross-platform application natively to Android.

By testing models against these demanding scenarios, the benchmark provides a much clearer picture of how artificial intelligence performs when faced with the messy, interconnected realities of production-grade mobile software architecture.

Complex Tasks Require a More Nuanced Evaluation and Scoring

Evaluating multi-day engineering tasks requires a departure from traditional evaluation methods. In complex software development, a simple binary pass-or-fail grading system fails to capture the full picture of an AI model’s true capability.

Consider a scenario where an autonomous coding agent successfully refactors 40 distinct user interface screens to Jetpack Compose, establishes complex database tables, and fulfills 90% of the core project requirements, but ultimately stumbles on a single edge-case assertion. Under a traditional binary scoring mechanism, that entire multi-hour, highly intricate run would be rated as zero percent, completely obscuring the model’s impressive architectural and structural capabilities. To remedy this, Android Bench 2.0 is transitioning to continuous scoring. This approach provides a much more meaningful signal, offering clearer insights for model developers and helping human engineers understand precisely how and where AI can assist them.

The completion rate under this new continuous system is calculated through a combination of vital factors, including core functionality, visual fidelity, and the successful avoidance of regressions. Furthermore, the evaluation framework applies objective scoring penalties for any deviations from specific evaluation instructions or structural constraints. Developers exploring the updated leaderboard can click into individual model card views to examine granular elements such as pass rates, completion rates, and average financial or computational costs calculated per model and per task.

Reflecting the immense difficulty of these new challenges, the highest pass rate recorded for long-horizon tasks currently sits at around 28%. This stands in stark contrast to the roughly 91% pass rates historically achieved on the original, simpler tasks featured in the benchmark.

Long-Horizon Tasks Uncover Helpful Insights for AI Assistance

Beyond simply measuring how well artificial intelligence handles long-running engineering chores, the new long-horizon task dataset yields valuable insights into the specific strengths and weaknesses of the tested models, offering practical guidance for the developer community.

Across various model tiers, current AI systems demonstrate a distinct proficiency in writing new code rather than refactoring or modifying existing codebases. Refactors and migrations prove significantly trickier because success relies heavily on understanding and navigating architectural complexity rather than simply generating a high volume of code.

When dealing with well-established, deterministic transformations, models show strong capabilities. Tasks such as converting legacy Java code to Kotlin, swapping a Retrofit networking layer for Ktor, or introducing a robust ViewModel layer are handled reliably. Models can apply these established patterns consistently, even when the work spans across more than 125 files and exceeds 8,000 lines of code.

Conversely, models consistently struggle when tasks require runtime validation—such as resolving missing dependency injection graphs—or when they involve breaking framework changes or dealing with knowledge gaps regarding unreleased libraries. Additionally, porting cross-platform applications natively to Android remains a significant hurdle. No model currently achieves a 100% pass rate on migration tasks, and even the most advanced frontier models cap out at roughly an 80% completion rate.

Introducing Agent Evaluations

To provide a more realistic assessment of how models perform when integrated directly into everyday agentic development workflows, Android Bench 2.0 is incorporating commonly used coding agents into its evaluation pipeline. The project is initiating this shift by running new models against long-horizon tasks using native agents provided by the corresponding model creators. For example, evaluations paired GPT 5.6 Sol with Codex, and Gemini 3.8 Flash with Google Antigravity.

This pairing demonstrates how intelligent harness design positively impacts developer outcomes. Observations from these tests indicate that architectural features like prompt caching and compact tool windowing can lead to significant reductions in token usage. The benchmark maintainers plan to expand this approach in the future by highlighting performance results across a wide variety of cross-vendor model and agent combinations, helping development teams discover which setups work best for their specific needs. Investing in this type of measurement ensures that developers retain the flexibility to use their preferred agent and model choices for Android development.

Looking Ahead

Android Bench 2.0 delivers a robust, modern environment for measuring the utility of artificial intelligence in Android software development. By combining long-horizon tasks, multimodal evaluation, agentic workflows, and continuous scoring, the platform aims to empower AI research teams to build more capable and dependable coding partners while offering human developers complete transparency into their AI options.

Developers and researchers can explore the updated leaderboard alongside the revised methodology documentation online. Because community input directly influences how the benchmark continues to evolve, the project maintainers encourage engineers to share feedback and contributions via GitHub, as well as official channels on X and LinkedIn.