Android Bench 2.0 Released with Long-Horizon Tasks and Agentic Evaluation for AI Coding Models

The ecosystem for evaluating artificial intelligence in mobile development has taken a significant leap forward with the launch of Android Bench 2.0. When the benchmark framework was first introduced, it provided a rigorous foundation for measuring how large language models could assist developers with real-world Android tasks. However, as AI models and coding agents have rapidly evolved from simple assistants into complex problem-solvers, the methodology required a fundamental modernization. By aligning with the Harbor framework and expanding its scope to include complex, multi-day engineering challenges, Android Bench 2.0 aims to better reflect the scale, ambiguity, and multi-step reasoning that mobile developers encounter on a daily basis.

At the heart of this major upgrade is the introduction of long-horizon tasks, commonly referred to as LHTs. While early AI coding benchmarks were largely limited to incremental bug fixes or minor feature requests in existing repositories, Android Bench 2.0 mirrors the ambitious, high-complexity work that engineers routinely delegate to AI. These long-horizon tasks encompass demanding scenarios such as upgrading core dependencies, building comprehensive mobile applications completely from scratch, introducing major new features, or porting intricate cross-platform applications natively to Android.

From Incremental Fixes to Long-Horizon Tasks

The shift from localized bug fixes to long-horizon tasks represents a broader maturation in how software engineers interact with AI assistants. Earlier iterations of testing reflected an era when AI capabilities were restricted to minor edits and localized adjustments. Today, as engineering teams increasingly rely on AI to handle broad architectural shifts, testing environments must adapt to measure these advanced capabilities accurately.

The new long-horizon tasks in Android Bench 2.0 capture this reality by presenting models with challenges that would typically take a human engineer multiple days or even a full week to complete. By testing how models perform across vast codebases involving dozens of files and thousands of lines of code, the benchmark provides a much clearer picture of how artificial intelligence fares when confronted with expansive, multi-layered software engineering projects.

Complex Tasks Require a More Nuanced Evaluation and Scoring

Evaluating multi-day engineering tasks requires moving beyond traditional binary pass or fail grading systems, which often fail to capture the full scope of an AI model’s performance. Under a binary scoring model, if an agent successfully refactors forty different screens to Jetpack Compose, establishes complex database tables, and satisfies ninety percent of the project requirements, a failure on a single edge-case assertion results in a zero-percent grade. This approach obscures the underlying architectural capabilities and genuine progress made by the model.

To address this limitation, Android Bench 2.0 introduces continuous scoring to provide a more meaningful signal for both model developers and software engineers seeking to understand AI utility. This completion rate is calculated through a combination of crucial factors, including functional correctness, visual fidelity, and the successful avoidance of regression issues. Furthermore, the evaluation framework applies objective scoring penalties for deviations from specific evaluation instructions or structural constraints.

Exploring these nuances reveals a stark drop in overall success rates when moving to more complex challenges. The highest pass rate achieved for long-horizon tasks currently sits at approximately 28 percent, a notable decrease compared to the roughly 91 percent pass rate typical of the original, simpler tasks featured in the initial benchmark framework. To help developers navigate these differences, the updated leaderboard features a detailed model card view where users can explore specific metrics, including pass rates, completion rates, and average costs per model and per task.

Long-Horizon Tasks Uncover Helpful Insights for AI Assistance

The introduction of the long-horizon task dataset has yielded valuable insights regarding the current strengths and limitations of tested AI models. Across various model tiers, artificial intelligence consistently demonstrates a stronger aptitude for writing new code rather than refactoring or migrating existing codebases. Refactoring projects prove significantly trickier because success relies heavily on architectural complexity and understanding existing system design rather than simply generating raw code volume.

When dealing with well-established, deterministic transformations, models show remarkably strong capabilities. Tasks such as converting legacy Java code to Kotlin, swapping out Retrofit for Ktor, or introducing a structured ViewModel layer are executed with high consistency, even when spread across more than 125 files and exceeding 8,000 lines of code.

Conversely, models encounter substantial difficulties when tasks demand runtime validation—such as resolving missing dependency injection graphs—or when they involve breaking framework changes and knowledge gaps regarding unreleased libraries. Porting cross-platform applications directly to Android remains a particularly stubborn challenge for the industry. No tested model achieves a 100 percent pass rate in this category, and frontier models max out at roughly an 80 percent completion rate.

Introducing Agent Evaluations

To provide a more realistic representation of how models perform when integrated directly into established developer workflows, Android Bench 2.0 incorporates commonly used coding agents into its evaluation process. The benchmark has begun running new models against long-horizon tasks using agents provided directly by the corresponding model creators. For instance, OpenAI’s GPT 5.6 Sol has been evaluated using Codex, while Gemini 3.8 Flash has been tested using Google Antigravity.

This pairing demonstrates how intelligent harness design positively influences developer outcomes. Observations from these evaluations indicate that effective implementation techniques, such as prompt caching and compact tool windowing, can lead to substantial reductions in token consumption. Moving forward, the project plans to expand these evaluations by highlighting results across a wide variety of model and agent combinations, helping development teams discover which specific configurations best suit their unique workflows.

New Models Added to the Leaderboard

To ensure that developers have access to the most current data when making technical decisions, Android Bench 2.0 has expanded its leaderboard with several prominent new additions. The updated rankings now feature Gemini 3.8 Flash, Gemini 3.7 Flash, OpenAI’s GPT-6, Anthropic’s Fable 5.1, Kimi K3, and Qwen 3.8 Max. Among these additions, OpenAI’s GPT-6 Astra currently occupies the top position on the leaderboard with a 28 percent pass rate on long-horizon tasks.

Looking Ahead

Android Bench 2.0 establishes a robust environment for measuring the true efficacy of artificial intelligence in Android development. By combining long-horizon tasks, multimodal evaluation, agentic workflows, and continuous scoring systems, the initiative seeks to empower AI research teams to build more capable and dependable coding partners while offering developers complete transparency regarding their available tools.

Developers and researchers can explore the updated leaderboard and review the refined methodology on the official Android developer portals. Community involvement remains a cornerstone of the project’s evolution, with feedback welcomed through the GitHub community dataset repository as well as official social channels on X and LinkedIn.