Loading dataset details…
harbor run -d android-bench/android-benchharbor run -d android-bench/android-benchA dataset of 100 Android software engineering tasks derived from real-world repositories.
harbor run -d android-bench/android-benchA dataset of 100 Android software engineering tasks derived from real-world repositories.
harbor run -d android-bench/android-benchAndroid Bench is a benchmark dataset consisting of 100 tasks representative of day-to-day Android software engineering. Sourced from open-source repositories with >= 500 stars and expert contributors, it is used to evaluate agent and model competency on mobile codebases.
Running evaluations locally requires nested virtualization (KVM) to run Android emulators.
| Resource | Minimum requirement |
|---|---|
| Free disk space | 120 GB (for task images) |
| System RAM | 32 GB |
| CPU cores | 8 cores |
| Architecture | x86_64 with KVM |
The dataset contains 88 tasks sourced from GitHub pull requests and 12 expert-authored tasks designed to cover underrepresented areas.
The dataset exercises various Android framework APIs, build systems, and libraries:
| Dimension | Distribution / Coverage |
|---|---|
| Programming languages | 71% Kotlin, 25% Java |
| UI frameworks | 41% Jetpack Compose, 59% View-based layouts |
| Project targets | 42% applications, 58% libraries |
| Multimodal assets | 20 tasks include screenshots of the UI issues |
| Platform APIs | Jetpack Compose, Room, Coroutines, Flows, Hilt, Gradle, SDK upgrades |
Task complexity is balanced across different pull request patch sizes:
| Change size | Percentage | Range |
|---|---|---|
| Small | 46% | < 27 lines of code (LOC) |
| Medium | 33% | 27 - 136 LOC |
| Large | 21% | > 136 LOC (max 435 LOC) |
The median task size is 32 changed lines.
All task files include the standard BIG-BENCH canary string to prevent inclusion in LLM training corpora. Success trajectories are manually audited to ensure legitimate fixes rather than reward hacking.
Download the dataset:
harbor download android-bench/android-benchRun a task evaluation:
# Run with an agent
harbor run -d android-bench/android-bench -i "*airbnb__lottie-android-pr_2427" -a <agent_name>
# Or verify with the oracle baseline
harbor run -d android-bench/android-bench -i "*airbnb__lottie-android-pr_2427" -a oracleAndroid Bench is a benchmark dataset consisting of 100 tasks representative of day-to-day Android software engineering. Sourced from open-source repositories with >= 500 stars and expert contributors, it is used to evaluate agent and model competency on mobile codebases.
Running evaluations locally requires nested virtualization (KVM) to run Android emulators.
| Resource | Minimum requirement |
|---|---|
| Free disk space | 120 GB (for task images) |
| System RAM | 32 GB |
| CPU cores | 8 cores |
| Architecture | x86_64 with KVM |
The dataset contains 88 tasks sourced from GitHub pull requests and 12 expert-authored tasks designed to cover underrepresented areas.
The dataset exercises various Android framework APIs, build systems, and libraries:
| Dimension | Distribution / Coverage |
|---|---|
| Programming languages | 71% Kotlin, 25% Java |
| UI frameworks | 41% Jetpack Compose, 59% View-based layouts |
| Project targets | 42% applications, 58% libraries |
| Multimodal assets | 20 tasks include screenshots of the UI issues |
| Platform APIs | Jetpack Compose, Room, Coroutines, Flows, Hilt, Gradle, SDK upgrades |
Task complexity is balanced across different pull request patch sizes:
| Change size | Percentage | Range |
|---|---|---|
| Small | 46% | < 27 lines of code (LOC) |
| Medium | 33% | 27 - 136 LOC |
| Large | 21% | > 136 LOC (max 435 LOC) |
The median task size is 32 changed lines.
All task files include the standard BIG-BENCH canary string to prevent inclusion in LLM training corpora. Success trajectories are manually audited to ensure legitimate fixes rather than reward hacking.
Download the dataset:
harbor download android-bench/android-benchRun a task evaluation:
# Run with an agent
harbor run -d android-bench/android-bench -i "*airbnb__lottie-android-pr_2427" -a <agent_name>
# Or verify with the oracle baseline
harbor run -d android-bench/android-bench -i "*airbnb__lottie-android-pr_2427" -a oracle