Datasets
| Dataset | Tasks |
|---|---|
| Dataset | Tasks |
|---|---|
| Dataset | Tasks |
|---|---|
A dataset of 100 Android software engineering tasks derived from real-world repositories.
harbor run -d android-bench/android-benchAndroid Bench is a benchmark dataset consisting of 100 tasks representative of day-to-day Android software engineering. Sourced from open-source repositories with >= 500 stars and expert contributors, it is used to evaluate agent and model competency on mobile codebases.
Running evaluations locally requires nested virtualization (KVM) to run Android emulators.
| Resource | Minimum requirement |
|---|---|
| Free disk space | 120 GB (for task images) |
| System RAM | 16 GB |
| CPU cores | 8 cores |
| Architecture | x86_64 with KVM |
The dataset contains 88 tasks sourced from GitHub pull requests and 12 expert-authored tasks designed to cover underrepresented areas.
The dataset exercises various Android framework APIs, build systems, and libraries:
| Dimension | Distribution / Coverage |
|---|---|
| Programming languages | 71% Kotlin, 25% Java |
| UI frameworks | 41% Jetpack Compose, 59% View-based layouts |
| Project targets | 42% applications, 58% libraries |
| Multimodal assets | 20 tasks include screenshots of the UI issues |
| Platform APIs | Jetpack Compose, Room, Coroutines, Flows, Hilt, Gradle, SDK upgrades |
Task complexity is balanced across different pull request patch sizes:
| Change size | Percentage | Range |
|---|---|---|
| Small | 46% | < 27 lines of code (LOC) |
| Medium | 33% | 27 - 136 LOC |
| Large | 21% | > 136 LOC (max 435 LOC) |
The median task size is 32 changed lines.
All task files include the standard BIG-BENCH canary string to prevent inclusion in LLM training corpora. Success trajectories are manually audited to ensure legitimate fixes rather than reward hacking.
Run a single task evaluation locally:
run_task --model gemini/gemini-2.5-flash --task airbnb__lottie-android-pr_2427A dataset of 100 Android software engineering tasks derived from real-world repositories.
harbor run -d android-bench/android-benchAndroid Bench is a benchmark dataset consisting of 100 tasks representative of day-to-day Android software engineering. Sourced from open-source repositories with >= 500 stars and expert contributors, it is used to evaluate agent and model competency on mobile codebases.
Running evaluations locally requires nested virtualization (KVM) to run Android emulators.
| Resource | Minimum requirement |
|---|---|
| Free disk space | 120 GB (for task images) |
| System RAM | 16 GB |
| CPU cores | 8 cores |
| Architecture | x86_64 with KVM |
The dataset contains 88 tasks sourced from GitHub pull requests and 12 expert-authored tasks designed to cover underrepresented areas.
The dataset exercises various Android framework APIs, build systems, and libraries:
| Dimension | Distribution / Coverage |
|---|---|
| Programming languages | 71% Kotlin, 25% Java |
| UI frameworks | 41% Jetpack Compose, 59% View-based layouts |
| Project targets | 42% applications, 58% libraries |
| Multimodal assets | 20 tasks include screenshots of the UI issues |
| Platform APIs | Jetpack Compose, Room, Coroutines, Flows, Hilt, Gradle, SDK upgrades |
Task complexity is balanced across different pull request patch sizes:
| Change size | Percentage | Range |
|---|---|---|
| Small | 46% | < 27 lines of code (LOC) |
| Medium | 33% | 27 - 136 LOC |
| Large | 21% | > 136 LOC (max 435 LOC) |
The median task size is 32 changed lines.
All task files include the standard BIG-BENCH canary string to prevent inclusion in LLM training corpora. Success trajectories are manually audited to ensure legitimate fixes rather than reward hacking.
Run a single task evaluation locally:
run_task --model gemini/gemini-2.5-flash --task airbnb__lottie-android-pr_2427| Task |
|---|
android-bench/badoo__reaktive-pr_735 |
android-bench/airbnb__lottie-android-pr_2427 |
android-bench/pocketcasts-room-3 |
android-bench/thunderbird__thunderbird-android-pr_8510 |
android-bench/coil-kt__coil-pr_2669 |
android-bench/catimaloyalty__android-pr_1524 |
android-bench/nowinandroid-custom-modifier-node |
android-bench/lemmynet__jerboa-pr_1114 |
android-bench/getsentry__sentry-java-pr_2891 |
android-bench/thunderbird__thunderbird-android-pr_7103 |
android-bench/badoo__reaktive-pr_764 |
android-bench/automattic__pocket-casts-android-pr_757 |
android-bench/automattic__pocket-casts-android-pr_3970 |
android-bench/mohamedrejeb__compose-rich-editor-pr_363 |
android-bench/mohamedrejeb__compose-rich-editor-pr_319 |
android-bench/thunderbird__thunderbird-android-pr_7190 |
android-bench/badoo__reaktive-pr_786 |
android-bench/mohamedrejeb__compose-rich-editor-pr_367 |
android-bench/googlemaps__android-maps-compose-pr_336 |
android-bench/you-apps__clockyou-pr_190 |
android-bench/googlemaps__android-maps-compose-pr_327 |
android-bench/automattic__pocket-casts-android-pr_1114 |
android-bench/googlemaps__android-maps-compose-pr_603 |
android-bench/google__android-fhir-pr_1822 |
android-bench/lemmynet__jerboa-pr_941 |
android-bench/droidkaigi__conference-app-2023-pr_896 |
android-bench/bumble-tech__appyx-pr_425 |
android-bench/lemmynet__jerboa-pr_809 |
android-bench/mohamedrejeb__compose-rich-editor-pr_445 |
android-bench/alphawallet__alpha-wallet-android-pr_3329 |
android-bench/usuiat__zoomable-pr_322 |
android-bench/lemmynet__jerboa-pr_1485 |
android-bench/pocketcasts-room-4 |
android-bench/mohamedrejeb__compose-rich-editor-pr_335 |
android-bench/coil-kt__coil-pr_3137 |
android-bench/arkivanov__decompose-pr_646 |
android-bench/android__nowinandroid-pr_720 |
android-bench/thunderbird__thunderbird-android-pr_8020 |
android-bench/komikku-app__komikku-pr_992 |
android-bench/nowinandroid-compose-fling_behavior |
android-bench/getsentry__sentry-java-pr_4262 |
android-bench/element-hq__element-x-android-pr_1848 |
android-bench/badoo__reaktive-pr_763 |
android-bench/getsentry__sentry-java-pr_4724 |
android-bench/lemmynet__jerboa-pr_868 |
android-bench/chrisbanes__haze-pr_250 |
android-bench/tuskyapp__tusky-pr_4723 |
android-bench/usuiat__zoomable-pr_239 |
android-bench/bitfireat__davx5-ose-pr_1512 |
android-bench/bitfireat__davx5-ose-pr_1504 |
android-bench/nowinandroid-adaptive-1 |
android-bench/google__accompanist-pr_903 |
android-bench/you-apps__clockyou-pr_292 |
android-bench/lemmynet__jerboa-pr_1122 |
android-bench/braintree__braintree_android-pr_1323 |
android-bench/bitfireat__davx5-ose-pr_1267 |
android-bench/cahier-adaptive-1 |
android-bench/coil-kt__coil-pr_2033 |
android-bench/thunderbird__thunderbird-android-pr_8151 |
android-bench/wgtunnel__wgtunnel-pr_623 |
android-bench/antennapod__antennapod-pr_6838 |
android-bench/getsentry__sentry-java-pr_4582 |
android-bench/thunderbird__thunderbird-android-pr_8806 |
android-bench/lemmynet__jerboa-pr_894 |
android-bench/mohamedrejeb__compose-rich-editor-pr_357 |
android-bench/lihenggui__blocker-pr_725 |
android-bench/arkivanov__decompose-pr_394 |
android-bench/google__android-fhir-pr_1894 |
android-bench/nowinandroid-hide-toolbar-when-scroll |
android-bench/coil-kt__coil-pr_2945 |
android-bench/pocketcasts-room-1 |
android-bench/lemmynet__jerboa-pr_946 |
android-bench/bumble-tech__appyx-pr_694 |
android-bench/lemmynet__jerboa-pr_1068 |
android-bench/thunderbird__thunderbird-android-pr_9152 |
android-bench/mohamedrejeb__compose-rich-editor-pr_523 |
android-bench/android_snippets_1 |
android-bench/arkivanov__decompose-pr_388 |
android-bench/tuskyapp__tusky-pr_3341 |
android-bench/lemmynet__jerboa-pr_1198 |
android-bench/mohamedrejeb__compose-rich-editor-pr_403 |
android-bench/pocketcasts-room-2 |
android-bench/coil-kt__coil-pr_2121 |
android-bench/catimaloyalty__android-pr_1588 |
android-bench/android__nowinandroid-pr_553 |
android-bench/element-hq__element-x-android-pr_2073 |
android-bench/lemmynet__jerboa-pr_947 |
android-bench/seedvault-app__seedvault-pr_736 |
android-bench/nowinandroid-workmanager-2 |
android-bench/usuiat__zoomable-pr_334 |
android-bench/bumble-tech__appyx-pr_638 |
android-bench/googlemaps__android-maps-compose-pr_478 |
android-bench/element-hq__element-x-android-pr_2342 |
android-bench/lemmynet__jerboa-pr_985 |
android-bench/android_androidify_gradleplugin |
android-bench/mohamedrejeb__compose-rich-editor-pr_379 |
android-bench/lemmynet__jerboa-pr_991 |
android-bench/chrisbanes__haze-pr_710 |
android-bench/tuskyapp__tusky-pr_4821 |
android-bench/coil-kt__coil-pr_1964 |
Displaying 100 of 100 tasks
| Task |
|---|
android-bench/badoo__reaktive-pr_735 |
android-bench/airbnb__lottie-android-pr_2427 |
android-bench/pocketcasts-room-3 |
android-bench/thunderbird__thunderbird-android-pr_8510 |
android-bench/coil-kt__coil-pr_2669 |
android-bench/catimaloyalty__android-pr_1524 |
android-bench/nowinandroid-custom-modifier-node |
android-bench/lemmynet__jerboa-pr_1114 |
android-bench/getsentry__sentry-java-pr_2891 |
android-bench/thunderbird__thunderbird-android-pr_7103 |
android-bench/badoo__reaktive-pr_764 |
android-bench/automattic__pocket-casts-android-pr_757 |
android-bench/automattic__pocket-casts-android-pr_3970 |
android-bench/mohamedrejeb__compose-rich-editor-pr_363 |
android-bench/mohamedrejeb__compose-rich-editor-pr_319 |
android-bench/thunderbird__thunderbird-android-pr_7190 |
android-bench/badoo__reaktive-pr_786 |
android-bench/mohamedrejeb__compose-rich-editor-pr_367 |
android-bench/googlemaps__android-maps-compose-pr_336 |
android-bench/you-apps__clockyou-pr_190 |
android-bench/googlemaps__android-maps-compose-pr_327 |
android-bench/automattic__pocket-casts-android-pr_1114 |
android-bench/googlemaps__android-maps-compose-pr_603 |
android-bench/google__android-fhir-pr_1822 |
android-bench/lemmynet__jerboa-pr_941 |
android-bench/droidkaigi__conference-app-2023-pr_896 |
android-bench/bumble-tech__appyx-pr_425 |
android-bench/lemmynet__jerboa-pr_809 |
android-bench/mohamedrejeb__compose-rich-editor-pr_445 |
android-bench/alphawallet__alpha-wallet-android-pr_3329 |
android-bench/usuiat__zoomable-pr_322 |
android-bench/lemmynet__jerboa-pr_1485 |
android-bench/pocketcasts-room-4 |
android-bench/mohamedrejeb__compose-rich-editor-pr_335 |
android-bench/coil-kt__coil-pr_3137 |
android-bench/arkivanov__decompose-pr_646 |
android-bench/android__nowinandroid-pr_720 |
android-bench/thunderbird__thunderbird-android-pr_8020 |
android-bench/komikku-app__komikku-pr_992 |
android-bench/nowinandroid-compose-fling_behavior |
android-bench/getsentry__sentry-java-pr_4262 |
android-bench/element-hq__element-x-android-pr_1848 |
android-bench/badoo__reaktive-pr_763 |
android-bench/getsentry__sentry-java-pr_4724 |
android-bench/lemmynet__jerboa-pr_868 |
android-bench/chrisbanes__haze-pr_250 |
android-bench/tuskyapp__tusky-pr_4723 |
android-bench/usuiat__zoomable-pr_239 |
android-bench/bitfireat__davx5-ose-pr_1512 |
android-bench/bitfireat__davx5-ose-pr_1504 |
android-bench/nowinandroid-adaptive-1 |
android-bench/google__accompanist-pr_903 |
android-bench/you-apps__clockyou-pr_292 |
android-bench/lemmynet__jerboa-pr_1122 |
android-bench/braintree__braintree_android-pr_1323 |
android-bench/bitfireat__davx5-ose-pr_1267 |
android-bench/cahier-adaptive-1 |
android-bench/coil-kt__coil-pr_2033 |
android-bench/thunderbird__thunderbird-android-pr_8151 |
android-bench/wgtunnel__wgtunnel-pr_623 |
android-bench/antennapod__antennapod-pr_6838 |
android-bench/getsentry__sentry-java-pr_4582 |
android-bench/thunderbird__thunderbird-android-pr_8806 |
android-bench/lemmynet__jerboa-pr_894 |
android-bench/mohamedrejeb__compose-rich-editor-pr_357 |
android-bench/lihenggui__blocker-pr_725 |
android-bench/arkivanov__decompose-pr_394 |
android-bench/google__android-fhir-pr_1894 |
android-bench/nowinandroid-hide-toolbar-when-scroll |
android-bench/coil-kt__coil-pr_2945 |
android-bench/pocketcasts-room-1 |
android-bench/lemmynet__jerboa-pr_946 |
android-bench/bumble-tech__appyx-pr_694 |
android-bench/lemmynet__jerboa-pr_1068 |
android-bench/thunderbird__thunderbird-android-pr_9152 |
android-bench/mohamedrejeb__compose-rich-editor-pr_523 |
android-bench/android_snippets_1 |
android-bench/arkivanov__decompose-pr_388 |
android-bench/tuskyapp__tusky-pr_3341 |
android-bench/lemmynet__jerboa-pr_1198 |
android-bench/mohamedrejeb__compose-rich-editor-pr_403 |
android-bench/pocketcasts-room-2 |
android-bench/coil-kt__coil-pr_2121 |
android-bench/catimaloyalty__android-pr_1588 |
android-bench/android__nowinandroid-pr_553 |
android-bench/element-hq__element-x-android-pr_2073 |
android-bench/lemmynet__jerboa-pr_947 |
android-bench/seedvault-app__seedvault-pr_736 |
android-bench/nowinandroid-workmanager-2 |
android-bench/usuiat__zoomable-pr_334 |
android-bench/bumble-tech__appyx-pr_638 |
android-bench/googlemaps__android-maps-compose-pr_478 |
android-bench/element-hq__element-x-android-pr_2342 |
android-bench/lemmynet__jerboa-pr_985 |
android-bench/android_androidify_gradleplugin |
android-bench/mohamedrejeb__compose-rich-editor-pr_379 |
android-bench/lemmynet__jerboa-pr_991 |
android-bench/chrisbanes__haze-pr_710 |
android-bench/tuskyapp__tusky-pr_4821 |
android-bench/coil-kt__coil-pr_1964 |
Displaying 100 of 100 tasks