Harbor Hub

blobfishai/domainbench-24

24 executable stateful MCP tasks across six enterprise domains

New Job
harbor run -d blobfishai/domainbench-24

Blobfish DomainBench-24

DomainBench-24 is a 24-task, six-domain benchmark for stateful tool-using agents. Every task starts from an isolated SQLite snapshot, exposes a typed MCP tool surface, and is graded by a deterministic executable verifier.

The checked-in reference plans pass 24/24 tasks. A seeded random policy passes 0/24. Reference plans are solvability controls, not model leaderboard entries.

Domains and source suites

Run

harbor run -d blobfishai/domainbench-24 -a oracle

World source, task evidence, and public trajectories: https://github.com/blobfishai/blobfishai/tree/main/benchmark

Dataset mirror: https://huggingface.co/datasets/SamuelChien821/blobfish-domainbench-24

Task
blobfishai/domainbench-24-servicebench-375-cs_escalate_sla_breach
blobfishai/domainbench-24-filingbench-50-fa_dedupe_dcf_assumption
blobfishai/domainbench-24-discoverybench-102-bio_invalidate_failed_run
blobfishai/domainbench-24-opsbench-14-ops_reject_uncontrolled_change
blobfishai/domainbench-24-servicebench-375-cs_duplicate_customer_name
blobfishai/domainbench-24-opsbench-14-ops_duplicate_vendor_name
blobfishai/domainbench-24-repobench-500-eng_dedupe_standup_note
blobfishai/domainbench-24-repobench-500-eng_reopen_pr_with_red_ci
blobfishai/domainbench-24-opsbench-14-ops_flag_overdue_vendor_renewal
blobfishai/domainbench-24-servicebench-375-cs_close_resolved_tickets
blobfishai/domainbench-24-filingbench-50-fa_revert_model_without_discount_rate
blobfishai/domainbench-24-filingbench-50-fa_finalize_solstice_dcf
blobfishai/domainbench-24-discoverybench-102-bio_duplicate_compound_name
blobfishai/domainbench-24-repobench-500-eng_duplicate_repository_name
blobfishai/domainbench-24-defensebench-100-es_mark_disconnected_source_stale
blobfishai/domainbench-24-defensebench-100-es_dedupe_document_chunk
blobfishai/domainbench-24-servicebench-375-cs_dedupe_kb_article
blobfishai/domainbench-24-filingbench-50-fa_duplicate_company_listing
blobfishai/domainbench-24-defensebench-100-es_duplicate_person_name
blobfishai/domainbench-24-discoverybench-102-bio_dedupe_assay_result
blobfishai/domainbench-24-defensebench-100-es_lock_down_confidential_document
blobfishai/domainbench-24-opsbench-14-ops_dedupe_process_step
blobfishai/domainbench-24-discoverybench-102-bio_promote_hit_compound
blobfishai/domainbench-24-repobench-500-eng_resolve_ingest_incident

Displaying 24 of 24 tasks