Harbor Hub

blobfishai/domainbench-24

24 deeply realistic stateful MCP tasks across six enterprise domains

New Job
harbor run -d blobfishai/domainbench-24

Blobfish DomainBench-24 v3.3.2

DomainBench-24 v3.3.2 is a 24-task, six-domain benchmark for stateful tool-using agents. Every task is a high-level employee request with exactly 26 provider-native evidence reads plus 17–36 task-relevant live-domain reads, for 43–62 causal evidence reads before the decision. Each task also includes 28 agent-visible source files in nine native formats, three exact costed and dated operating options with authority status, 13 dependent derivations, 56–67 task-specific causal criteria, an isolated SQLite snapshot, a typed MCP tool surface, a semantically graded natural Slack stakeholder handoff, provider-native exact-record readbacks for every changed domain row, persisted result readback, and deterministic state/trace/workpaper verification. Hidden reference wording and JSON serialization are not graded. Exact total call order is not graded; causal read-before-write and post-write readback are.

The checked-in reference plans and exact replays pass 24/24 tasks. Thirteen task-applicable adversarial controls—including correct-state/wrong-reasoning, wrong-target, and keyword-stuffing controls—are rejected for every task: 312/312 correct rejections across 360 pristine executions. Reference plans are solvability controls, not model leaderboard entries.

Domains and source suites

Run

harbor run -d blobfishai/domainbench-24 -a oracle

World source, task evidence, and public trajectories: https://huggingface.co/datasets/SamuelChien821/blobfish-domainbench-24/tree/main/worlds

Dataset mirror: https://huggingface.co/datasets/SamuelChien821/blobfish-domainbench-24

Task
blobfishai/domainbench-24-repobench-500-eng_duplicate_repository_name
blobfishai/domainbench-24-filingbench-50-fa_revert_model_without_discount_rate
blobfishai/domainbench-24-opsbench-14-ops_dedupe_process_step
blobfishai/domainbench-24-opsbench-14-ops_reject_uncontrolled_change
blobfishai/domainbench-24-defensebench-100-es_duplicate_person_name
blobfishai/domainbench-24-discoverybench-102-bio_duplicate_compound_name
blobfishai/domainbench-24-defensebench-100-es_mark_disconnected_source_stale
blobfishai/domainbench-24-discoverybench-102-bio_invalidate_failed_run
blobfishai/domainbench-24-defensebench-100-es_lock_down_confidential_document
blobfishai/domainbench-24-servicebench-375-cs_close_resolved_tickets
blobfishai/domainbench-24-servicebench-375-cs_duplicate_customer_name
blobfishai/domainbench-24-defensebench-100-es_dedupe_document_chunk
blobfishai/domainbench-24-filingbench-50-fa_duplicate_company_listing
blobfishai/domainbench-24-servicebench-375-cs_escalate_sla_breach
blobfishai/domainbench-24-discoverybench-102-bio_promote_hit_compound
blobfishai/domainbench-24-opsbench-14-ops_flag_overdue_vendor_renewal
blobfishai/domainbench-24-repobench-500-eng_resolve_ingest_incident
blobfishai/domainbench-24-filingbench-50-fa_dedupe_dcf_assumption
blobfishai/domainbench-24-servicebench-375-cs_dedupe_kb_article
blobfishai/domainbench-24-repobench-500-eng_reopen_pr_with_red_ci
blobfishai/domainbench-24-repobench-500-eng_dedupe_standup_note
blobfishai/domainbench-24-filingbench-50-fa_finalize_solstice_dcf
blobfishai/domainbench-24-opsbench-14-ops_duplicate_vendor_name
blobfishai/domainbench-24-discoverybench-102-bio_dedupe_assay_result

Displaying 24 of 24 tasks