CounselBench-100

CounselBench-100 is a public, executable legal-agent benchmark with 100 original synthetic matters across ten practice areas. Each task contains 96 seeded records in 12 folders, a pinned filesystem MCP world, a 109-call reference trajectory, and a deterministic 182-criterion verifier.

Release v1.1.0

• 100 tasks and 9,600 unique text-native records
• seven formats: Markdown, TXT, EML, CSV, JSON, XML, and HTML
• 25% procedure, 55% structured findings, and 20% counsel memo
• continuous criteria score plus a separate strict full-task pass
• no model, network, clock, locale, or randomness in grading
• 100/100 Dockerized oracle trials and zero false accepts across 400 negative controls

harbor download blobfishai/counselbench-100@v1.1.0 \
  --output-dir ./counselbench

The v1.1 release replaces the launch version's brittle full-output equality with field-level criteria whose expected values are recoverable from the seeded records. Full exact match remains a diagnostic; it no longer erases legitimate partial credit.

• Benchmark explorer: https://blobfish.ai/benchmarks/counselbench-100
• Dataset and trajectories: https://huggingface.co/datasets/SamuelChien821/counselbench-100
• Builder and verifier: https://github.com/blobfishai/legal-agent-simulation/tree/master/benchmark/counselbench100