Loading dataset details…
Legal consent sweep
A complete acquisition consent workstream: review a populated data room, reconcile contract amendments and existing consents, prepare effective requests and notices, and advise the sponsor on the Required Consents closing condition.
Provenance. This environment is built from real legal work done by real organizations on the DOE platform, including their requests, documents, data and corrections. Every file is a synthetic version: every person, firm, company, figure and document is replaced, and no customer content is included. The data room, mailbox and records mirror how these files and workflows look and are used in practice.
The deal changes from an asset purchase to a reverse triangular merger and the closing date moves. The associate must apply the revised instructions across the matter, identify the governing documents, calculate receipt deadlines and distinguish customer ACV from the merger agreement's revenue measure.
The environment includes contracts, scanned records, finance workbooks, correspondence and local mail, Drive, Teams and EDGAR tools. Agents investigate the matter, obtain information through simulated correspondence and deliver a consent tracker, letters, a closing-condition record and a client memo. The package includes the complete workspace, separate verifier and reference solution.
Maximum-effort results
Twelve scored attempts on task 1.4.0, using native harnesses at their maximum effort setting. All missed at least one critical requirement. The interesting evidence is in the work: which facts agents reconcile, what they ask for, and whether their final advice follows through.
| Model | Trial scores | Mean | Agent time allowance |
|---|---|---|---|
| Claude Opus 5.5 | 0.3961, 0.3032, 0.4385 | 0.379 | 90m, 90m, 22h |
| Sol 6.1 | 0.2117, 0.3889, 0.3775 | 0.326 | 22h each |
| GPT-6 Astra · normal context | 0.0000, 0.3017, 0.3081 | 0.203 | 22h each |
| Grok 4.7 | 0.0000, 0.0000, 0.0000† | 0.000 | 90m each |
Explore the rollouts, files and grading evidence · Download scores · Vector figure
This is a small, mixed-budget sample, with different provider routes. It is not a controlled test of effort settings. Opus, Sol and Astra used max; Grok used xhigh. Sol and Astra used the normal Codex context window. Verifier-only repairs reused the original submitted outputs.
† Grok has two completed runs and one execution failure counted retrospectively; the failed run is marked separately in the graph. Its native logs retain images and child sessions omitted from structured ATIF. Three Astra runs with aggressive compaction were cancelled and excluded; two encountered a mail-ID defect whose wider impact is unmeasured. GLM 5.3 could not run because of Bedrock account eligibility and has no score.
Earlier published baseline
The previous release reports high effort, rather than medium. These historical results are retained separately and are not pooled with the maximum-effort sample.
| Model | Native harness | Trials | Mean | Median | Range | Strict passes |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | Claude Code · high | 10 | 0.234 | 0.244 | 0.000–0.343 | 0/10 |
| GPT-6 Astra | Codex CLI · high | 6 | 0.200 | 0.222 | 0.000–0.289 | 0/6 |
Both used E2B. The earlier Astra summary excludes provider request-size, rate and quota interruptions. These are the aggregates published in release 1.4.3; this documentation update does not reconstruct that earlier cohort.
How to run
Run with Harbor 0.23 or later. Replace the placeholders with your agent, model and environment:
harbor run -d doe/legal-consent-sweep@1.4.4 -a <agent> -m <model> -e <docker|e2b>Use docker with a running Docker environment, or e2b with E2B_API_KEY set in the shell running Harbor. Configure the agent's own model credentials as required by that agent. For an agent that calls a model API, permit its host with --allow-agent-host, such as --allow-agent-host api.anthropic.com for Claude Code.
The separate verifier uses two judge families, Anthropic and OpenAI. Set ANTHROPIC_API_KEY and OPENAI_API_KEY in the environment of the Harbor process, with access to the judge models configured in task.toml. The default panel is JUDGE_FAMILIES=anthropic,openai. Credentials supplied only to the agent do not reach the verifier.
Alternatively, set JUDGE_TRANSPORT=bedrock and AWS_BEARER_TOKEN_BEDROCK to route both judge families through Bedrock. Harbor's preflight still requires ANTHROPIC_API_KEY and OPENAI_API_KEY to be set; in this mode they can both have the literal value bedrock, because the verifier uses the Bedrock bearer token for authentication.
The task defaults to a 90-minute agent limit and a separate 30-minute verifier limit; the campaign above used longer agent allowances where shown. Deliverables are written to /app/output, the artifact directory collected by Harbor and passed to the separate verifier.
Deterministic checks assess documents, calculations and recorded actions. The two judge families assess written advice and correspondence. A strict pass requires every evaluated critical criterion to pass. score measures partial credit; required judge results must be available for a score to be valid. GRADING.md describes the scoring rules.
Dataset release 1.4.4; task doe/legal-coc-consent-sweep@1.4.0. This release changes documentation only. See TERMS.txt for evaluation and training rights.