Loading dataset details…
harbor run -d dissei/financial-judgment-fullharbor run -d dissei/financial-judgment-fullSeven runnable point-in-time financial judgment tasks with evidence, a separate verifier and historical evaluation records.
harbor run -d dissei/financial-judgment-fullSeven runnable point-in-time financial judgment tasks with evidence, a separate verifier and historical evaluation records.
harbor run -d dissei/financial-judgment-fullDraft only; no publication performed. v2.0.0 was released publicly on GitHub and Harbor; the Hugging Face mirror is manually gated. This separate v2.0.1 candidate does not change that published release.
Re-identification and contamination notice. Names are suppressed, not anonymized. Models may recognize the real deal from dates, amounts and market facts. In our October 2026 runs, 65 of 100 valid Gemini 3.8 Flash trials named the real companies. This is an observed re-identification count, not proof of pretraining contamination or a causal estimate of score inflation. Public reference answers, grading assets and historical answers create additional contamination risks. Separate verifier isolation does not prevent recall or public-network answer lookup.
v2.0.0 and v2.0.1 scores are not directly comparable. Evidence availability, judge date context and the agent runtime changed. Historical scores below remain v2.0.0/earlier observations; no new model scores are claimed for v2.0.1.
fn3 from all three snapshots and their source catalogues. Its footer said © 2026; historical availability was unproven. No task rubric, expected source retrieval or oracle referenced it.tests/runtime. Removed privileged --corpus-root, --case and --anchor overrides.dg04, ex01, cp02), 4 changed. Added the re-identification notice and cross-version comparison warning.Local image tags: dissei/financial-judgment-{2021-12,2022-06,2022-09}:20261003-v2.0.1, each with linux/amd64 and linux/arm64. Offline stub-judge checks establish execution mechanics only, not financial grading quality.
Use the complete candidate/ directory with its seven tasks/fab01-*/ directories and three images/ archives. No v2.0.1 remote download URL is asserted by this draft. From that directory, using Docker and Harbor 0.23.0:
shasum -a 256 -c images/SHA256SUMS
docker load --input images/financial-judgment-2021-12.tar.gz
docker load --input images/financial-judgment-2022-06.tar.gz
docker load --input images/financial-judgment-2022-09.tar.gz
harbor run --path ./tasks \
-a terminus-2 -m "${DISSEI_AGENT_MODEL:?Set your authorized provider/model ID}" \
-n 1 -k 1 --ak max_turns=40 --ak reasoning_effort=high \
--ve DISSEI_JUDGE_BASE_URL="${DISSEI_JUDGE_BASE_URL:?Set your judge base URL}" \
--ve DISSEI_JUDGE_MODEL="${DISSEI_JUDGE_MODEL:?Set your judge model}" \
--ve DISSEI_JUDGE_API_KEY="${DISSEI_JUDGE_API_KEY:-}"These model/judge calls require separate authorization and may cost money; they were not run to prepare this candidate. Local jobs are not automatically uploaded. See USAGE.v2.0.1-draft.md for contract and failure semantics.
Each task adds tests/runtime/ and pinned tests/requirements.txt to the existing instruction/config/verifier/oracle layout. The one-line environment Dockerfile references its supplied agent image. Harbor builds a separate verifier with evaluator source at /tests/runtime, absent from the agent filesystem. Both phases still permit network access by default. Host owners can inspect all supplied grading assets; keep tests, solutions and historical answers out of agent mounts.
This draft describes local task packages version 2.0.1. Local instance IDs match the historical records.
| Harbor task package | Local instance | Anchor | Family | Criteria |
|---|---|---|---|---|
dissei/financial-judgment-dg04 |
fab01-2112-dg04 |
2021-12 | Diagnostic | 12 |
dissei/financial-judgment-pr02 |
fab01-2112-pr02 |
2021-12 | Predictive | 11 |
dissei/financial-judgment-ex01 |
fab01-2206-ex01 |
2022-06 | Explanatory | 13 |
dissei/financial-judgment-qn02 |
fab01-2206-qn02 |
2022-06 | Quantitative | 10 |
dissei/financial-judgment-cf01 |
fab01-2209-cf01 |
2022-09 | Counterfactual | 13 |
dissei/financial-judgment-cp02 |
fab01-2209-cp02 |
2022-09 | Comparative | 10 |
dissei/financial-judgment-st06 |
fab01-2209-st06 |
2022-09 | Strategic | 10 |
Each task is a single question at one decision date in the same transaction. Three image snapshots support linux/amd64 and linux/arm64. The dated corpus changes above remove invalid historical evidence; retained financial amounts and source notices are preserved.
The release includes the evaluator, rubrics, judge prompts and reference answers. Dataroom source and evidence are in the images; evaluator source is delivered in tests/runtime for the verifier only. The separate verifier limits ordinary task-agent access to task-specific grading files; recipients controlling the host can inspect them. Task-authoring machinery and unrelated private production workflows are excluded.
Case names are suppressed (name_suppressed). Dates, amounts and market figures are retained and may permit re-identification.
The results below were recorded on 2026-09-24–25 (UTC), including a GLM-5.3 recovery completed after midnight UTC. Version 2.0.0 preserves these historical outcomes. Reward / 100 is the equal-weight mean of seven continuous task rewards, multiplied by 100. The score reflects the documented retrieval, reasoning, submission and judging setup. Accuracy requires a separate measurement.
| Model | Historical reward / 100 | Submitted / 7 |
|---|---|---|
| Claude Fable 5.1 | 60.73 | 7 |
| GPT-6 Astra | 52.45 | 7 |
| Claude Opus 5.5 | 52.17 | 7 |
| GLM-5.3 | 44.91 | 7 |
| Kimi K3 | 35.79 | 7 |
| Gemini 3.8 Flash | 9.01 | 1 |
The 42 selected outcomes used the same seven historical task versions, Harbor 0.23.0, Terminus-2 JSON, high agent reasoning, 40 turns, 16,384 output tokens per request and a gemini-3.1-pro-preview judge with low reasoning and one vote. Historical pins document the versions evaluated. The reported September 29 runs used version-2.0.0 packages; this candidate executes version 2.0.1. The Quickstart covers local execution with your chosen credentials and omits some historical provider and token settings recorded in evaluation-protocol.json. Exact replication has not been established.
Gemini 3.8 Flash is protocol-limited: one submission, six turn-limit outcomes and 50 rejected JSON responses; five infrastructure-failed tasks were recovered without replacing the original valid outcomes. GLM-5.3's completed recovery is retained alongside its original timeout and interrupted recovery; six original successful outcomes were preserved. No successful trial was rerun to seek a higher score. This single-case pilot does not establish repeated-run variance or significance of small ranking differences.
On 2026-09-29 (UTC), Claude Opus 5.5 and Claude Sonnet 5.5 ran on the version-2.0.0 packages of this dataset. The setup matched the historical protocol: Harbor 0.23.0, Terminus-2 JSON, high agent reasoning, 40 turns, 16,384 output tokens per request, one attempt per task and one concurrent trial. The judge was gemini-3.1-pro-preview with low reasoning and one vote. Google Cloud Vertex AI served the agent and judge models.
| Model | Reward / 100 | Submitted / 7 |
|---|---|---|
| Claude Opus 5.5 | 53.97 | 7 |
| Claude Sonnet 5.5 | 37.37 | 7 |
| Task | Anchor | Family | Opus 5.5 | Sonnet 5.5 |
|---|---|---|---|---|
dg04 |
2021-12 | Diagnostic | 70.94 | 0.00 |
pr02 |
2021-12 | Predictive | 78.82 | 55.18 |
ex01 |
2022-06 | Explanatory | 22.85 | 45.70 |
qn02 |
2022-06 | Quantitative | 34.67 | 34.67 |
cf01 |
2022-09 | Counterfactual | 66.29 | 58.93 |
cp02 |
2022-09 | Comparative | 74.55 | 59.64 |
st06 |
2022-09 | Strategic | 29.65 | 7.50 |
Both jobs completed seven of seven trials with no trial errors. Every trial submitted an answer and passed the verifier integrity check. No judge verdict was unavailable because of an endpoint failure.
Claude Sonnet 5.5 scored 0 on dg04. The judge failed the task's single critical criterion, so the gate stayed closed and the graded criteria were not scored (gate_not_open).
These results are new single attempts. They are not part of the 42 historical selected outcomes above. The historical Claude Opus 5.5 row used the historical task versions and a different model route. Three of the seven Opus task rewards are identical to the historical run, and four are different (dg04, ex01 and cp02 are identical). One attempt per task cannot separate run-to-run variation from route or package differences. The native Harbor jobs for these runs are private, and the release attachments do not include their records.
| File | Contents |
|---|---|
run-records.tar.gz |
Sanitized derivative of 16 jobs and 86 attempts, including submitted answers, trajectories, verifier outputs, failures/recoveries and separately labeled execution QA/no-op/stub runs. |
run-index.json |
Stable pseudonymous job/trial relationships, archive paths and selected outcomes. |
evaluation-protocol.json |
Historical task provenance, exact model identities, parameter values and selection rules; private endpoints are redacted. |
redaction-summary.json |
Declared identity, infrastructure and historical source-excerpt transformations and limitations. |
image-inventory.json |
Measured archive checksums and clean image index/platform descriptors for the three supplied images; v2.0.1 archives are rebuilt and have new hashes. |
The comparison uses 42 selected outcomes from the 86 recorded attempts. Seven additional selected outcomes belong to a separately labeled historical Gemini 3.1 Pro baseline: seven submissions, mean reward 0.3124. All numerical outcomes are preserved. Logs, including duplicated trajectory and terminal representations, contain the redactions declared in redaction-summary.json. Native Harbor jobs and interactive trajectories remain private.
Release preparation used deterministic local judge smoke checks to test dataroom and verifier mechanics. Those checks provide no new measurement of financial grading accuracy or model quality.
License category: Other / restricted. The owner confirmed source redistribution rights. Downstream training, adaptation and redistribution remain subject to LICENSE.md. Preserve copyright and source notices.
This release uses new Harbor package identities. Earlier package histories, images and source records remain private. The publication plan records the plan preceding release.
Contact tech@dissei.credit for licensing, questions and private security reports.
Draft only; no publication performed. v2.0.0 was released publicly on GitHub and Harbor; the Hugging Face mirror is manually gated. This separate v2.0.1 candidate does not change that published release.
Re-identification and contamination notice. Names are suppressed, not anonymized. Models may recognize the real deal from dates, amounts and market facts. In our October 2026 runs, 65 of 100 valid Gemini 3.8 Flash trials named the real companies. This is an observed re-identification count, not proof of pretraining contamination or a causal estimate of score inflation. Public reference answers, grading assets and historical answers create additional contamination risks. Separate verifier isolation does not prevent recall or public-network answer lookup.
v2.0.0 and v2.0.1 scores are not directly comparable. Evidence availability, judge date context and the agent runtime changed. Historical scores below remain v2.0.0/earlier observations; no new model scores are claimed for v2.0.1.
fn3 from all three snapshots and their source catalogues. Its footer said © 2026; historical availability was unproven. No task rubric, expected source retrieval or oracle referenced it.tests/runtime. Removed privileged --corpus-root, --case and --anchor overrides.dg04, ex01, cp02), 4 changed. Added the re-identification notice and cross-version comparison warning.Local image tags: dissei/financial-judgment-{2021-12,2022-06,2022-09}:20261003-v2.0.1, each with linux/amd64 and linux/arm64. Offline stub-judge checks establish execution mechanics only, not financial grading quality.
Use the complete candidate/ directory with its seven tasks/fab01-*/ directories and three images/ archives. No v2.0.1 remote download URL is asserted by this draft. From that directory, using Docker and Harbor 0.23.0:
shasum -a 256 -c images/SHA256SUMS
docker load --input images/financial-judgment-2021-12.tar.gz
docker load --input images/financial-judgment-2022-06.tar.gz
docker load --input images/financial-judgment-2022-09.tar.gz
harbor run --path ./tasks \
-a terminus-2 -m "${DISSEI_AGENT_MODEL:?Set your authorized provider/model ID}" \
-n 1 -k 1 --ak max_turns=40 --ak reasoning_effort=high \
--ve DISSEI_JUDGE_BASE_URL="${DISSEI_JUDGE_BASE_URL:?Set your judge base URL}" \
--ve DISSEI_JUDGE_MODEL="${DISSEI_JUDGE_MODEL:?Set your judge model}" \
--ve DISSEI_JUDGE_API_KEY="${DISSEI_JUDGE_API_KEY:-}"These model/judge calls require separate authorization and may cost money; they were not run to prepare this candidate. Local jobs are not automatically uploaded. See USAGE.v2.0.1-draft.md for contract and failure semantics.
Each task adds tests/runtime/ and pinned tests/requirements.txt to the existing instruction/config/verifier/oracle layout. The one-line environment Dockerfile references its supplied agent image. Harbor builds a separate verifier with evaluator source at /tests/runtime, absent from the agent filesystem. Both phases still permit network access by default. Host owners can inspect all supplied grading assets; keep tests, solutions and historical answers out of agent mounts.
This draft describes local task packages version 2.0.1. Local instance IDs match the historical records.
| Harbor task package | Local instance | Anchor | Family | Criteria |
|---|---|---|---|---|
dissei/financial-judgment-dg04 |
fab01-2112-dg04 |
2021-12 | Diagnostic | 12 |
dissei/financial-judgment-pr02 |
fab01-2112-pr02 |
2021-12 | Predictive | 11 |
dissei/financial-judgment-ex01 |
fab01-2206-ex01 |
2022-06 | Explanatory | 13 |
dissei/financial-judgment-qn02 |
fab01-2206-qn02 |
2022-06 | Quantitative | 10 |
dissei/financial-judgment-cf01 |
fab01-2209-cf01 |
2022-09 | Counterfactual | 13 |
dissei/financial-judgment-cp02 |
fab01-2209-cp02 |
2022-09 | Comparative | 10 |
dissei/financial-judgment-st06 |
fab01-2209-st06 |
2022-09 | Strategic | 10 |
Each task is a single question at one decision date in the same transaction. Three image snapshots support linux/amd64 and linux/arm64. The dated corpus changes above remove invalid historical evidence; retained financial amounts and source notices are preserved.
The release includes the evaluator, rubrics, judge prompts and reference answers. Dataroom source and evidence are in the images; evaluator source is delivered in tests/runtime for the verifier only. The separate verifier limits ordinary task-agent access to task-specific grading files; recipients controlling the host can inspect them. Task-authoring machinery and unrelated private production workflows are excluded.
Case names are suppressed (name_suppressed). Dates, amounts and market figures are retained and may permit re-identification.
The results below were recorded on 2026-09-24–25 (UTC), including a GLM-5.3 recovery completed after midnight UTC. Version 2.0.0 preserves these historical outcomes. Reward / 100 is the equal-weight mean of seven continuous task rewards, multiplied by 100. The score reflects the documented retrieval, reasoning, submission and judging setup. Accuracy requires a separate measurement.
| Model | Historical reward / 100 | Submitted / 7 |
|---|---|---|
| Claude Fable 5.1 | 60.73 | 7 |
| GPT-6 Astra | 52.45 | 7 |
| Claude Opus 5.5 | 52.17 | 7 |
| GLM-5.3 | 44.91 | 7 |
| Kimi K3 | 35.79 | 7 |
| Gemini 3.8 Flash | 9.01 | 1 |
The 42 selected outcomes used the same seven historical task versions, Harbor 0.23.0, Terminus-2 JSON, high agent reasoning, 40 turns, 16,384 output tokens per request and a gemini-3.1-pro-preview judge with low reasoning and one vote. Historical pins document the versions evaluated. The reported September 29 runs used version-2.0.0 packages; this candidate executes version 2.0.1. The Quickstart covers local execution with your chosen credentials and omits some historical provider and token settings recorded in evaluation-protocol.json. Exact replication has not been established.
Gemini 3.8 Flash is protocol-limited: one submission, six turn-limit outcomes and 50 rejected JSON responses; five infrastructure-failed tasks were recovered without replacing the original valid outcomes. GLM-5.3's completed recovery is retained alongside its original timeout and interrupted recovery; six original successful outcomes were preserved. No successful trial was rerun to seek a higher score. This single-case pilot does not establish repeated-run variance or significance of small ranking differences.
On 2026-09-29 (UTC), Claude Opus 5.5 and Claude Sonnet 5.5 ran on the version-2.0.0 packages of this dataset. The setup matched the historical protocol: Harbor 0.23.0, Terminus-2 JSON, high agent reasoning, 40 turns, 16,384 output tokens per request, one attempt per task and one concurrent trial. The judge was gemini-3.1-pro-preview with low reasoning and one vote. Google Cloud Vertex AI served the agent and judge models.
| Model | Reward / 100 | Submitted / 7 |
|---|---|---|
| Claude Opus 5.5 | 53.97 | 7 |
| Claude Sonnet 5.5 | 37.37 | 7 |
| Task | Anchor | Family | Opus 5.5 | Sonnet 5.5 |
|---|---|---|---|---|
dg04 |
2021-12 | Diagnostic | 70.94 | 0.00 |
pr02 |
2021-12 | Predictive | 78.82 | 55.18 |
ex01 |
2022-06 | Explanatory | 22.85 | 45.70 |
qn02 |
2022-06 | Quantitative | 34.67 | 34.67 |
cf01 |
2022-09 | Counterfactual | 66.29 | 58.93 |
cp02 |
2022-09 | Comparative | 74.55 | 59.64 |
st06 |
2022-09 | Strategic | 29.65 | 7.50 |
Both jobs completed seven of seven trials with no trial errors. Every trial submitted an answer and passed the verifier integrity check. No judge verdict was unavailable because of an endpoint failure.
Claude Sonnet 5.5 scored 0 on dg04. The judge failed the task's single critical criterion, so the gate stayed closed and the graded criteria were not scored (gate_not_open).
These results are new single attempts. They are not part of the 42 historical selected outcomes above. The historical Claude Opus 5.5 row used the historical task versions and a different model route. Three of the seven Opus task rewards are identical to the historical run, and four are different (dg04, ex01 and cp02 are identical). One attempt per task cannot separate run-to-run variation from route or package differences. The native Harbor jobs for these runs are private, and the release attachments do not include their records.
| File | Contents |
|---|---|
run-records.tar.gz |
Sanitized derivative of 16 jobs and 86 attempts, including submitted answers, trajectories, verifier outputs, failures/recoveries and separately labeled execution QA/no-op/stub runs. |
run-index.json |
Stable pseudonymous job/trial relationships, archive paths and selected outcomes. |
evaluation-protocol.json |
Historical task provenance, exact model identities, parameter values and selection rules; private endpoints are redacted. |
redaction-summary.json |
Declared identity, infrastructure and historical source-excerpt transformations and limitations. |
image-inventory.json |
Measured archive checksums and clean image index/platform descriptors for the three supplied images; v2.0.1 archives are rebuilt and have new hashes. |
The comparison uses 42 selected outcomes from the 86 recorded attempts. Seven additional selected outcomes belong to a separately labeled historical Gemini 3.1 Pro baseline: seven submissions, mean reward 0.3124. All numerical outcomes are preserved. Logs, including duplicated trajectory and terminal representations, contain the redactions declared in redaction-summary.json. Native Harbor jobs and interactive trajectories remain private.
Release preparation used deterministic local judge smoke checks to test dataroom and verifier mechanics. Those checks provide no new measurement of financial grading accuracy or model quality.
License category: Other / restricted. The owner confirmed source redistribution rights. Downstream training, adaptation and redistribution remain subject to LICENSE.md. Preserve copyright and source notices.
This release uses new Harbor package identities. Earlier package histories, images and source records remain private. The publication plan records the plan preceding release.
Contact tech@dissei.credit for licensing, questions and private security reports.