Loading dataset details…
harbor run -d llamaindex/extractbench-proharbor run -d llamaindex/extractbench-proExtractBench Pro: 100 synthetic document extraction tasks across six challenges; deterministic Value F1 reward.
harbor run -d llamaindex/extractbench-proExtractBench Pro: 100 synthetic document extraction tasks across six challenges; deterministic Value F1 reward.
harbor run -d llamaindex/extractbench-proQuick links: [🌐 Website] [💻 Code] [📚 ExtractBench]
ExtractBench Pro is the hard companion to ExtractBench. The task is the same: given a full document and a user-written schema, return a schema-valid JSON object with the correct, exhaustive values and evidence for each one. The difference is what makes a value hard. In ExtractBench Pro, many values cannot be copied from the page. They have to be read off a chart or a drawing, counted under the document’s own rules, or decoded with its own vocabulary. The documents are derived from real long-tail documents, and grouped into six challenges.
ExtractBench Pro contains 100 documents (2,622 pages, 1–83 pages each) across 86 document types, with 144,141 expected values. Every document is synthetic: its values are fictional and fixed before the page is rendered, so the expected output is exact by construction.
Challenge names are the canonical split identifiers, case-ID prefixes, and file paths. This layout update preserves the same 100 documents, schemas, expected values, and scoring rules. Earlier HF revisions retain their original layout. The evaluation framework translates legacy inputs, and split-name-migration.json records the old-to-new case mapping and unchanged PDF hashes. Harbor exports use new task IDs without legacy aliases.
Each document belongs to exactly one subset: the challenge a naive reader most often gets wrong in it. A document can also contain values of other challenges, and values that are read directly from the page.
| Subset | Challenge | Documents | Pages | Expected values |
|---|---|---|---|---|
interpretation_rules |
Interpretation rules. Document instructions govern records, versions, scopes, codes and units | 22 | 875 | 121,287 |
custom_terminology |
Custom terminology. Definitions in the document determine the meanings of terms and categories | 14 | 606 | 15,242 |
conditional_aggregation |
Conditional aggregation. Conditions determine which records contribute to counts and sums | 12 | 225 | 409 |
relational_diagrams |
Relational diagrams. All or part of the value depends on a line in a diagram rather than a printed value | 20 | 352 | 2,349 |
charts_and_maps |
Charts and maps. Read charts, maps and other graphics alongside the document’s instructions | 25 | 557 | 1,347 |
long_single_pages |
Long single pages. Hundreds of values on one extremely long, narrow chart sheet, such as a well log or a time-location chart | 7 | 7 | 3,507 |
| Total | 100 | 2,622 | 144,141 |
Each document is also tagged by how most of its pages were captured, by length and by business domain. The axes are independent of the challenge subset.
| Perception | Documents | What most pages look like |
|---|---|---|
digital |
25 | Born-digital pages with a text layer |
image |
8 | Clean page images without scanner noise |
scan |
44 | Light or medium scan |
heavy_scan |
16 | Heavy or extreme scan |
handwritten |
7 | Hand-filled forms or handwritten values |
short (10 pages or fewer) 8 documents, medium (11–50 pages) 81 documents, long (more than 50 pages) 11 documents.D1 finance (27), D2 energy (19), D3 government (16), D5 supply chain (9), D6 healthcare (13), D7 legal (3), D8 real estate (13), using the ExtractBench domain codes.Seven of the eight short documents are single plotter sheets 13 to 17 feet long (well logs, a borehole core log, a track chart and time-location charts), scanned or photocopied, with no text layer; each asks for 390 to 715 values. The eighth is a two-page Gantt schedule board. Charts and maps and drawings in other documents are often page images too, so their values cannot be read from a text layer. Some scanned documents carry a text layer that does not match the printed page. The printed page is authoritative. One document prints two pages on each sheet, turned sideways.
Load a split with datasets:
from datasets import load_dataset
import json
ds = load_dataset("llamaindex/ExtractBenchPro", split="charts_and_maps")
case = ds[0]
schema = json.loads(case["data_schema"])
expected = json.loads(case["expected_output"])To run an extraction system, download the files in the layout expected by the evaluation framework, then run a pipeline:
uv run extract-bench download --dataset pro # data-pro/<split>/<doc>.pdf + <doc>.test.json
uv run extract-bench run llamaextract_agentic --dataset pro # inference -> evaluation -> reportsExtractBench Pro is scored the same way as ExtractBench. Value accuracy asks whether a system returned the right values, and is scored with unified value F1 on every document. Grounding asks whether the system can show where each value came from, and is scored only on the 81 documents with box ground truth.
Scoring is fully deterministic: the same predictions and ground truth always produce the same score, with no model in the loop. The code repository contains the full metric definitions.
The evaluation dataset files include:
Each line in a JSONL file is one document-schema test case:
{
"id": "long_single_pages/wireline_log_01", // Unique test-case id: <subset>/<stem>
"category": "long_single_pages", // Subset name
"pdf": "docs/long_single_pages/wireline_log_01.pdf", // Relative path to the source document
"data_schema": "{...}", // JSON-encoded JSON Schema the output must validate against
"expected_output": "{...}", // JSON-encoded ground-truth extraction
"field_rules": "{...}", // JSON-encoded per-field scoring rules
"repeated_structure": "{}", // JSON-encoded record-identity config (empty: records are matched on their values)
"tags": ["source:synthetic", "challenge:long_single_pages", "perception:scan", "length:short", "domain:D2"]
}data_schema, expected_output, field_rules and repeated_structure are stored as JSON-encoded strings because schemas differ by document. Decode them with json.loads. The largest field_rules cell is about 6 MB, so the dataset viewer shows only a preview of each cell. The downloaded file contains the complete data.
field_rules maps each value path to a scoring rule:
evidence: accepted readings and their source locations, stored as {page, bbox, quote, value}; any accepted reading counts as correctsource_policy: whether the value must appear verbatim in the documentevidence_required / verified: whether located evidence exists for the valuenormalizers: value normalization applied before comparisonAll documents are synthetic and contain fictional values. The dataset is released under the Apache 2.0 License. For questions, contact us through the GitHub repository.
Requires Harbor 0.24.0 or later, Git with Git LFS, and Docker.
Run git lfs install before the first use, then run the tasks directly from this repository:
harbor run --repo https://huggingface.co/datasets/llamaindex/ExtractBenchPro -p tasks \
-a codex -m MODEL --allow-agent-host MODEL_API_HOSTInstructions reproduce the published coding-agent prompt, with a standardized document filename.
Agents receive only document.pdf and schema.json and write output.json.
During agent execution, network access is denied except for explicitly allowed
model API hosts. Set MODEL_API_HOST to the hostname of your model endpoint;
repeat --allow-agent-host only for additional required model API hosts.
Do not allow dataset hosts, public proxies, or broad IP ranges. Setup can download
agent tools before extraction begins; the separate verifier runs without networking.
The separate verifier uses the existing ExtractBench evaluator at commit 2f27f100730afc7c395602e26a2a87e07ea66b31.
The reward is Value F1 in [0, 1]; metrics.json also records perfect-document status.
feedback.json records aligned output/reference cell paths, value outcomes, and
each matched cell's contribution to the same F1 reward. Paths refer to normalized
scoring inputs; one-sided cells contribute to the denominator. This feedback does
not change the reward or assign credit to reasoning steps.
This prompt does not request citations, so this lane does not measure grounding.
The agent timeout is 1800 seconds per document.
The JSONL files and ordinary Hugging Face dataset configuration are preserved.
harbor-manifest.json records source revisions and hashes for reproducibility.
For a no-inference packaging smoke test, use -a oracle -l 1 instead of -a codex -m MODEL.
The oracle copies the reference output; it checks packaging and grading, not extraction ability.
This release is based on HF commit fdf734a4d241a011bb262948b90cd7fbdf9d9ed7 and uses task names derived from document identifiers, independent of challenge labels. Original Harbor task names have no aliases in this release.
harbor run -d llamaindex/extractbench-pro@hf-fdf734a4d241 -a codex -m MODEL --allow-agent-host MODEL_API_HOSTQuick links: [🌐 Website] [💻 Code] [📚 ExtractBench]
ExtractBench Pro is the hard companion to ExtractBench. The task is the same: given a full document and a user-written schema, return a schema-valid JSON object with the correct, exhaustive values and evidence for each one. The difference is what makes a value hard. In ExtractBench Pro, many values cannot be copied from the page. They have to be read off a chart or a drawing, counted under the document’s own rules, or decoded with its own vocabulary. The documents are derived from real long-tail documents, and grouped into six challenges.
ExtractBench Pro contains 100 documents (2,622 pages, 1–83 pages each) across 86 document types, with 144,141 expected values. Every document is synthetic: its values are fictional and fixed before the page is rendered, so the expected output is exact by construction.
Challenge names are the canonical split identifiers, case-ID prefixes, and file paths. This layout update preserves the same 100 documents, schemas, expected values, and scoring rules. Earlier HF revisions retain their original layout. The evaluation framework translates legacy inputs, and split-name-migration.json records the old-to-new case mapping and unchanged PDF hashes. Harbor exports use new task IDs without legacy aliases.
Each document belongs to exactly one subset: the challenge a naive reader most often gets wrong in it. A document can also contain values of other challenges, and values that are read directly from the page.
| Subset | Challenge | Documents | Pages | Expected values |
|---|---|---|---|---|
interpretation_rules |
Interpretation rules. Document instructions govern records, versions, scopes, codes and units | 22 | 875 | 121,287 |
custom_terminology |
Custom terminology. Definitions in the document determine the meanings of terms and categories | 14 | 606 | 15,242 |
conditional_aggregation |
Conditional aggregation. Conditions determine which records contribute to counts and sums | 12 | 225 | 409 |
relational_diagrams |
Relational diagrams. All or part of the value depends on a line in a diagram rather than a printed value | 20 | 352 | 2,349 |
charts_and_maps |
Charts and maps. Read charts, maps and other graphics alongside the document’s instructions | 25 | 557 | 1,347 |
long_single_pages |
Long single pages. Hundreds of values on one extremely long, narrow chart sheet, such as a well log or a time-location chart | 7 | 7 | 3,507 |
| Total | 100 | 2,622 | 144,141 |
Each document is also tagged by how most of its pages were captured, by length and by business domain. The axes are independent of the challenge subset.
| Perception | Documents | What most pages look like |
|---|---|---|
digital |
25 | Born-digital pages with a text layer |
image |
8 | Clean page images without scanner noise |
scan |
44 | Light or medium scan |
heavy_scan |
16 | Heavy or extreme scan |
handwritten |
7 | Hand-filled forms or handwritten values |
short (10 pages or fewer) 8 documents, medium (11–50 pages) 81 documents, long (more than 50 pages) 11 documents.D1 finance (27), D2 energy (19), D3 government (16), D5 supply chain (9), D6 healthcare (13), D7 legal (3), D8 real estate (13), using the ExtractBench domain codes.Seven of the eight short documents are single plotter sheets 13 to 17 feet long (well logs, a borehole core log, a track chart and time-location charts), scanned or photocopied, with no text layer; each asks for 390 to 715 values. The eighth is a two-page Gantt schedule board. Charts and maps and drawings in other documents are often page images too, so their values cannot be read from a text layer. Some scanned documents carry a text layer that does not match the printed page. The printed page is authoritative. One document prints two pages on each sheet, turned sideways.
Load a split with datasets:
from datasets import load_dataset
import json
ds = load_dataset("llamaindex/ExtractBenchPro", split="charts_and_maps")
case = ds[0]
schema = json.loads(case["data_schema"])
expected = json.loads(case["expected_output"])To run an extraction system, download the files in the layout expected by the evaluation framework, then run a pipeline:
uv run extract-bench download --dataset pro # data-pro/<split>/<doc>.pdf + <doc>.test.json
uv run extract-bench run llamaextract_agentic --dataset pro # inference -> evaluation -> reportsExtractBench Pro is scored the same way as ExtractBench. Value accuracy asks whether a system returned the right values, and is scored with unified value F1 on every document. Grounding asks whether the system can show where each value came from, and is scored only on the 81 documents with box ground truth.
Scoring is fully deterministic: the same predictions and ground truth always produce the same score, with no model in the loop. The code repository contains the full metric definitions.
The evaluation dataset files include:
Each line in a JSONL file is one document-schema test case:
{
"id": "long_single_pages/wireline_log_01", // Unique test-case id: <subset>/<stem>
"category": "long_single_pages", // Subset name
"pdf": "docs/long_single_pages/wireline_log_01.pdf", // Relative path to the source document
"data_schema": "{...}", // JSON-encoded JSON Schema the output must validate against
"expected_output": "{...}", // JSON-encoded ground-truth extraction
"field_rules": "{...}", // JSON-encoded per-field scoring rules
"repeated_structure": "{}", // JSON-encoded record-identity config (empty: records are matched on their values)
"tags": ["source:synthetic", "challenge:long_single_pages", "perception:scan", "length:short", "domain:D2"]
}data_schema, expected_output, field_rules and repeated_structure are stored as JSON-encoded strings because schemas differ by document. Decode them with json.loads. The largest field_rules cell is about 6 MB, so the dataset viewer shows only a preview of each cell. The downloaded file contains the complete data.
field_rules maps each value path to a scoring rule:
evidence: accepted readings and their source locations, stored as {page, bbox, quote, value}; any accepted reading counts as correctsource_policy: whether the value must appear verbatim in the documentevidence_required / verified: whether located evidence exists for the valuenormalizers: value normalization applied before comparisonAll documents are synthetic and contain fictional values. The dataset is released under the Apache 2.0 License. For questions, contact us through the GitHub repository.
Requires Harbor 0.24.0 or later, Git with Git LFS, and Docker.
Run git lfs install before the first use, then run the tasks directly from this repository:
harbor run --repo https://huggingface.co/datasets/llamaindex/ExtractBenchPro -p tasks \
-a codex -m MODEL --allow-agent-host MODEL_API_HOSTInstructions reproduce the published coding-agent prompt, with a standardized document filename.
Agents receive only document.pdf and schema.json and write output.json.
During agent execution, network access is denied except for explicitly allowed
model API hosts. Set MODEL_API_HOST to the hostname of your model endpoint;
repeat --allow-agent-host only for additional required model API hosts.
Do not allow dataset hosts, public proxies, or broad IP ranges. Setup can download
agent tools before extraction begins; the separate verifier runs without networking.
The separate verifier uses the existing ExtractBench evaluator at commit 2f27f100730afc7c395602e26a2a87e07ea66b31.
The reward is Value F1 in [0, 1]; metrics.json also records perfect-document status.
feedback.json records aligned output/reference cell paths, value outcomes, and
each matched cell's contribution to the same F1 reward. Paths refer to normalized
scoring inputs; one-sided cells contribute to the denominator. This feedback does
not change the reward or assign credit to reasoning steps.
This prompt does not request citations, so this lane does not measure grounding.
The agent timeout is 1800 seconds per document.
The JSONL files and ordinary Hugging Face dataset configuration are preserved.
harbor-manifest.json records source revisions and hashes for reproducibility.
For a no-inference packaging smoke test, use -a oracle -l 1 instead of -a codex -m MODEL.
The oracle copies the reference output; it checks packaging and grading, not extraction ability.
This release is based on HF commit fdf734a4d241a011bb262948b90cd7fbdf9d9ed7 and uses task names derived from document identifiers, independent of challenge labels. Original Harbor task names have no aliases in this release.
harbor run -d llamaindex/extractbench-pro@hf-fdf734a4d241 -a codex -m MODEL --allow-agent-host MODEL_API_HOST