Explicit Edit

Explicit Edit measures whether coding agents can make precise file changes while preserving everything else. The public suite contains 226 deterministic tasks covering replacements, insertions, deletions, copies, moves, large files, multiple file types, and Unicode edge cases.

This repository is the reproducible benchmark core. It owns task generation, the byte-exact verifier, the reference multi-harness runner, normalized result schema, acceptance checks, aggregation semantics, tests, and publication workflows. It does not store the accepted result corpus or deploy the analytics site.

Run it from Harbor

The runnable benchmark is published as alexshpunt/explicit-edit:

harbor run -d alexshpunt/explicit-edit@1.0.0 -a <agent> -m <provider/model>

Harbor also hosts the official public leaderboard. See Harbor publication for the immutable task package and curated result projection.

Run the reference harness

Use Linux or WSL2 with Node.js 24+, Python 3, and Bubblewrap (bwrap). Install and authenticate the agent harness you want to test separately. Model calls use your account and may cost money; no model is selected silently.

npm ci
npm run benchmark -- init
npm run benchmark -- check --config benchmark.config.ts
npm run benchmark -- run --config benchmark.config.ts \
  --smoke --task replace-all-10-plain --run-id smoke

Review the smoke result and effective model settings before a full run. Then export a safe normalized bundle:

npm run benchmark -- run --config benchmark.config.ts \
  --oracle-recoveries 5 --concurrency 10 --timeout-seconds 120 --run-id my-run
npm run benchmark -- export results/my-run
npm run benchmark -- report results/my-run/normalized

The exporter keeps recomputable result and tool-use facts. It excludes raw prompts, commands, output, sessions, workspaces, and credentials. See Running Explicit Edit and Methodology.

Results and analysis

Accepted observations live in the Hugging Face Dataset alexshpunt/explicit-edit-results. Its retained source/accepted/ bundles are the canonical public evidence.

The public Explicit Edit Explorer reads checksum-pinned Dataset shards in the browser. It provides filters, configuration comparisons, task-family views, recovery analysis, and charts. The Explorer is a derived analytical view, not a second result store.

The official Score is:

coverage × (0.75 × first exact rate + 0.25 × final exact rate)

First exact, final exact, EOF-normalized results, recovery, timeouts, and efficiency remain separate facts. Read Architecture for the source-of-truth boundaries.

Contribute a result or harness

Use Contributing results to export, inspect, and submit an observation. New harness adapters must preserve the normalized identity and safety contract.

Development

npm run check

npm run check formats, lints, type-checks, checks for machine-local paths, and runs unit tests. Integration tests require the real harness dependencies described in Running Explicit Edit.