Loading dataset details…
harbor run -d elfsong/mercury-evalharbor run -d elfsong/mercury-evalMercury eval: 256 Python code-efficiency tasks; CC-BY-NC-4.0.
harbor run -d elfsong/mercury-evalMercury eval: 256 Python code-efficiency tasks; CC-BY-NC-4.0.
harbor run -d elfsong/mercury-evalAll 256 problems from the eval split of
Elfsong/Mercury, with 1,946
original test cases and 4,722 original reference solutions. Difficulty:
88 easy, 81 medium, 87 hard. No training problems are included.
This is a Harbor adaptation of Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024), by Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. Data and adapted tasks retain CC-BY-NC-4.0. See ATTRIBUTION.md.
The agent implements a Python Solution method in /app/solution.py. Only that file
is transferred to a separate verifier container. Hidden tests and references are
absent from the agent image. In the verifier container, worker subprocesses run as
an unprivileged user and cannot read the root-owned reference suite or write the
verifier's reward files. Python 3.12.13 and runtime dependencies are pinned.
All original offline test cases and converter/comparator functions are preserved;
random generators are not invoked during scoring.
The verifier reports pass_at_1, beyond_at_1, percentile, and efficiency_valid.
reward is the binary correctness result. A correct candidate and every reference
are timed in separate subprocesses in the same verifier environment: one warm-up,
128 timed repetitions per case, independent arguments reconstructed from deep-copied
JSON inputs before
timing, process user+system CPU time, and the median across cases. Each solution
has a 30-second wall-clock limit and 2 GiB address-space limit. Three expensive
tasks (expression-add-operators, burst-balloons, and
arithmetic-slices-ii-subsequence) allow 900 seconds per solution to keep all 128
repetitions. Verifier timeouts cover all reference budgets plus 60 seconds, with
a minimum of 30 minutes; verifiers use one CPU and 4 GiB memory. Full efficiency
evaluation, especially reference collection for these tasks, can be slow.
Only references passing every test enter the timing distribution. The efficiency
score is clamp((slowest_reference - candidate) / (slowest_reference - fastest_reference), 0, 1).
Equal reference runtimes yield 1 iff the candidate is at least as fast. Percentile
uses the fraction of slower references plus half the ties. Incorrect submissions
score zero. Missing reference baselines are recorded as invalid efficiency, never
as a measured zero without an accompanying validity indicator.
The dataset metric averages correctness and efficiency over all trials,
including failures and missing results. Use one trial per task for Pass@1 and
Beyond@1. efficiency_coverage exposes missing baselines; percentiles are averaged
only over valid efficiency results.
The scoring formula and 128-repeat CPU-time protocol follow Mercury_Eval. This adapter fixes in-place verification by checking the arguments actually passed to the candidate, instead of the unmodified originals. It includes failures in the dataset efficiency denominator, avoids upstream cross-machine reference caches, omits per-iteration wall-clock instrumentation from the CPU timing loop, and uses an explicit result file and resource limits. Standard LeetCode node constructors accept optional values/children, and inputs are reconstructed before each repetition to avoid recursive deepcopy failures on long linked lists. Additional standard-library module imports are available in addition to the original wildcard prelude.
Consequently, this is the mercury-harbor-v1 protocol, not a claim of numerically identical paper/LeetCode/Mercury_Eval results. CPU measurements remain hardware and noise dependent. Run one concurrent trial on a dedicated CPU for reported scores. The original test suite is finite and may accept incorrect programs; no new hidden cases are added in this port. Subprocess limits are not a security sandbox: run agent submissions only inside the disposable Harbor verifier container.
Requires Harbor 0.23.0 or later and a supported container backend (Docker by default).
cd mercury-eval
harbor run -c local-eval.yaml -p . -a oracle
harbor run -c local-eval.yaml -p . -a nop
harbor run -c local-eval.yaml -p . -a codex -m "<model>"The oracle installs a validated original reference solution. Its correctness should
be 1; its efficiency is measured and need not be 1. The no-op starter raises
NotImplementedError and must receive zero correctness and efficiency.
For local paths, local-eval.yaml explicitly loads the custom aggregation script;
Harbor 0.23 does not automatically load a local directory's metric.py.
After publishing, use harbor run -d <your-namespace>/mercury-eval -a <agent> -m <model> -n 1;
Harbor Hub packages automatically load the included metric.
The exact source revision, task digests, oracle selections, and image digest are
recorded in provenance.json and dataset.toml.
@inproceedings{du2024mercury,
title={Mercury: A code efficiency benchmark for code large language models},
author={Du, Mingzhe and Luu, Anh Tuan and Ji, Bin and Liu, Qian and Ng, See-Kiong},
booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2024}
}All 256 problems from the eval split of
Elfsong/Mercury, with 1,946
original test cases and 4,722 original reference solutions. Difficulty:
88 easy, 81 medium, 87 hard. No training problems are included.
This is a Harbor adaptation of Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024), by Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. Data and adapted tasks retain CC-BY-NC-4.0. See ATTRIBUTION.md.
The agent implements a Python Solution method in /app/solution.py. Only that file
is transferred to a separate verifier container. Hidden tests and references are
absent from the agent image. In the verifier container, worker subprocesses run as
an unprivileged user and cannot read the root-owned reference suite or write the
verifier's reward files. Python 3.12.13 and runtime dependencies are pinned.
All original offline test cases and converter/comparator functions are preserved;
random generators are not invoked during scoring.
The verifier reports pass_at_1, beyond_at_1, percentile, and efficiency_valid.
reward is the binary correctness result. A correct candidate and every reference
are timed in separate subprocesses in the same verifier environment: one warm-up,
128 timed repetitions per case, independent arguments reconstructed from deep-copied
JSON inputs before
timing, process user+system CPU time, and the median across cases. Each solution
has a 30-second wall-clock limit and 2 GiB address-space limit. Three expensive
tasks (expression-add-operators, burst-balloons, and
arithmetic-slices-ii-subsequence) allow 900 seconds per solution to keep all 128
repetitions. Verifier timeouts cover all reference budgets plus 60 seconds, with
a minimum of 30 minutes; verifiers use one CPU and 4 GiB memory. Full efficiency
evaluation, especially reference collection for these tasks, can be slow.
Only references passing every test enter the timing distribution. The efficiency
score is clamp((slowest_reference - candidate) / (slowest_reference - fastest_reference), 0, 1).
Equal reference runtimes yield 1 iff the candidate is at least as fast. Percentile
uses the fraction of slower references plus half the ties. Incorrect submissions
score zero. Missing reference baselines are recorded as invalid efficiency, never
as a measured zero without an accompanying validity indicator.
The dataset metric averages correctness and efficiency over all trials,
including failures and missing results. Use one trial per task for Pass@1 and
Beyond@1. efficiency_coverage exposes missing baselines; percentiles are averaged
only over valid efficiency results.
The scoring formula and 128-repeat CPU-time protocol follow Mercury_Eval. This adapter fixes in-place verification by checking the arguments actually passed to the candidate, instead of the unmodified originals. It includes failures in the dataset efficiency denominator, avoids upstream cross-machine reference caches, omits per-iteration wall-clock instrumentation from the CPU timing loop, and uses an explicit result file and resource limits. Standard LeetCode node constructors accept optional values/children, and inputs are reconstructed before each repetition to avoid recursive deepcopy failures on long linked lists. Additional standard-library module imports are available in addition to the original wildcard prelude.
Consequently, this is the mercury-harbor-v1 protocol, not a claim of numerically identical paper/LeetCode/Mercury_Eval results. CPU measurements remain hardware and noise dependent. Run one concurrent trial on a dedicated CPU for reported scores. The original test suite is finite and may accept incorrect programs; no new hidden cases are added in this port. Subprocess limits are not a security sandbox: run agent submissions only inside the disposable Harbor verifier container.
Requires Harbor 0.23.0 or later and a supported container backend (Docker by default).
cd mercury-eval
harbor run -c local-eval.yaml -p . -a oracle
harbor run -c local-eval.yaml -p . -a nop
harbor run -c local-eval.yaml -p . -a codex -m "<model>"The oracle installs a validated original reference solution. Its correctness should
be 1; its efficiency is measured and need not be 1. The no-op starter raises
NotImplementedError and must receive zero correctness and efficiency.
For local paths, local-eval.yaml explicitly loads the custom aggregation script;
Harbor 0.23 does not automatically load a local directory's metric.py.
After publishing, use harbor run -d <your-namespace>/mercury-eval -a <agent> -m <model> -n 1;
Harbor Hub packages automatically load the included metric.
The exact source revision, task digests, oracle selections, and image digest are
recorded in provenance.json and dataset.toml.
@inproceedings{du2024mercury,
title={Mercury: A code efficiency benchmark for code large language models},
author={Du, Mingzhe and Luu, Anh Tuan and Ji, Bin and Liu, Qian and Ng, See-Kiong},
booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2024}
}