Loading dataset details…
harbor run -d ykhli/tetrisbenchharbor run -d ykhli/tetrisbenchTetrisBench strategy track: submit one JS placement-scoring function; mean raw game score over 20 fixed public seeds x 500 pieces, separately verified.
harbor run -d ykhli/tetrisbenchTetrisBench strategy track: submit one JS placement-scoring function; mean raw game score over 20 fixed public seeds x 500 pieces, separately verified.
harbor run -d ykhli/tetrisbenchPrivate preview: strategy-generation track. A coding agent writes and improves a Tetris placement-scoring function. A clean, deterministic verifier evaluates the final submission across the same 20 piece sequences. No TetrisBench website account, Supabase connection, or benchmark-owner API key is needed.
This is a versioned adaptation of TetrisBench, not a reproduction of its historical head-to-head leaderboard. See PROVENANCE.md for preserved rules and intentional changes.
The agent has 15 minutes to implement a JavaScript scoring-function body in /app/score.js. It can run practice games, inspect the engine, and revise its code. The trusted harness enumerates reachable placements and chooses the one with the highest returned value.
The final submission is graded on 20 fixed seeds, up to 500 pieces each. A run therefore evaluates one authored strategy, not 20 separately authored strategies.
Primary score: arithmetic mean of the original game's points across all 20 seeds. Higher is better. This is a continuous, unnormalized reward, not a pass percentage. Line clears, pieces placed, and per-seed outcomes are supplementary diagnostics. Invalid or missing policies score zero; failed seeds cannot be discarded to improve the average.
This measures an agent and its configuration, not an isolated base model. Publish the agent version, exact model ID, reasoning settings, development budget, benchmark digest, and all attempts alongside scores.
Requirements: Node.js 24, Docker, and Harbor 0.23.0. A model provider key is needed only for actual model-agent runs, not local strategy evaluation, the reference run, or the no-op check.
npm ci
npm test
node scripts/evaluate.mjs --policy reference/score.js --seeds 1,2,3 --pieces 500The reference policy is the original project's simple built-in heuristic. It is a plumbing/sanity baseline, not an optimal player or a model result.
The generated task directory contains self-contained agent and verifier Docker build contexts. Rebuild them after changing the source, then verify the package before running:
node scripts/build-harbor.mjs
uvx --from harbor==0.23.0 harbor run -p tasks/strategy -a oracle -n 1
uvx --from harbor==0.23.0 harbor run -p tasks/strategy -a nop -n 1Expected: the reference run produces a positive game score, and no-op produces zero. This benchmark does not use the binary oracle convention of reward exactly 1.
To evaluate a coding agent, supply its own provider credentials through your normal Harbor configuration:
uvx --from harbor==0.23.0 harbor run \
-p tasks/strategy \
-a claude-code -m '<supported-model-id>' \
-k 3 -n 1-k 3 requests three independent strategy-development attempts. Each is graded on the same 20 seeds. Report all attempts, not only the best. Do not combine their 60 seed results as though they were 60 independently generated strategies.
These commands run locally and do not upload results. Uploading a job, publishing a task, and publishing a dataset are separate explicit actions.
If an agent reaches its development time limit, Harbor can still grade the last saved policy. Report the AgentTimeoutError alongside that score; do not describe it as a cleanly completed agent run. A timeout or invalid value inside policy evaluation instead invalidates the whole submission.
The submission is a function body, with no exports, imports, function wrapper, or Markdown fences:
return linesCleared * 1.0
- metrics.aggregateHeight * 0.5
- metrics.holes * 1.5
- metrics.bumpiness * 0.4
- metrics.maxHeight * 0.6
- landingHeight * 0.2;Inputs:
| Name | Meaning |
|---|---|
board |
20 by 10 binary board after placing the piece and clearing complete rows |
metrics |
heights, aggregateHeight, maxHeight, holes, bumpiness, rowTransitions, columnTransitions, wells |
linesCleared |
Lines cleared by this placement, 0 to 4 |
landingHeight |
Board height minus the vertical center of the placed piece's bounding box before clearing rows |
pieceType |
I, O, T, S, Z, J, or L |
Return a finite JavaScript number. The highest-scoring candidate wins; exact ties use the first candidate in deterministic enumeration order. Each invocation is isolated, so globals and mutations do not persist or alter other candidates or game state. Host access and nondeterministic time/random functions are unavailable. Resource-limit failures invalidate the submission rather than invoking a fallback.
The agent can modify its practice copy of the engine, but that does not change grading. Harbor transfers the submitted score function to a separate verifier container built from trusted code. The verifier ignores agent-authored reward files, scores, game logs, and package files. It recomputes every game using the pinned engine and fixed protocol.
The policy itself executes in QuickJS/WASM, not Node eval, Function, or the Node vm module. The VM receives only the placement inputs, with memory and execution limits. The verifier does not have model-provider credentials and does not need network access at runtime.
The seeds are generated deterministically from a documented release string and are fixed before model testing. They are not hand-selected for a model or the reference policy.
This is a public, inspectable suite, not a secret holdout. Evaluation seeds are not copied into the agent's practice image, but they remain visible in the distributed task's verifier and source. Do not claim contamination resistance or private-test generalization. A higher-stakes leaderboard should also audit submitted policies on a separately committed, unreleased seed set.
The scoring-function interface does not explicitly expose the seed or future sequence, but the board itself can identify a known trajectory. Statelessness is not a defense against public-seed memorization. Publish the submitted policy source with any claimed score, and use an unreleased seed set for generalization claims.
Private publication was authorized on 2026-09-24 under the author's Harbor namespace, ykhli. The intended identifiers are ykhli/tetrisbench (dataset) and ykhli/tetrisbench-strategy (task). See RELEASE.md for the validation checklist and public-visibility approval gate. Nothing in this repository publishes automatically.
A private dataset is accessible only to authorized accounts. Once the owner explicitly approves making both packages public, others will be able to run the dataset without access to the original website or private source repository. Hosted Harbor compute access is separate from registry access; downloading a benchmark and running it locally does not require hosted-compute entitlement.
Private preview: strategy-generation track. A coding agent writes and improves a Tetris placement-scoring function. A clean, deterministic verifier evaluates the final submission across the same 20 piece sequences. No TetrisBench website account, Supabase connection, or benchmark-owner API key is needed.
This is a versioned adaptation of TetrisBench, not a reproduction of its historical head-to-head leaderboard. See PROVENANCE.md for preserved rules and intentional changes.
The agent has 15 minutes to implement a JavaScript scoring-function body in /app/score.js. It can run practice games, inspect the engine, and revise its code. The trusted harness enumerates reachable placements and chooses the one with the highest returned value.
The final submission is graded on 20 fixed seeds, up to 500 pieces each. A run therefore evaluates one authored strategy, not 20 separately authored strategies.
Primary score: arithmetic mean of the original game's points across all 20 seeds. Higher is better. This is a continuous, unnormalized reward, not a pass percentage. Line clears, pieces placed, and per-seed outcomes are supplementary diagnostics. Invalid or missing policies score zero; failed seeds cannot be discarded to improve the average.
This measures an agent and its configuration, not an isolated base model. Publish the agent version, exact model ID, reasoning settings, development budget, benchmark digest, and all attempts alongside scores.
Requirements: Node.js 24, Docker, and Harbor 0.23.0. A model provider key is needed only for actual model-agent runs, not local strategy evaluation, the reference run, or the no-op check.
npm ci
npm test
node scripts/evaluate.mjs --policy reference/score.js --seeds 1,2,3 --pieces 500The reference policy is the original project's simple built-in heuristic. It is a plumbing/sanity baseline, not an optimal player or a model result.
The generated task directory contains self-contained agent and verifier Docker build contexts. Rebuild them after changing the source, then verify the package before running:
node scripts/build-harbor.mjs
uvx --from harbor==0.23.0 harbor run -p tasks/strategy -a oracle -n 1
uvx --from harbor==0.23.0 harbor run -p tasks/strategy -a nop -n 1Expected: the reference run produces a positive game score, and no-op produces zero. This benchmark does not use the binary oracle convention of reward exactly 1.
To evaluate a coding agent, supply its own provider credentials through your normal Harbor configuration:
uvx --from harbor==0.23.0 harbor run \
-p tasks/strategy \
-a claude-code -m '<supported-model-id>' \
-k 3 -n 1-k 3 requests three independent strategy-development attempts. Each is graded on the same 20 seeds. Report all attempts, not only the best. Do not combine their 60 seed results as though they were 60 independently generated strategies.
These commands run locally and do not upload results. Uploading a job, publishing a task, and publishing a dataset are separate explicit actions.
If an agent reaches its development time limit, Harbor can still grade the last saved policy. Report the AgentTimeoutError alongside that score; do not describe it as a cleanly completed agent run. A timeout or invalid value inside policy evaluation instead invalidates the whole submission.
The submission is a function body, with no exports, imports, function wrapper, or Markdown fences:
return linesCleared * 1.0
- metrics.aggregateHeight * 0.5
- metrics.holes * 1.5
- metrics.bumpiness * 0.4
- metrics.maxHeight * 0.6
- landingHeight * 0.2;Inputs:
| Name | Meaning |
|---|---|
board |
20 by 10 binary board after placing the piece and clearing complete rows |
metrics |
heights, aggregateHeight, maxHeight, holes, bumpiness, rowTransitions, columnTransitions, wells |
linesCleared |
Lines cleared by this placement, 0 to 4 |
landingHeight |
Board height minus the vertical center of the placed piece's bounding box before clearing rows |
pieceType |
I, O, T, S, Z, J, or L |
Return a finite JavaScript number. The highest-scoring candidate wins; exact ties use the first candidate in deterministic enumeration order. Each invocation is isolated, so globals and mutations do not persist or alter other candidates or game state. Host access and nondeterministic time/random functions are unavailable. Resource-limit failures invalidate the submission rather than invoking a fallback.
The agent can modify its practice copy of the engine, but that does not change grading. Harbor transfers the submitted score function to a separate verifier container built from trusted code. The verifier ignores agent-authored reward files, scores, game logs, and package files. It recomputes every game using the pinned engine and fixed protocol.
The policy itself executes in QuickJS/WASM, not Node eval, Function, or the Node vm module. The VM receives only the placement inputs, with memory and execution limits. The verifier does not have model-provider credentials and does not need network access at runtime.
The seeds are generated deterministically from a documented release string and are fixed before model testing. They are not hand-selected for a model or the reference policy.
This is a public, inspectable suite, not a secret holdout. Evaluation seeds are not copied into the agent's practice image, but they remain visible in the distributed task's verifier and source. Do not claim contamination resistance or private-test generalization. A higher-stakes leaderboard should also audit submitted policies on a separately committed, unreleased seed set.
The scoring-function interface does not explicitly expose the seed or future sequence, but the board itself can identify a known trajectory. Statelessness is not a defense against public-seed memorization. Publish the submitted policy source with any claimed score, and use an unreleased seed set for generalization claims.
Private publication was authorized on 2026-09-24 under the author's Harbor namespace, ykhli. The intended identifiers are ykhli/tetrisbench (dataset) and ykhli/tetrisbench-strategy (task). See RELEASE.md for the validation checklist and public-visibility approval gate. Nothing in this repository publishes automatically.
A private dataset is accessible only to authorized accounts. Once the owner explicitly approves making both packages public, others will be able to run the dataset without access to the original website or private source repository. Hosted Harbor compute access is separate from registry access; downloading a benchmark and running it locally does not require hosted-compute entitlement.