Loading dataset details…
harbor run -d svgap/svgap-reset-releaseharbor run -d svgap/svgap-reset-releaseSome RTL agents pass every test but still wire reset incorrectly. Eight SystemVerilog tasks expose that hidden failure.
harbor run -d svgap/svgap-reset-releaseSome RTL agents pass every test but still wire reset incorrectly. Eight SystemVerilog tasks expose that hidden failure.
harbor run -d svgap/svgap-reset-releaseMost coding-agent evals ask one question: did the generated program pass its tests? For chip-design code, that can miss an important class of mistakes. A design can behave correctly in the supplied simulation while being wired in a way that violates a hardware rule in the prompt.
This dataset turns that possibility into a small, reproducible experiment. Each of its eight SystemVerilog tasks asks an agent to build ordinary control logic such as a counter, timer, or state machine. The task also says how reset must be released. Reset puts circuit state into a known starting value. Its release must reach ordinary state through the requested clock-aligned path.
The evaluator asks two questions about the same generated design:
The interesting case is a design that passes question 1 and fails question 2.
Harbor records that result as gap_member = 1.
You do not need to be a reset-domain expert to try the experiment. If you work on agent evaluation, the central research question is familiar: what important requirements remain invisible when an eval stops at test success?
You need Harbor and a running Docker daemon. The built-in Oracle installs the known-good answer for each task, which lets you check the complete pipeline before connecting a model:
harbor run \
-d svgap/svgap-reset-release@0.2 \
-a oracle \
-o jobs/svgap-oracle \
--n-concurrent 1All eight tasks should receive reward = 1. The evaluator image already
contains the open-source simulation and circuit-analysis tools, so the host
does not need an EDA installation.
Use any agent supported by Harbor:
harbor run \
-d svgap/svgap-reset-release@0.2 \
-a YOUR_AGENT \
-m YOUR_MODEL \
-o jobs/svgap-agent \
--n-concurrent 1 \
--n-attempts 1Then inspect the job:
harbor view jobs/svgap-agentThe verifier compiles and simulates generated RTL. Treat that code as untrusted input and run it on an isolated host. The first run may take longer while Harbor prepares the agent environment.
One complete Codex gpt-5.5 run produced:
| Result | Tasks |
|---|---|
| Passed the simulation tests | 7 / 8 |
| Passed both the tests and the wiring rule | 5 / 8 |
| Passed the tests but failed the wiring rule | 2 / 8 |
| Unanswered checks or tool failures | 0 / 8 |
The two revealing cases were reset-counter and reset-credits. Both passed
their simulations. Both connected reset directly to state that the prompt
required to resume through a clock-aligned release path.
Explore the readable evidence or inspect the content-addressed reports.
This is one run, not a model ranking or an estimate of how often the problem occurs. We are publishing it because the two failures are concrete examples of a broader eval question worth testing across more agents and task families.
| Harbor metric | Plain-language meaning |
|---|---|
functional_accept |
The simulation tests passed. |
structural_accept |
The generated circuit obeyed the stated wiring rule. |
gap_member |
Tests passed, but the wiring rule failed on the same design. |
reward |
Both checks passed. |
unknown |
At least one configured question was not answered. |
tool_error |
An evidence-producing tool failed. |
An unanswered question or tool failure never becomes a pass. Each trial keeps the candidate RTL, the full SV-Gap JSON report, the Harbor verdict, and a readable HTML evidence page. The number is a convenient summary; the retained evidence shows why the design received it.
We would like collaborators to run different agents, examine surprising cases, challenge the rule, and propose tasks where test success may conceal a different hardware mistake. A complete Harbor job can become a reviewable SV-Gap contribution without cloning this repository.
The smallest useful public contribution is a two-minute run report. Completed runs, partial runs, setup blockers, counterexamples, and disagreement with the rule are all useful:
Report what happened or follow the researcher run guide.
Install SV-Gap 0.3.0 alpha 8 or newer, then point the importer at the job directory Harbor created:
pip install "svgap>=0.3.0a8"
svgap submission from-harbor jobs/svgap-agent/JOB_DIRECTORY \
--dataset svgap/svgap-reset-release@0.2 \
--id YOUR-SUBMISSION-ID \
--title "YOUR TITLE" \
--provenance-level public \
--provider YOUR_PROVIDER \
--contributor "YOUR NAME" \
--output results/submissions/YOUR-SUBMISSION-IDThe importer checks that the job is complete and that its task versions, reports, verdicts, and numeric scores agree. Open a pull request containing the resulting directory. Harbor provides the common execution surface; the SV-Gap repository keeps the reviewed evidence and the limits on what each result can support. Uploading a job to Harbor Hub is optional and is not required to contribute.
Other useful contributions include counterexamples, critiques of the reset rule, new task ideas, and reviews of the open evaluation method. See how to submit results or start a discussion in the SV-Gap repository.
See the current public research pulse, including the distinction between author-run evidence and independent replication.
This dataset is intentionally narrow: eight digital RTL tasks and one reset rule. It does not claim silicon signoff, certification, comprehensive CDC or RDC coverage, or hardware safety. Its purpose is to make one blind spot of test-only evaluation visible, inspectable, and open to challenge.
The tasks are reset-config, reset-counter, reset-credits, reset-events,
reset-fsm, reset-status, reset-timer, and reset-watchdog.
Most coding-agent evals ask one question: did the generated program pass its tests? For chip-design code, that can miss an important class of mistakes. A design can behave correctly in the supplied simulation while being wired in a way that violates a hardware rule in the prompt.
This dataset turns that possibility into a small, reproducible experiment. Each of its eight SystemVerilog tasks asks an agent to build ordinary control logic such as a counter, timer, or state machine. The task also says how reset must be released. Reset puts circuit state into a known starting value. Its release must reach ordinary state through the requested clock-aligned path.
The evaluator asks two questions about the same generated design:
The interesting case is a design that passes question 1 and fails question 2.
Harbor records that result as gap_member = 1.
You do not need to be a reset-domain expert to try the experiment. If you work on agent evaluation, the central research question is familiar: what important requirements remain invisible when an eval stops at test success?
You need Harbor and a running Docker daemon. The built-in Oracle installs the known-good answer for each task, which lets you check the complete pipeline before connecting a model:
harbor run \
-d svgap/svgap-reset-release@0.2 \
-a oracle \
-o jobs/svgap-oracle \
--n-concurrent 1All eight tasks should receive reward = 1. The evaluator image already
contains the open-source simulation and circuit-analysis tools, so the host
does not need an EDA installation.
Use any agent supported by Harbor:
harbor run \
-d svgap/svgap-reset-release@0.2 \
-a YOUR_AGENT \
-m YOUR_MODEL \
-o jobs/svgap-agent \
--n-concurrent 1 \
--n-attempts 1Then inspect the job:
harbor view jobs/svgap-agentThe verifier compiles and simulates generated RTL. Treat that code as untrusted input and run it on an isolated host. The first run may take longer while Harbor prepares the agent environment.
One complete Codex gpt-5.5 run produced:
| Result | Tasks |
|---|---|
| Passed the simulation tests | 7 / 8 |
| Passed both the tests and the wiring rule | 5 / 8 |
| Passed the tests but failed the wiring rule | 2 / 8 |
| Unanswered checks or tool failures | 0 / 8 |
The two revealing cases were reset-counter and reset-credits. Both passed
their simulations. Both connected reset directly to state that the prompt
required to resume through a clock-aligned release path.
Explore the readable evidence or inspect the content-addressed reports.
This is one run, not a model ranking or an estimate of how often the problem occurs. We are publishing it because the two failures are concrete examples of a broader eval question worth testing across more agents and task families.
| Harbor metric | Plain-language meaning |
|---|---|
functional_accept |
The simulation tests passed. |
structural_accept |
The generated circuit obeyed the stated wiring rule. |
gap_member |
Tests passed, but the wiring rule failed on the same design. |
reward |
Both checks passed. |
unknown |
At least one configured question was not answered. |
tool_error |
An evidence-producing tool failed. |
An unanswered question or tool failure never becomes a pass. Each trial keeps the candidate RTL, the full SV-Gap JSON report, the Harbor verdict, and a readable HTML evidence page. The number is a convenient summary; the retained evidence shows why the design received it.
We would like collaborators to run different agents, examine surprising cases, challenge the rule, and propose tasks where test success may conceal a different hardware mistake. A complete Harbor job can become a reviewable SV-Gap contribution without cloning this repository.
The smallest useful public contribution is a two-minute run report. Completed runs, partial runs, setup blockers, counterexamples, and disagreement with the rule are all useful:
Report what happened or follow the researcher run guide.
Install SV-Gap 0.3.0 alpha 8 or newer, then point the importer at the job directory Harbor created:
pip install "svgap>=0.3.0a8"
svgap submission from-harbor jobs/svgap-agent/JOB_DIRECTORY \
--dataset svgap/svgap-reset-release@0.2 \
--id YOUR-SUBMISSION-ID \
--title "YOUR TITLE" \
--provenance-level public \
--provider YOUR_PROVIDER \
--contributor "YOUR NAME" \
--output results/submissions/YOUR-SUBMISSION-IDThe importer checks that the job is complete and that its task versions, reports, verdicts, and numeric scores agree. Open a pull request containing the resulting directory. Harbor provides the common execution surface; the SV-Gap repository keeps the reviewed evidence and the limits on what each result can support. Uploading a job to Harbor Hub is optional and is not required to contribute.
Other useful contributions include counterexamples, critiques of the reset rule, new task ideas, and reviews of the open evaluation method. See how to submit results or start a discussion in the SV-Gap repository.
See the current public research pulse, including the distinction between author-run evidence and independent replication.
This dataset is intentionally narrow: eight digital RTL tasks and one reset rule. It does not claim silicon signoff, certification, comprehensive CDC or RDC coverage, or hardware safety. Its purpose is to make one blind spot of test-only evaluation visible, inspectable, and open to challenge.
The tasks are reset-config, reset-counter, reset-credits, reset-events,
reset-fsm, reset-status, reset-timer, and reset-watchdog.