Hypernym prompt comparison — native Harbor tasks

This package compares the complete published Hypernym prompt, including its hints, with the frozen public control prompt for four SWE-bench Verified problems. The control is the repository's public recipe, including declared failing-test names. Results are not comparable to issue-only SWE-bench Verified resolve rates. Each pair uses the same repository, reference solution, tests and resource limits. Only the instruction bytes and their identifying metadata differ.

Current status: all eight native tasks and the dataset manifest exist. All eight passed oracle, no-op and both archived regrades: 32 reference checks. qualification.json records each trial, named-test outcomes and file hashes. All eight original subscription attempts are complete: control solved 4/4; Hypernym treatment solved 2/4, with eight valid grades and no harness errors. See RESULTS.md and the complete certified jobs in live-archives.json. This four-task, one-attempt-per-arm pilot is descriptive and does not isolate compression.

The four source problems are astropy__astropy-13398, pytest-dev__pytest-5787, pytest-dev__pytest-6197 and sphinx-doc__sphinx-7590. Source revisions, prompt hashes, canonical test lists, image digests and grader build inputs are recorded in inputs.lock.json. Its first_task_prequalification block is historical; current qualified helper hashes and task identities are in qualification.json and the native dataset manifest. The source material comes from Hypernym's experiment repository.

What runs where

Harbor controls Docker containers, agent installation, task execution, grading, saved jobs and regrading. The task files follow Harbor's native schema. The verifier starts in a separate container and actually applies the saved candidate patch to its trusted base before running the pinned official scorer. The verifier's native Docker Compose definition sets network_mode: none; the grader checks that only loopback is up before executing candidate code. Harbor's dynamic policy stays at its default because its nftables probe fails on this Mac's Docker kernel. Static Docker isolation supplies the restriction.

The selected agent is Codex CLI 0.153.4, model gpt-6-astra, reasoning effort xhigh, authenticated by an existing ChatGPT login. This uses the account's Codex subscription allowance. It makes model requests to OpenAI; the model is not running on the Mac. The helper refuses API keys, proxy/provider overrides, resumed sessions and modified prompt bytes. Subscription availability and model availability are prerequisites for new agent attempts. Archived patch regrading needs neither a subscription nor an available model.

Docker runs locally. This Mac can build and start the pinned Linux x86_64 images under emulation, and all eight tasks passed reference qualification. A compatible Linux Docker host can use the same definitions. A cloud account is not required.

The supplied Hypernym prompts exceed Linux's single-argument size limit. codex_stdin.py retains Harbor's installed Codex adapter and sends the exact instruction through stdin from /tmp/harbor-prompt/instruction.md. It also recovers original line endings from the packaged file after checking Harbor's normalized input against that same file, preserving the frozen prompt bytes. accept_archive.py validates and copies saved jobs before native regrade or export. These are the package's two compatibility helpers; Harbor remains the runner.

Pinned Harbor installation

Use Harbor commit 547e7d606d42c5ccc01ffdeee4fac0b0e3bbe8aa and its own lock:

git clone https://github.com/harbor-framework/harbor.git harbor-pinned
git -C harbor-pinned checkout 547e7d606d42c5ccc01ffdeee4fac0b0e3bbe8aa
cd harbor-pinned
uv sync --frozen --no-default-groups --python 3.13.12
export PATH="$PWD/.venv/bin:$PATH"

The commands below run from this package directory. Docker and Docker Compose must be available in the client process environment (docker info and docker compose version must succeed). A fresh Docker configuration may need Docker Desktop's CLI plugin directory configured; no developer Docker credentials are required for these public pinned images. Clients do not need Forge or any Forge service. The download section below uses Harbor's organization dataset and pins the returned full content hash for replay. After a registry download, run the timestamp refresh in that section before starting any Docker task or regrade.

Reference checks and saved-job replay

These commands use no model account. They reproduce the first task's completed qualification procedure:

harbor run -p ./pytest-dev__pytest-6197-control -a oracle -e docker -n 1 -k 1 -r 0 --job-name pytest6197-oracle
harbor run -p ./pytest-dev__pytest-6197-control -a nop -e docker -n 1 -k 1 -r 0 --job-name pytest6197-nop
mkdir -p ./accepted
python accept_archive.py ./jobs/pytest6197-oracle ./accepted/oracle
python accept_archive.py ./jobs/pytest6197-nop ./accepted/nop
harbor job regrade ./accepted/oracle/job -p ./pytest-dev__pytest-6197-control -e docker -n 1
harbor job regrade ./accepted/nop/job -p ./pytest-dev__pytest-6197-control -e docker -n 1

The reference repair must score 1. The unchanged repository must score 0 because the named repair tests fail, with the named existing tests still passing. Both regrades must reproduce their source verdicts and named-test outcomes. A missing capture, rejected patch or unparseable test output is an error, not a valid zero. Missing named tests are recorded as qualification failures; valid evaluations retain the pinned official scorer's reward. Inspect each native trial's verifier/application.json, verifier/named-tests.json, verifier/report.json and verifier/qualification.json.

The package also includes the 16 accepted reference jobs as reference-<variant>-<oracle|nop>.tar.gz. Their archive and contained-file hashes are in reference-archives.json; together they occupy about 355 KB compressed. Reproduce both saved outcomes for every variant without running an agent:

mkdir -p ./accepted
for task in ./*/task.toml; do
  variant=$(basename "$(dirname "$task")")
  for agent in oracle nop; do
    name="reference-$variant-$agent"
    python accept_archive.py "./$name.tar.gz" "./accepted/$name" || exit 1
    harbor job regrade "./accepted/$name/job" -p "./$variant" \
      -e docker -n 1 --job-name "$name-regrade" || exit 1
    jq -e '.exception_info == null' "jobs/$name-regrade/"*/result.json || exit 1
    for file in reward.txt named-tests.json qualification.json status-map.json report.json; do
      cmp "./accepted/$name/job/"*/verifier/"$file" \
        "./jobs/$name-regrade/"*/verifier/"$file" || exit 1
    done
  done
done

Use a clean package directory and new destination/job names for another replay. The archive checker rejects symlinks in source and destination paths; on macOS, use the physical directory reported by pwd -P when providing absolute paths.

Subscription agent profile

The maintainer's live pilot is separately gated and has exactly eight original attempts: one per variant, one at a time, with no automatic retries. The following is the invocation shape for an authorized slot. Do not add a connectivity smoke or replacement attempt to that pilot:

export CODEX_AUTH_JSON_PATH="$HOME/.codex/auth.json"
PYTHONPATH=. harbor run -p ./pytest-dev__pytest-6197-control \
  -a codex_stdin:CodexStdin -m gpt-6-astra \
  --ak version=0.153.4 --ak reasoning_effort=xhigh --ak config=./codex.toml \
  -e docker -n 1 -k 1 -r 0 --job-name authorized-slot-name

Use a clean process environment with no model API or proxy overrides. The auth file must contain the existing ChatGPT session; it is runtime input and must not be added to the package, Docker image, dataset or saved-job export.

Adapter-owned raw CLI output and sessions are filtered using the actual initial and refreshed credential values before normal log persistence. The complete job is scanned when the native CLI process exits, with those values held only in memory. Live-job export requires the adjacent accepted credential-scan certificate and exact file hashes. Interrupted or failed scanning cannot authorize an export. Harbor's native host mounts still permit arbitrary agent writes; the guarantee covers the adapter's capture and certified exported bytes. It does not claim to prevent arbitrary or encoded exfiltration by a malicious agent.

Accept a known live job with its process-exit certificate before sharing or regrading it. Use a new destination each time:

mkdir -p ./accepted
python accept_archive.py ./jobs/authorized-slot-name ./accepted/authorized-slot-name \
  --require-credential-scan \
  --credential-scan ./jobs/authorized-slot-name.credential-scan.json
harbor job regrade ./accepted/authorized-slot-name/job \
  -p ./pytest-dev__pytest-6197-control -e docker -n 1 --job-name replayed-slot
python accept_archive.py ./jobs/replayed-slot ./accepted/replayed-slot \
  --require-credential-scan --source-acceptance ./accepted/authorized-slot-name/acceptance.json

On authentication or quota failure, stop before another invocation and preserve the original failed attempt. Report unresolved tasks and invalid grades separately. The pilot has no required performance threshold.

Observed parser limitation

A diagnostic patch printed fabricated PASSED lines for the two pytest-6197 repair tests. The pinned upstream log parser accepted those statuses, with found_applied=true and no parsed test exit code. This fixture did not execute the canonical tests or produce a benchmark reward, so it does not establish a complete scoring exploit. Clients should retain the patches and raw test logs when inspecting results; the package does not claim resistance to arbitrary adversarial test-output spoofing. The scorer remains pinned and unchanged.

Team CI reference checks

A CI runner with Docker, the pinned Harbor installation and this package can run all eight references without a model login. Preserve the native job folders as CI artifacts. Check results explicitly: Harbor can exit successfully even when a trial records an exception.

harbor run -p . -a oracle -e docker -n 1 -k 1 -r 0 --job-name ci-oracle
harbor run -p . -a nop -e docker -n 1 -k 1 -r 0 --job-name ci-nop
for job in ci-oracle ci-nop; do
  jq -e '.n_total_trials == 8 and .stats.n_completed_trials == 8 and .stats.n_errored_trials == 0 and .stats.n_retries == 0' "jobs/$job/result.json"
done
jq -se 'length == 8 and all(.[]; .all_fail_to_pass_present and .all_pass_to_pass_present and .all_pass_to_pass_observed_passing and .all_fail_to_pass_observed_passing)' jobs/ci-oracle/*/verifier/qualification.json
jq -se 'length == 8 and all(.[]; .all_fail_to_pass_present and .all_pass_to_pass_present and .all_pass_to_pass_observed_passing and .all_fail_to_pass_observed_failing)' jobs/ci-nop/*/verifier/qualification.json

Use a fresh job name on each CI invocation. Regrade accepted saved jobs using the reference commands above to verify changes to the verifier independently of a model run. After intentionally editing task or dataset-level files, run harbor sync to refresh the native manifest digests before publication.

Download the organization dataset

The public dataset is hypernymai/hypernym-swebench-artifact-replay. After installing the pinned Harbor version, obtain the full content hash and download that exact version into an empty directory:

harbor version show hypernymai/hypernym-swebench-artifact-replay@v1.0.2 --files --tasks --json > release-version.json
DATASET_DIGEST=$(jq -r '.content_hash' release-version.json)
DATASET_DIGEST=${DATASET_DIGEST#sha256:}
harbor download "hypernymai/hypernym-swebench-artifact-replay@sha256:$DATASET_DIGEST" --export -o received
cd received/hypernym-swebench-artifact-replay
python - <<'PY'
from pathlib import Path
import os
import time

files = sorted(
    path
    for task in Path('.').iterdir()
    if task.is_dir() and (task / 'task.toml').is_file()
    for folder in ('environment', 'tests')
    for path in (task / folder).rglob('*')
    if path.is_file()
)
stamp = time.time_ns()
for index, path in enumerate(files):
    timestamp = stamp + index * 1000
    os.utime(path, ns=(timestamp, timestamp))
print(f'Refreshed timestamps on {len(files)} Docker context files; contents unchanged.')
PY

Save release-version.json with your results. The full content hash pins the version; a tag can move. Export includes the eight task folders and the listed dataset files. It does not create dataset.toml; local execution discovers the task folders directly. Public download and reference/regrade commands need no Harbor login or model login.

Run the timestamp refresh after each new registry extraction, before building containers. On the tested Mac, the pinned downloader extracted files with zero timestamps and Docker reused another task's copied base metadata. The verifier rejected those trials instead of assigning a score. Refreshing timestamps forced the correct copy in a subsequent oracle check. This step changes no file contents, task digests, prompts or tests. Version v1.0.1 added the timestamp instructions; v1.0.2 also corrects the accepted live-archive path below. Both change only this README. Their eight task definitions and original results match v1.0.0.

Original paired pilot and replay

RESULTS.md, results.csv and results.json describe all eight original attempts. live-archives.json maps each variant to its complete native job archive, credential-scan certificate and file hashes. These archives preserve the original patch, trajectory, prompt receipt, verifier logs and grade. A valid score of zero is an unresolved task. Regrades are separate observations and do not replace original scores.

For example, accept and regrade the first original job using its certificate:

mkdir -p accepted
python accept_archive.py ./live-slot-01-pytest-dev__pytest-6197-control.tar.gz ./accepted/live-slot-01 \
  --require-credential-scan \
  --credential-scan ./live-slot-01-pytest-dev__pytest-6197-control.credential-scan.json
LIVE_JOB=$(python -c 'import json; print(json.load(open("accepted/live-slot-01/acceptance.json"))["job_path"])')
harbor job regrade "$LIVE_JOB" \
  -p ./pytest-dev__pytest-6197-control -e docker -n 1 --job-name replay-live-slot-01
jq -e '.exception_info == null' jobs/replay-live-slot-01/*/result.json
for file in reward.txt named-tests.json qualification.json status-map.json report.json; do
  cmp "$LIVE_JOB/"*/verifier/"$file" jobs/replay-live-slot-01/*/verifier/"$file" || exit 1
done

Repeat with the variant, archive and certificate listed for each slot in live-archives.json, always using fresh destination and job names. No model session or new agent attempt is involved. Use the returned job_path: the helper preserves an archive's optional wrapper directory, so accepted live archives have a different depth from directory inputs.

Browse with Harbor's viewer

harbor view . --tasks
harbor view ./jobs --jobs

Open the local URL printed by Harbor. The task view exposes the eight definitions; the jobs view exposes trial identities, rewards, trajectories and verifier logs. Run these commands in separate terminals if using both views. A source checkout automatically builds the viewer on first use and requires Bun; the clean-client build used Bun 1.3.11 with Harbor's checked-in viewer lock unchanged.

To inspect one accepted original job directly, use its containing directory: harbor view "$(dirname "$LIVE_JOB")" --jobs.