Loading dataset details…
harbor run -d fritzprix/libragent-diverse-9harbor run -d fritzprix/libragent-diverse-9Stratified 9-task smoke suite from Terminal-Bench 2.1 for LibrAgent harness eval. Difficulty mix: 2 easy / 4 medium / 3 hard across 7 categories (SE, debugging, sysadmin, security, data-science, scientific-computing, data-querying). Disjoint from NovitaAI/tb21-file-recovery. See README for sampling rationale and per-task notes.
harbor run -d fritzprix/libragent-diverse-9Stratified 9-task smoke suite from Terminal-Bench 2.1 for LibrAgent harness eval. Difficulty mix: 2 easy / 4 medium / 3 hard across 7 categories (SE, debugging, sysadmin, security, data-science, scientific-computing, data-querying). Disjoint from NovitaAI/tb21-file-recovery. See README for sampling rationale and per-task notes.
harbor run -d fritzprix/libragent-diverse-9Stratified 9-task evaluation slice sampled from
Terminal-Bench 2.1
(terminal-bench/terminal-bench-2-1, 89 tasks).
Hub: https://hub.harborframework.com/datasets/fritzprix/libragent-diverse-9
Tags: v1.0, latest · Visibility: public · Author: fritzprix
This is a curation, not a re-adaptation. Every task is a pinned registry
reference to an existing terminal-bench/<task> package (deterministic
verifiers, same timeouts/resources as upstream). No task archives were
republished under the fritzprix org.
Use this dataset when you need a short, repeatable smoke / harness-regression suite that still stretches an agent across meaningfully different skills.
Typical uses:
pnpm bench:diverse)It is not a substitute for full Terminal-Bench 2.1 or Harbor Index leaderboard runs. Scores on 9 tasks have high variance; treat them as a diagnostic slice, not a ranking claim.
The previous LibrAgent default registry smoke set,
NovitaAI/tb21-file-recovery
(also 9 tasks from TB2.1), is useful but distributionally thin:
| Property | NovitaAI/tb21-file-recovery |
fritzprix/libragent-diverse-9 |
|---|---|---|
| Tasks | 9 | 9 |
| Difficulty | 8× medium, 1× hard | 2× easy, 4× medium, 3× hard |
| Categories | 2 (file-operations, data-processing) |
7 distinct categories |
| Overlap with this set | — | none (disjoint task IDs) |
Repeated runs on that set over-fit local iteration to OCR / log / recovery-style file work. This slice deliberately rebalances difficulty and category so smoke runs exercise a wider agent surface (VCS, LaTeX debug, nginx, secret hygiene, SQL, biology primers, async Python, forensics, SPARQL).
terminal-bench/terminal-bench-2-1 (metadata
from each task’s task.toml: metadata.difficulty, metadata.category).metadata.category values under that
size → 7 categories (SE and security appear twice at different
difficulties).NovitaAI/tb21-file-recovery.dataset.toml stores name + content digest
(sha256:…) so harbor run -d fritzprix/libragent-diverse-9 resolves a
fixed task revision.Selection was manual stratified sampling, not random. Preference went to tasks that (a) fill an empty difficulty×category cell, (b) are widely cited in TB discussions, and (c) keep verifier complexity reasonable for local Docker smoke (no GPU-required tasks in this slice).
| # | Task | Difficulty | Category | Agent timeout | Expert / junior (min) |
|---|---|---|---|---|---|
| 1 | fix-git |
easy | software-engineering | 900s | 5 / 20 |
| 2 | overfull-hbox |
easy | debugging | 750s | 60 / 60 |
| 3 | nginx-request-logging |
medium | system-administration | 900s | 20 / 120 |
| 4 | sanitize-git-repo |
medium | security | 900s | 30 / 60 |
| 5 | query-optimize |
medium | data-science | 900s | 60 / 480 |
| 6 | dna-insert |
medium | scientific-computing | 1800s | 30 / 120 |
| 7 | cancel-async-tasks |
hard | software-engineering | 900s | 120 / 600 |
| 8 | password-recovery |
hard | security | 900s | 100 / 300 |
| 9 | sparql-university |
hard | data-querying | 900s | 800 / 10000 |
Difficulty: 2 easy · 4 medium · 3 hard
Categories: software-engineering ×2, security ×2, debugging, system-administration, data-science, scientific-computing, data-querying
Upstream registry IDs are terminal-bench/<id>. Descriptions below are taken
from each task’s task.toml (trimmed).
fix-git — easy · software-engineeringterminal-bench/fix-gitmaster.overfull-hbox — easy · debuggingterminal-bench/overfull-hboxnginx-request-logging — medium · system-administrationterminal-bench/nginx-request-loggingsanitize-git-repo — medium · securityterminal-bench/sanitize-git-repoquery-optimize — medium · data-scienceterminal-bench/query-optimizedna-insert — medium · scientific-computingterminal-bench/dna-insertcancel-async-tasks — hard · software-engineeringterminal-bench/cancel-async-taskspassword-recovery — hard · securityterminal-bench/password-recoverysanitize-git-repo.sparql-university — hard · data-queryingterminal-bench/sparql-universitydata-querying); hardest expert
estimate in this slice — good stress task for tool + reasoning agents.| Capability cluster | Tasks |
|---|---|
| Version control / repo surgery | fix-git, sanitize-git-repo |
| Document / toolchain debugging | overfull-hbox |
| Service install & config | nginx-request-logging |
| SQL rewriting | query-optimize |
| Scientific domain reasoning | dna-insert |
| Concurrent Python correctness | cancel-async-tasks |
| Forensics / low-level investigation | password-recovery |
| Knowledge-graph querying | sparql-university |
harbor run -d fritzprix/libragent-diverse-9 -a <agent> -m <model>
harbor run -d fritzprix/libragent-diverse-9@v1.0 -a <agent> -m <model> -k 1Pin a digest if you need bit-for-bit reproducibility of the dataset manifest revision shown on Hub after publish.
pnpm bench:diverse # full 9-task suite (default n-attempts=1)
pnpm bench:diverse:n1 # first task only
pnpm bench:registry # same default dataset preset
pnpm xbench:registry # Hermes on the same defaultOfficial-style multi-attempt:
pnpm bench:diverse --n-attempts 5Previous narrow smoke set (still available):
pnpm bench:registry --dataset NovitaAI/tb21-file-recoveryharbor run -p /path/to/libragent-diverse-9 -a <agent> -m <model>| Item | Value |
|---|---|
| Dataset ID | fritzprix/libragent-diverse-9 |
| Initial tag | v1.0 (+ latest) |
| Task ownership | Upstream terminal-bench/* digests only |
| Publish mode | harbor publish … --public --no-tasks |
Bump the dataset tag (e.g. v1.1) when the task list or pinned digests
change. Do not silently retarget latest to a different skill mix if
downstream jobs treat this ID as a fixed contract—document the change here and
in the Hub description.
To refresh digests to newer published task revisions:
cd /path/to/libragent-diverse-9
harbor sync --upgrade # or re-add each terminal-bench/<task>
harbor publish . -t v1.1 --public --no-taskstask.toml
[[task.authors]]).fritzprix (LibrAgent harness smoke design).If you cite results, cite Terminal-Bench for task credit and note that scores are on this 9-task stratified subset, not the full 89-task benchmark.
benchmarks/harbor/README.md (in the LibrAgent repo)Stratified 9-task evaluation slice sampled from
Terminal-Bench 2.1
(terminal-bench/terminal-bench-2-1, 89 tasks).
Hub: https://hub.harborframework.com/datasets/fritzprix/libragent-diverse-9
Tags: v1.0, latest · Visibility: public · Author: fritzprix
This is a curation, not a re-adaptation. Every task is a pinned registry
reference to an existing terminal-bench/<task> package (deterministic
verifiers, same timeouts/resources as upstream). No task archives were
republished under the fritzprix org.
Use this dataset when you need a short, repeatable smoke / harness-regression suite that still stretches an agent across meaningfully different skills.
Typical uses:
pnpm bench:diverse)It is not a substitute for full Terminal-Bench 2.1 or Harbor Index leaderboard runs. Scores on 9 tasks have high variance; treat them as a diagnostic slice, not a ranking claim.
The previous LibrAgent default registry smoke set,
NovitaAI/tb21-file-recovery
(also 9 tasks from TB2.1), is useful but distributionally thin:
| Property | NovitaAI/tb21-file-recovery |
fritzprix/libragent-diverse-9 |
|---|---|---|
| Tasks | 9 | 9 |
| Difficulty | 8× medium, 1× hard | 2× easy, 4× medium, 3× hard |
| Categories | 2 (file-operations, data-processing) |
7 distinct categories |
| Overlap with this set | — | none (disjoint task IDs) |
Repeated runs on that set over-fit local iteration to OCR / log / recovery-style file work. This slice deliberately rebalances difficulty and category so smoke runs exercise a wider agent surface (VCS, LaTeX debug, nginx, secret hygiene, SQL, biology primers, async Python, forensics, SPARQL).
terminal-bench/terminal-bench-2-1 (metadata
from each task’s task.toml: metadata.difficulty, metadata.category).metadata.category values under that
size → 7 categories (SE and security appear twice at different
difficulties).NovitaAI/tb21-file-recovery.dataset.toml stores name + content digest
(sha256:…) so harbor run -d fritzprix/libragent-diverse-9 resolves a
fixed task revision.Selection was manual stratified sampling, not random. Preference went to tasks that (a) fill an empty difficulty×category cell, (b) are widely cited in TB discussions, and (c) keep verifier complexity reasonable for local Docker smoke (no GPU-required tasks in this slice).
| # | Task | Difficulty | Category | Agent timeout | Expert / junior (min) |
|---|---|---|---|---|---|
| 1 | fix-git |
easy | software-engineering | 900s | 5 / 20 |
| 2 | overfull-hbox |
easy | debugging | 750s | 60 / 60 |
| 3 | nginx-request-logging |
medium | system-administration | 900s | 20 / 120 |
| 4 | sanitize-git-repo |
medium | security | 900s | 30 / 60 |
| 5 | query-optimize |
medium | data-science | 900s | 60 / 480 |
| 6 | dna-insert |
medium | scientific-computing | 1800s | 30 / 120 |
| 7 | cancel-async-tasks |
hard | software-engineering | 900s | 120 / 600 |
| 8 | password-recovery |
hard | security | 900s | 100 / 300 |
| 9 | sparql-university |
hard | data-querying | 900s | 800 / 10000 |
Difficulty: 2 easy · 4 medium · 3 hard
Categories: software-engineering ×2, security ×2, debugging, system-administration, data-science, scientific-computing, data-querying
Upstream registry IDs are terminal-bench/<id>. Descriptions below are taken
from each task’s task.toml (trimmed).
fix-git — easy · software-engineeringterminal-bench/fix-gitmaster.overfull-hbox — easy · debuggingterminal-bench/overfull-hboxnginx-request-logging — medium · system-administrationterminal-bench/nginx-request-loggingsanitize-git-repo — medium · securityterminal-bench/sanitize-git-repoquery-optimize — medium · data-scienceterminal-bench/query-optimizedna-insert — medium · scientific-computingterminal-bench/dna-insertcancel-async-tasks — hard · software-engineeringterminal-bench/cancel-async-taskspassword-recovery — hard · securityterminal-bench/password-recoverysanitize-git-repo.sparql-university — hard · data-queryingterminal-bench/sparql-universitydata-querying); hardest expert
estimate in this slice — good stress task for tool + reasoning agents.| Capability cluster | Tasks |
|---|---|
| Version control / repo surgery | fix-git, sanitize-git-repo |
| Document / toolchain debugging | overfull-hbox |
| Service install & config | nginx-request-logging |
| SQL rewriting | query-optimize |
| Scientific domain reasoning | dna-insert |
| Concurrent Python correctness | cancel-async-tasks |
| Forensics / low-level investigation | password-recovery |
| Knowledge-graph querying | sparql-university |
harbor run -d fritzprix/libragent-diverse-9 -a <agent> -m <model>
harbor run -d fritzprix/libragent-diverse-9@v1.0 -a <agent> -m <model> -k 1Pin a digest if you need bit-for-bit reproducibility of the dataset manifest revision shown on Hub after publish.
pnpm bench:diverse # full 9-task suite (default n-attempts=1)
pnpm bench:diverse:n1 # first task only
pnpm bench:registry # same default dataset preset
pnpm xbench:registry # Hermes on the same defaultOfficial-style multi-attempt:
pnpm bench:diverse --n-attempts 5Previous narrow smoke set (still available):
pnpm bench:registry --dataset NovitaAI/tb21-file-recoveryharbor run -p /path/to/libragent-diverse-9 -a <agent> -m <model>| Item | Value |
|---|---|
| Dataset ID | fritzprix/libragent-diverse-9 |
| Initial tag | v1.0 (+ latest) |
| Task ownership | Upstream terminal-bench/* digests only |
| Publish mode | harbor publish … --public --no-tasks |
Bump the dataset tag (e.g. v1.1) when the task list or pinned digests
change. Do not silently retarget latest to a different skill mix if
downstream jobs treat this ID as a fixed contract—document the change here and
in the Hub description.
To refresh digests to newer published task revisions:
cd /path/to/libragent-diverse-9
harbor sync --upgrade # or re-add each terminal-bench/<task>
harbor publish . -t v1.1 --public --no-taskstask.toml
[[task.authors]]).fritzprix (LibrAgent harness smoke design).If you cite results, cite Terminal-Bench for task credit and note that scores are on this 9-task stratified subset, not the full 89-task benchmark.
benchmarks/harbor/README.md (in the LibrAgent repo)