fritzprix/libragent-diverse-9

Stratified 9-task evaluation slice sampled from Terminal-Bench 2.1 (terminal-bench/terminal-bench-2-1, 89 tasks).

Hub: https://hub.harborframework.com/datasets/fritzprix/libragent-diverse-9
Tags: v1.0, latest · Visibility: public · Author: fritzprix

This is a curation, not a re-adaptation. Every task is a pinned registry reference to an existing terminal-bench/<task> package (deterministic verifiers, same timeouts/resources as upstream). No task archives were republished under the fritzprix org.

---

Purpose

Use this dataset when you need a short, repeatable smoke / harness-regression suite that still stretches an agent across meaningfully different skills.

Typical uses:

• LibrAgent Harbor adapter smoke (pnpm bench:diverse)
• Cross-agent comparisons (e.g. LibrAgent vs Hermes) on a fixed 9-task contract
• Catching harness regressions that a narrow file/data-only set would miss (sysadmin, security, async SE, SPARQL, scientific tooling, etc.)

It is not a substitute for full Terminal-Bench 2.1 or Harbor Index leaderboard runs. Scores on 9 tasks have high variance; treat them as a diagnostic slice, not a ranking claim.

---

Motivation

The previous LibrAgent default registry smoke set, NovitaAI/tb21-file-recovery (also 9 tasks from TB2.1), is useful but distributionally thin:

Property	NovitaAI/tb21-file-recovery	fritzprix/libragent-diverse-9
Tasks	9	9
Difficulty	8× medium, 1× hard	2× easy, 4× medium, 3× hard
Categories	2 (file-operations, data-processing)	7 distinct categories
Overlap with this set	—	none (disjoint task IDs)

Repeated runs on that set over-fit local iteration to OCR / log / recovery-style file work. This slice deliberately rebalances difficulty and category so smoke runs exercise a wider agent surface (VCS, LaTeX debug, nginx, secret hygiene, SQL, biology primers, async Python, forensics, SPARQL).

---

Sampling method

1. Source pool: all 89 tasks in terminal-bench/terminal-bench-2-1 (metadata from each task’s task.toml: metadata.difficulty, metadata.category).
2. Target size: 9 tasks (same budget as the prior smoke set).
3. Difficulty quota: prefer coverage of easy / medium / hard → final mix 2 / 4 / 3.
4. Category quota: maximize distinct metadata.category values under that size → 7 categories (SE and security appear twice at different difficulties).
5. Exclusion: no task ID from NovitaAI/tb21-file-recovery.
6. Pinning: each entry in dataset.toml stores name + content digest (sha256:…) so harbor run -d fritzprix/libragent-diverse-9 resolves a fixed task revision.

Selection was manual stratified sampling, not random. Preference went to tasks that (a) fill an empty difficulty×category cell, (b) are widely cited in TB discussions, and (c) keep verifier complexity reasonable for local Docker smoke (no GPU-required tasks in this slice).

---

Composition overview

#	Task	Difficulty	Category	Agent timeout	Expert / junior (min)
1	fix-git	easy	software-engineering	900s	5 / 20
2	overfull-hbox	easy	debugging	750s	60 / 60
3	nginx-request-logging	medium	system-administration	900s	20 / 120
4	sanitize-git-repo	medium	security	900s	30 / 60
5	query-optimize	medium	data-science	900s	60 / 480
6	dna-insert	medium	scientific-computing	1800s	30 / 120
7	cancel-async-tasks	hard	software-engineering	900s	120 / 600
8	password-recovery	hard	security	900s	100 / 300
9	sparql-university	hard	data-querying	900s	800 / 10000

Difficulty: 2 easy · 4 medium · 3 hard
Categories: software-engineering ×2, security ×2, debugging, system-administration, data-science, scientific-computing, data-querying

---

Task notes

Upstream registry IDs are terminal-bench/<id>. Descriptions below are taken from each task’s task.toml (trimmed).

1. fix-git — easy · software-engineering

• Registry: terminal-bench/fix-git
• What it tests: Recover lost Git commits from a detached HEAD and merge them back onto master.
• Tags: coding, version-control
• Author: TheMikeMerrill
• Why included: Cheap easy SE / VCS smoke; fast expert estimate (~5 min).

2. overfull-hbox — easy · debugging

• Registry: terminal-bench/overfull-hbox
• What it tests: Fix LaTeX overfull hbox warnings by swapping words for valid synonyms while still compiling under constraints.
• Tags: latex, document-processing, combinatorial-optimization
• Author: Sam Buchanan
• Why included: Non-code debugging surface (document toolchain) missing from file-recovery smokes. Uses 2 CPUs / 4 GiB in the upstream env.

3. nginx-request-logging — medium · system-administration

• Registry: terminal-bench/nginx-request-logging
• What it tests: Install/configure Nginx with advanced request logging, rate limiting, and custom error pages.
• Tags: web-server
• Author: Junhong Shen
• Why included: Classic sysadmin / service-config path.

4. sanitize-git-repo — medium · security

• Registry: terminal-bench/sanitize-git-repo
• What it tests: Find and replace sensitive API keys/tokens in a Git repo with placeholders without touching unrelated files.
• Tags: security, system, version-control
• Author: jeffreywpli
• Why included: Secret-hygiene / careful edit discipline (security ≠ crypto-breaking).

5. query-optimize — medium · data-science

• Registry: terminal-bench/query-optimize
• What it tests: Rewrite a slow SQL query (correlated subqueries) with CTEs / window functions while preserving exact output.
• Tags: query-optimization, sql-query
• Author: Shreyas Pimpalgaonkar
• Why included: Structured data skill without collapsing into the file/OCR cluster of the old smoke set. Verifier timeout 1800s upstream.

6. dna-insert — medium · scientific-computing

• Registry: terminal-bench/dna-insert
• What it tests: Design PCR primers for site-directed mutagenesis from plasmid sequences under length / Tm constraints.
• Tags: biology, cloning
• Author: Karl Krauth
• Why included: Domain scientific tooling; longer agent budget (1800s).

7. cancel-async-tasks — hard · software-engineering

• Registry: terminal-bench/cancel-async-tasks
• What it tests: Async task concurrency control with correct cleanup on cancellation, including queued-task edge cases.
• Tags: async, concurrency, python
• Author: Alex Shaw
• Why included: Hard concurrent SE — stresses planning and careful implementation beyond “edit a file until tests pass.”

8. password-recovery — hard · security

• Registry: terminal-bench/password-recovery
• What it tests: Digital forensics: recover a deleted password from fragmented data in a disk image via CLI tools.
• Tags: system, file-operations, troubleshooting
• Author: Harsh Raj
• Why included: Hard security/forensics; distinct from medium sanitize-git-repo.

9. sparql-university — hard · data-querying

• Registry: terminal-bench/sparql-university
• What it tests: Complex SPARQL over an RDF knowledge graph (constraints, aggregations, date filters).
• Tags: knowledge-graph, sparql-query, information-retrieval
• Author: Orfeas Menis Mastromichalakis
• Why included: Unique TB2.1 category (data-querying); hardest expert estimate in this slice — good stress task for tool + reasoning agents.

---

What this suite exercises (agent surface)

Capability cluster	Tasks
Version control / repo surgery	fix-git, sanitize-git-repo
Document / toolchain debugging	overfull-hbox
Service install & config	nginx-request-logging
SQL rewriting	query-optimize
Scientific domain reasoning	dna-insert
Concurrent Python correctness	cancel-async-tasks
Forensics / low-level investigation	password-recovery
Knowledge-graph querying	sparql-university

---

How to run

Harbor CLI

harbor run -d fritzprix/libragent-diverse-9 -a <agent> -m <model>
harbor run -d fritzprix/libragent-diverse-9@v1.0 -a <agent> -m <model> -k 1

Pin a digest if you need bit-for-bit reproducibility of the dataset manifest revision shown on Hub after publish.

LibrAgent repo wrappers

pnpm bench:diverse          # full 9-task suite (default n-attempts=1)
pnpm bench:diverse:n1       # first task only
pnpm bench:registry         # same default dataset preset
pnpm xbench:registry        # Hermes on the same default

Official-style multi-attempt:

pnpm bench:diverse --n-attempts 5

Previous narrow smoke set (still available):

pnpm bench:registry --dataset NovitaAI/tb21-file-recovery

Local path (pre-publish / offline edit)

harbor run -p /path/to/libragent-diverse-9 -a <agent> -m <model>

---

Versioning & maintenance

Item	Value
Dataset ID	fritzprix/libragent-diverse-9
Initial tag	v1.0 (+ latest)
Task ownership	Upstream terminal-bench/* digests only
Publish mode	harbor publish … --public --no-tasks

Bump the dataset tag (e.g. v1.1) when the task list or pinned digests change. Do not silently retarget latest to a different skill mix if downstream jobs treat this ID as a fixed contract—document the change here and in the Hub description.

To refresh digests to newer published task revisions:

cd /path/to/libragent-diverse-9
harbor sync --upgrade   # or re-add each terminal-bench/<task>
harbor publish . -t v1.1 --public --no-tasks

---

Attribution

• Tasks & verifiers: Terminal-Bench authors (see each task’s task.toml [[task.authors]]).
• Source dataset: terminal-bench/terminal-bench-2-1.
• This curation: fritzprix (LibrAgent harness smoke design).

If you cite results, cite Terminal-Bench for task credit and note that scores are on this 9-task stratified subset, not the full 89-task benchmark.

---

Limitations

• n=9 → high run-to-run variance; unsuitable alone for SOTA claims.
• TB2.1 only → no SWE-bench issue patches, no GAIA web/tool-use judges, no Harbor Index multi-source mix.
• No GPU tasks in this slice.
• Timeouts/resources are upstream defaults; do not loosen them for “official” style comparisons.
• Expert/junior minute estimates are upstream metadata, not measured LibrAgent wall times.

---

Related links

• Harbor Hub dataset: https://hub.harborframework.com/datasets/fritzprix/libragent-diverse-9
• Terminal-Bench 2.1: https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2-1
• Prior LibrAgent smoke (narrow): https://hub.harborframework.com/datasets/NovitaAI/tb21-file-recovery
• LibrAgent Harbor guide: benchmarks/harbor/README.md (in the LibrAgent repo)