fritzprix/libragent-media-desktop-10

v1.2 (preferred): Hub tag v1.2 / latest. Dropped both HLE tasks (hle-identify-city-from-photo, hle-name-alkaloid-compound) — they need JUDGE_MODELS / LLM-as-judge and produced RewardFileNotFoundError without judge keys. Replaced with deterministic verifiers: harbor-index/gaia-find-chess-winning-move (exact string) and terminal-bench/gcode-to-text (pytest). Prefer harbor run -d fritzprix/libragent-media-desktop-10@v1.2.

v1.1: OSWorld ×3 removed; three custom web visual-bug tasks (web-theme-toggle-bug, web-nav-wrong-page, web-form-visible-error). Agents use browser__* against host ports 18765/18766/18767. Do not use desktop__* / media__captureScreen for these.

Curated 10-task Harbor dataset for evaluating LibrAgent media and browser builtin tools (media__seeContent, browser__*).

Hub: https://hub.harborframework.com/datasets/fritzprix/libragent-media-desktop-10
Tags: v1.2, latest · Visibility: public · Author: fritzprix

This is a curation, not a re-adaptation. Every entry is a pinned registry reference (name + content digest) to an existing published task package (plus three fritzprix/web-* customs). Dataset publish uses --no-tasks for upstream pins.

Source pool	Tasks	Role in this slice
Terminal-Bench 2.1	5	Docker vision / OCR / video / gcode→text
Harbor Index 1.0	2	Batch receipt OCR + GAIA chess (exact-match)
fritzprix/web-* customs	3	Published web visual-bug (browser__)

Local working copy (pre-/post-publish):
.libragent/work/datasets/media-desktop-eval/

---

Hub description (short)

10-task LibrAgent harness for media + browser tools. Mix: 5 Terminal-Bench vision/OCR/video + 2 Harbor Index (batch OCR, GAIA chess exact-match) + 3 custom web visual-bugs. All rewards are deterministic (pytest / exact string) — no JUDGE_MODELS. Pins upstream digests (--no-tasks). Smoke / trajectory checks for media__seeContent and browser__* — not a full TB leaderboard.

---

Purpose

Use this dataset when you need a short, repeatable suite that specifically exercises multimodal and GUI tool surfaces that Terminal-Bench SE smokes (fritzprix/libragent-diverse-9, NovitaAI/tb21-file-recovery) barely touch.

Typical uses:

• LibrAgent Harbor adapter smoke focused on media/desktop builtins
• ATIF trajectory audits: did the agent call media__* / desktop__* vs shell OCR / pure coding workarounds?
• Cross-agent comparisons on a fixed 10-task media/desktop contract
• Catching regressions in screen-capture ↔ click coordinate contracts

It is not:

• a substitute for full Terminal-Bench 2.1 (89), Harbor Index (82), or OSWorld-Verified (361)
• a claim that every task requires LibrAgent media tools (several TB tasks remain solvable via CLI OCR) — score reward + tool usage together

---

Motivation

Gap in existing LibrAgent smokes	How this slice responds
libragent-diverse-9 is SE / sysadmin / SQL / SPARQL	Adds OCR, spatial vision, video, browser bugs
tb21-file-recovery is file/data-heavy	Keeps vision but avoids overlapping that file set
HLE Index tasks need LLM judge API keys	Replaced with exact-match / pytest vision tasks
Host desktop__* cannot drive Harbor guest GUIs	Custom web-* tasks use browser__* + published ports

---

Sampling method

1. TB pool: scan cached task.toml tags + instruction.md for OCR / image / video signals; keep tasks where loading pixels into the LLM is a natural first action; require pytest (or equivalent deterministic) reward.
2. Harbor Index pool: keep batch OCR and exact-match vision (GAIA); exclude hle-* LLM-judge tasks unless judge env is guaranteed.
3. Custom web pool: published fritzprix/web-* visual-bug tasks with pytest verifiers and browser__* against host ports.
4. Exclusions: TB tasks whose verifier success path is primarily coding / ML without needing visual load (path-tracing, sam-cell-seg, headless-terminal, install-windows-3.11); all OSWorld (host desktop tools ≠ guest GUI); all HLE LLM-judge.

Manual stratified curation (same style as fritzprix/libragent-diverse-9).

---

Composition overview

#	Task (short id)	Pool	Agent timeout	Primary LibrAgent tools	Skill	Verifier
1	code-from-image	TB	1200s	media__seeContent	OCR pseudocode → implement	pytest
2	chess-best-move	TB	900s	media__seeContent	Spatial board → best move	pytest
3	financial-document-processor	TB	1200s	media__seeContent	JPG/PDF classify + extract	pytest
4	extract-moves-from-video	TB	1800s	media__seeContent	Video / on-screen text → moves	pytest
5	gcode-to-text	TB	900s	media__seeContent (or render+OCR)	G-code path → printed text	pytest
6	skillsbench-ocr-receipts-to-excel	Index	600s	media__seeContent	Batch receipt OCR → Excel	pytest
7	gaia-find-chess-winning-move	Index	1200s	media__seeContent	Chess PNG → algebraic move	exact string
8	web-theme-toggle-bug	custom	task default	browser__*	Theme toggle visual bug	pytest
9	web-nav-wrong-page	custom	task default	browser__*	Wrong nav link	pytest
10	web-form-visible-error	custom	task default	browser__*	Form error copy/highlight	pytest

Mix: 7 media/vision-first · 3 browser visual-bug. 0 LLM-judge tasks.

Pinned digests live in dataset.toml.

---

Task notes

1. code-from-image — medium · software-engineering (OCR)

• Registry: terminal-bench/code-from-image
• What it tests: Read /app/code.png (pseudocode), implement equivalent logic, write the printed value to /app/output.txt.
• Tags: ocr
• Why included: Cleanest TB probe for “load image → reason → code”.
• Tool expectation: Prefer media__seeContent on the PNG; shell tesseract alone is a weaker harness signal.

2. chess-best-move — medium · games

• Registry: terminal-bench/chess-best-move
• What it tests: Interpret chess_board.png (white to move); write best move(s) to /app/move.txt as e2e4-style lines.
• Why included: Spatial / piece-layout vision, not plain OCR.

3. financial-document-processor — medium · data-processing

• Registry: terminal-bench/financial-document-processor
• What it tests: Classify mixed JPG/PDF under /app/documents/ as invoice vs other; move files; extract totals/VAT into CSV.
• Tags: ocr, image-processing, financial, file-operations
• Why included: Multi-document visual extraction under file ops.

4. extract-moves-from-video — hard · file-operations

• Registry: terminal-bench/extract-moves-from-video
• What it tests: Obtain a Zork playthrough video, recover typed commands, write one move per line to /app/solution.txt.
• Tags: file-operations, web, video-processing
• Why included: Temporal / on-screen text; frames via seeContent or equivalent multimodal path. Agent timeout 1800s.

5. gcode-to-text — medium · file-operations (OCR)

• Registry: terminal-bench/gcode-to-text
• What it tests: Decode text.gcode toolpaths into the printed string; write to /app/out.txt. Natural path is render → OCR / seeContent.
• Why included (v1.2): Deterministic pytest reward; exercises visual reconstruction without LLM judge. (Also solvable by parsing alone — score reward + tool usage together.)

6. skillsbench-ocr-receipts-to-excel — Harbor Index

• Registry: harbor-index/skillsbench-ocr-receipts-to-excel
• What it tests: OCR all receipts under /app/workspace/dataset/img into a structured Excel workbook.
• Keywords: image ocr, data statistics
• Why included: Batch OCR → spreadsheet (Index, not TB). Deterministic pytest verifier.

7. gaia-find-chess-winning-move — Harbor Index (GAIA)

• Registry: harbor-index/gaia-find-chess-winning-move
• What it tests: Read attached chess PNG; write winning move for black in algebraic notation to /logs/artifacts/response.txt (expected Rd5).
• Verifier: tests/test.sh exact string match vs expected_answer.txt (case-insensitive). No JUDGE_MODELS.
• Why included (v1.2): Replaces HLE photo QA with a clearly verifiable vision answer while still exercising media__seeContent.

8–10. Custom web visual-bug tasks

• Registry: fritzprix/web-theme-toggle-bug, web-nav-wrong-page, web-form-visible-error (@v1.0)
• What they test: Fix published HTML/JS bugs via browser__* against host ports 18765–18767.
• Verifier: deterministic pytest. Do not use host desktop__* / captureScreen — those hit the monitor, not the task UI.

---

What this suite exercises (agent surface)

Capability cluster	Tasks
Static OCR / document vision	#1, #3, #5, #6
Spatial / board vision	#2, #7
Video / frame text	#4
Browser inspect + edit	#8, #9, #10

---

Harness caveats (read before scoring)

1. TB / Index run in Harbor Docker. media__seeContent resolves HTTP or local/file:// paths on the LibrAgent host workspace. On Docker-attach, the agent may need workspace sync before a container path is readable — skipping seeContent for shell OCR is itself a harness signal.
2. Custom web tasks (#8–#10) publish a static site on host ports 18765–18767. Use browser__*, not host desktop__* / captureScreen.
3. Reward ≠ tool quality. Several TB vision tasks are solvable without media__*. When judging tools, inspect ATIF for media__ / browser__ call patterns, not only verifier/reward.txt.
4. GAIA (#7) writes to /logs/artifacts/response.txt. Workspace file tools may reject paths outside /app; shell write still satisfies the exact-match verifier (unlike HLE, no judge keys required).
5. No LLM-judge tasks in v1.2+. Do not re-add hle-* without wiring --verifier-env JUDGE_MODELS=… (and related keys) into every job.

---

How to run

Harbor Hub

Published: prefer v1.2 / latest · 10 tasks · public · no LLM judge

harbor run -d fritzprix/libragent-media-desktop-10@v1.2 -a <agent> -m <model>
harbor run -d fritzprix/libragent-media-desktop-10 -a <agent> -m <model> -k 1

LibrAgent wrappers

# Full 10 (n-attempts defaults to 1)
pnpm bench:registry --dataset fritzprix/libragent-media-desktop-10

# First task only
pnpm bench:registry --dataset fritzprix/libragent-media-desktop-10 --n-tasks 1

Direct adapter invocation from repo root (pnpm tauri dev / API on http://localhost:3030/api):

export PYTHONPATH="$(pwd)"
harbor run \
  -d fritzprix/libragent-media-desktop-10 \
  -a benchmarks.harbor.libragent_agent:LibrAgentHarborAdapter \
  --ak api_url=http://localhost:3030/api \
  --ak execution_mode=unsafe \
  -k 1

Local path (offline / pre-publish edit)

harbor run \
  -p .libragent/work/datasets/media-desktop-eval \
  -a benchmarks.harbor.libragent_agent:LibrAgentHarborAdapter \
  --ak execution_mode=unsafe \
  -k 1

Vision-only slice (skip custom web)

harbor run -d fritzprix/libragent-media-desktop-10@v1.2 \
  --include 'terminal-bench/*' --include 'harbor-index/*' \
  -a benchmarks.harbor.libragent_agent:LibrAgentHarborAdapter \
  --ak execution_mode=unsafe -k 1

Official-style multi-attempt (do not loosen timeouts/resources):

pnpm bench:registry --dataset fritzprix/libragent-media-desktop-10 --n-attempts 5

---

Publishing to Harbor Hub

This dataset is published as a manifest of pinned upstream packages. Upstream task blobs already live on Hub under terminal-bench/*, harbor-index/*, and xlang-ai/osworld-verified__*. Publishing with --no-tasks registers only the dataset composition + digests (same pattern as fritzprix/libragent-diverse-9).

Initial publish (done: v1.0 + public)

Already published from this tree:

cd .libragent/work/datasets/media-desktop-eval
printf 'y\n' | harbor publish . -t v1.0 --public --no-tasks

Result:

Field	Value
Dataset	fritzprix/libragent-media-desktop-10
Hash	9656b03197f2
Revision	1
Tasks	10 (manifest refs only; Files=0 with --no-tasks)
Tags	v1.0, latest
Visibility	public

Hub: https://hub.harborframework.com/datasets/fritzprix/libragent-media-desktop-10

Flag meanings (for re-publish / bumps):

Flag	Meaning
-t v1.0	Named revision tag (also applies latest)
--public	Visible on Harbor Hub without private org access
--no-tasks	Do not re-upload task packages; only the dataset.toml composition

CLI may prompt that making the dataset public can make its tasks public — with --no-tasks no task blobs are uploaded; upstream packages stay owned by their original orgs. Pipe y if running non-interactively.

What not to do

• Do not omit --no-tasks unless you intentionally fork/republish every upstream task under fritzprix/ (large, unnecessary, breaks digest pins).
• Do not silently retarget latest to a different skill mix; bump v1.1+ and document the change here.
• Do not loosen upstream timeouts/resources for “official-style” comparisons.

Updating the composition later

cd .libragent/work/datasets/media-desktop-eval
harbor remove <org>/<task> --from .        # if removing
harbor add <org>/<task> --to .             # if adding
harbor sync .                              # refresh digests
# edit README.md + SELECTION.md
printf 'y\n' | harbor publish . -t v1.2 --public --no-tasks

---

Versioning & maintenance

Item	Value
Dataset ID	fritzprix/libragent-media-desktop-10
Current tag	v1.2 (+ latest)
Task ownership	Upstream digests + fritzprix/web-*
Publish mode	harbor publish … --public --no-tasks
Working tree	.libragent/work/datasets/media-desktop-eval/ (git-ignored)
Verifiers	Deterministic only (no HLE / JUDGE_MODELS)

---

Attribution

• Terminal-Bench task authors (see each task.toml)
• Harbor Index / HLE / SkillsBench upstream authors
• OSWorld authors + Harbor OSWorld adapter (xlang-ai/osworld-verified)
• Curation: fritzprix (LibrAgent media/desktop harness slice)

If you cite results, cite the upstream benchmarks for task credit and note that scores are on this 10-task media/desktop subset, not the full source datasets.

---

Limitations

• n=10 → high run-to-run variance; unsuitable alone for SOTA claims.
• Heterogeneous envs (Docker TB/Index vs OSWorld use.computer) → one Harbor job may need adapter support for both.
• Tool use is not forced by verifiers; trajectory inspection is required for media/desktop tool quality claims.
• Expert/junior minute estimates (where present) are upstream metadata, not measured LibrAgent wall times.

---

Related links

• This Hub page: https://hub.harborframework.com/datasets/fritzprix/libragent-media-desktop-10
• Diverse SE smoke: https://hub.harborframework.com/datasets/fritzprix/libragent-diverse-9
• Terminal-Bench 2.1: https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2-1
• Harbor Index 1.0: https://hub.harborframework.com/datasets/harbor-index/harbor-index-1.0
• OSWorld-Verified: https://hub.harborframework.com/datasets/xlang-ai/osworld-verified
• LibrAgent Harbor guide: benchmarks/harbor/README.md
• Builtin tools: docs/user/en/guides/builtin-tools.md