fritzprix/libragent-media-desktop-10

harbor run -d fritzprix/libragent-media-desktop-10

fritzprix/libragent-media-desktop-10

v1.2 (preferred): Hub tag v1.2 / latest. Dropped both HLE tasks (hle-identify-city-from-photo, hle-name-alkaloid-compound) — they need JUDGE_MODELS / LLM-as-judge and produced RewardFileNotFoundError without judge keys. Replaced with deterministic verifiers: harbor-index/gaia-find-chess-winning-move (exact string) and terminal-bench/gcode-to-text (pytest). Prefer harbor run -d fritzprix/libragent-media-desktop-10@v1.2.

v1.1: OSWorld ×3 removed; three custom web visual-bug tasks (web-theme-toggle-bug, web-nav-wrong-page, web-form-visible-error). Agents use browser__* against host ports 18765/18766/18767. Do not use desktop__* / media__captureScreen for these.

Curated 10-task Harbor dataset for evaluating LibrAgent media and browser builtin tools (media__seeContent, browser__*).

Hub: https://hub.harborframework.com/datasets/fritzprix/libragent-media-desktop-10
Tags: v1.2, latest · Visibility: public · Author: fritzprix

This is a curation, not a re-adaptation. Every entry is a pinned registry reference (name + content digest) to an existing published task package (plus three fritzprix/web-* customs). Dataset publish uses --no-tasks for upstream pins.

Source pool	Tasks	Role in this slice
Terminal-Bench 2.1	5	Docker vision / OCR / video / gcode→text
Harbor Index 1.0	2	Batch receipt OCR + GAIA chess (exact-match)
`fritzprix/web-*` customs	3	Published web visual-bug (browser__)

Local working copy (pre-/post-publish):
.libragent/work/datasets/media-desktop-eval/

Hub description (short)

10-task LibrAgent harness for media + browser tools. Mix: 5 Terminal-Bench vision/OCR/video + 2 Harbor Index (batch OCR, GAIA chess exact-match) + 3 custom web visual-bugs. All rewards are deterministic (pytest / exact string) — no JUDGE_MODELS. Pins upstream digests (--no-tasks). Smoke / trajectory checks for media__seeContent and browser__* — not a full TB leaderboard.

Purpose

Use this dataset when you need a short, repeatable suite that specifically exercises multimodal and GUI tool surfaces that Terminal-Bench SE smokes (fritzprix/libragent-diverse-9, NovitaAI/tb21-file-recovery) barely touch.

Typical uses:

LibrAgent Harbor adapter smoke focused on media/desktop builtins
ATIF trajectory audits: did the agent call media__* / desktop__* vs shell OCR / pure coding workarounds?
Cross-agent comparisons on a fixed 10-task media/desktop contract
Catching regressions in screen-capture ↔ click coordinate contracts

It is not:

a substitute for full Terminal-Bench 2.1 (89), Harbor Index (82), or OSWorld-Verified (361)
a claim that every task requires LibrAgent media tools (several TB tasks remain solvable via CLI OCR) — score reward + tool usage together

Motivation

Gap in existing LibrAgent smokes	How this slice responds
`libragent-diverse-9` is SE / sysadmin / SQL / SPARQL	Adds OCR, spatial vision, video, browser bugs
`tb21-file-recovery` is file/data-heavy	Keeps vision but avoids overlapping that file set
HLE Index tasks need LLM judge API keys	Replaced with exact-match / pytest vision tasks
Host `desktop__*` cannot drive Harbor guest GUIs	Custom `web-` tasks use `browser__` + published ports

Sampling method

TB pool: scan cached task.toml tags + instruction.md for OCR / image / video signals; keep tasks where loading pixels into the LLM is a natural first action; require pytest (or equivalent deterministic) reward.
Harbor Index pool: keep batch OCR and exact-match vision (GAIA); exclude hle-* LLM-judge tasks unless judge env is guaranteed.
Custom web pool: published fritzprix/web-* visual-bug tasks with pytest verifiers and browser__* against host ports.
Exclusions: TB tasks whose verifier success path is primarily coding / ML without needing visual load (path-tracing, sam-cell-seg, headless-terminal, install-windows-3.11); all OSWorld (host desktop tools ≠ guest GUI); all HLE LLM-judge.

Manual stratified curation (same style as fritzprix/libragent-diverse-9).

Composition overview

#	Task (short id)	Pool	Agent timeout	Primary LibrAgent tools	Skill	Verifier
1	`code-from-image`	TB	1200s	`media__seeContent`	OCR pseudocode → implement	pytest
2	`chess-best-move`	TB	900s	`media__seeContent`	Spatial board → best move	pytest
3	`financial-document-processor`	TB	1200s	`media__seeContent`	JPG/PDF classify + extract	pytest
4	`extract-moves-from-video`	TB	1800s	`media__seeContent`	Video / on-screen text → moves	pytest
5	`gcode-to-text`	TB	900s	`media__seeContent` (or render+OCR)	G-code path → printed text	pytest
6	`skillsbench-ocr-receipts-to-excel`	Index	600s	`media__seeContent`	Batch receipt OCR → Excel	pytest
7	`gaia-find-chess-winning-move`	Index	1200s	`media__seeContent`	Chess PNG → algebraic move	exact string
8	`web-theme-toggle-bug`	custom	task default	`browser__*`	Theme toggle visual bug	pytest
9	`web-nav-wrong-page`	custom	task default	`browser__*`	Wrong nav link	pytest
10	`web-form-visible-error`	custom	task default	`browser__*`	Form error copy/highlight	pytest

Mix: 7 media/vision-first · 3 browser visual-bug. 0 LLM-judge tasks.

Pinned digests live in dataset.toml.

Task notes

1. `code-from-image` — medium · software-engineering (OCR)

Registry: terminal-bench/code-from-image
What it tests: Read /app/code.png (pseudocode), implement equivalent logic, write the printed value to /app/output.txt.
Tags: ocr
Why included: Cleanest TB probe for “load image → reason → code”.
Tool expectation: Prefer media__seeContent on the PNG; shell tesseract alone is a weaker harness signal.

2. `chess-best-move` — medium · games

Registry: terminal-bench/chess-best-move
What it tests: Interpret chess_board.png (white to move); write best move(s) to /app/move.txt as e2e4-style lines.
Why included: Spatial / piece-layout vision, not plain OCR.

3. `financial-document-processor` — medium · data-processing

Registry: terminal-bench/financial-document-processor
What it tests: Classify mixed JPG/PDF under /app/documents/ as invoice vs other; move files; extract totals/VAT into CSV.
Tags: ocr, image-processing, financial, file-operations
Why included: Multi-document visual extraction under file ops.

4. `extract-moves-from-video` — hard · file-operations

Registry: terminal-bench/extract-moves-from-video
What it tests: Obtain a Zork playthrough video, recover typed commands, write one move per line to /app/solution.txt.
Tags: file-operations, web, video-processing
Why included: Temporal / on-screen text; frames via seeContent or equivalent multimodal path. Agent timeout 1800s.

5. `gcode-to-text` — medium · file-operations (OCR)

Registry: terminal-bench/gcode-to-text
What it tests: Decode text.gcode toolpaths into the printed string; write to /app/out.txt. Natural path is render → OCR / seeContent.
Why included (v1.2): Deterministic pytest reward; exercises visual reconstruction without LLM judge. (Also solvable by parsing alone — score reward + tool usage together.)

6. `skillsbench-ocr-receipts-to-excel` — Harbor Index

Registry: harbor-index/skillsbench-ocr-receipts-to-excel
What it tests: OCR all receipts under /app/workspace/dataset/img into a structured Excel workbook.
Keywords: image ocr, data statistics
Why included: Batch OCR → spreadsheet (Index, not TB). Deterministic pytest verifier.

7. `gaia-find-chess-winning-move` — Harbor Index (GAIA)

Registry: harbor-index/gaia-find-chess-winning-move
What it tests: Read attached chess PNG; write winning move for black in algebraic notation to /logs/artifacts/response.txt (expected Rd5).
Verifier: tests/test.sh exact string match vs expected_answer.txt (case-insensitive). No JUDGE_MODELS.
Why included (v1.2): Replaces HLE photo QA with a clearly verifiable vision answer while still exercising media__seeContent.

8–10. Custom web visual-bug tasks

Registry: fritzprix/web-theme-toggle-bug, web-nav-wrong-page, web-form-visible-error (@v1.0)
What they test: Fix published HTML/JS bugs via browser__* against host ports 18765–18767.
Verifier: deterministic pytest. Do not use host desktop__* / captureScreen — those hit the monitor, not the task UI.

What this suite exercises (agent surface)

Capability cluster	Tasks
Static OCR / document vision	#1, #3, #5, #6
Spatial / board vision	#2, #7
Video / frame text	#4
Browser inspect + edit	#8, #9, #10

Harness caveats (read before scoring)

TB / Index run in Harbor Docker. media__seeContent resolves HTTP or local/file:// paths on the LibrAgent host workspace. On Docker-attach, the agent may need workspace sync before a container path is readable — skipping seeContent for shell OCR is itself a harness signal.
Custom web tasks (#8–#10) publish a static site on host ports 18765–18767. Use browser__*, not host desktop__* / captureScreen.
Reward ≠ tool quality. Several TB vision tasks are solvable without media__*. When judging tools, inspect ATIF for media__ / browser__ call patterns, not only verifier/reward.txt.
GAIA (#7) writes to /logs/artifacts/response.txt. Workspace file tools may reject paths outside /app; shell write still satisfies the exact-match verifier (unlike HLE, no judge keys required).
No LLM-judge tasks in v1.2+. Do not re-add hle-* without wiring --verifier-env JUDGE_MODELS=… (and related keys) into every job.

How to run

Harbor Hub

Published: prefer v1.2 / latest · 10 tasks · public · no LLM judge

harbor run -d fritzprix/libragent-media-desktop-10@v1.2 -a <agent> -m <model>
harbor run -d fritzprix/libragent-media-desktop-10 -a <agent> -m <model> -k 1

LibrAgent wrappers

# Full 10 (n-attempts defaults to 1)
pnpm bench:registry --dataset fritzprix/libragent-media-desktop-10

# First task only
pnpm bench:registry --dataset fritzprix/libragent-media-desktop-10 --n-tasks 1

Direct adapter invocation from repo root (pnpm tauri dev / API on http://localhost:3030/api):

export PYTHONPATH="$(pwd)"
harbor run \
  -d fritzprix/libragent-media-desktop-10 \
  -a benchmarks.harbor.libragent_agent:LibrAgentHarborAdapter \
  --ak api_url=http://localhost:3030/api \
  --ak execution_mode=unsafe \
  -k 1

Local path (offline / pre-publish edit)

harbor run \
  -p .libragent/work/datasets/media-desktop-eval \
  -a benchmarks.harbor.libragent_agent:LibrAgentHarborAdapter \
  --ak execution_mode=unsafe \
  -k 1

Vision-only slice (skip custom web)

harbor run -d fritzprix/libragent-media-desktop-10@v1.2 \
  --include 'terminal-bench/*' --include 'harbor-index/*' \
  -a benchmarks.harbor.libragent_agent:LibrAgentHarborAdapter \
  --ak execution_mode=unsafe -k 1

Official-style multi-attempt (do not loosen timeouts/resources):

pnpm bench:registry --dataset fritzprix/libragent-media-desktop-10 --n-attempts 5

Publishing to Harbor Hub

This dataset is published as a manifest of pinned upstream packages. Upstream task blobs already live on Hub under terminal-bench/*, harbor-index/*, and xlang-ai/osworld-verified__*. Publishing with --no-tasks registers only the dataset composition + digests (same pattern as fritzprix/libragent-diverse-9).

Initial publish (done: v1.0 + public)

Already published from this tree:

cd .libragent/work/datasets/media-desktop-eval
printf 'y\n' | harbor publish . -t v1.0 --public --no-tasks

Result:

Field	Value
Dataset	`fritzprix/libragent-media-desktop-10`
Hash	`9656b03197f2`
Revision	`1`
Tasks	`10` (manifest refs only; `Files=0` with `--no-tasks`)
Tags	`v1.0`, `latest`
Visibility	public

Hub: https://hub.harborframework.com/datasets/fritzprix/libragent-media-desktop-10

Flag meanings (for re-publish / bumps):

Flag	Meaning
`-t v1.0`	Named revision tag (also applies `latest`)
`--public`	Visible on Harbor Hub without private org access
`--no-tasks`	Do not re-upload task packages; only the dataset.toml composition

CLI may prompt that making the dataset public can make its tasks public — with --no-tasks no task blobs are uploaded; upstream packages stay owned by their original orgs. Pipe y if running non-interactively.

What not to do

Do not omit --no-tasks unless you intentionally fork/republish every upstream task under fritzprix/ (large, unnecessary, breaks digest pins).
Do not silently retarget latest to a different skill mix; bump v1.1+ and document the change here.
Do not loosen upstream timeouts/resources for “official-style” comparisons.

Updating the composition later

cd .libragent/work/datasets/media-desktop-eval
harbor remove <org>/<task> --from .        # if removing
harbor add <org>/<task> --to .             # if adding
harbor sync .                              # refresh digests
# edit README.md + SELECTION.md
printf 'y\n' | harbor publish . -t v1.2 --public --no-tasks

Versioning & maintenance

Item	Value
Dataset ID	`fritzprix/libragent-media-desktop-10`
Current tag	`v1.2` (+ `latest`)
Task ownership	Upstream digests + `fritzprix/web-*`
Publish mode	`harbor publish … --public --no-tasks`
Working tree	`.libragent/work/datasets/media-desktop-eval/` (git-ignored)
Verifiers	Deterministic only (no HLE / `JUDGE_MODELS`)

Attribution

Terminal-Bench task authors (see each task.toml)
Harbor Index / HLE / SkillsBench upstream authors
OSWorld authors + Harbor OSWorld adapter (xlang-ai/osworld-verified)
Curation: fritzprix (LibrAgent media/desktop harness slice)

If you cite results, cite the upstream benchmarks for task credit and note that scores are on this 10-task media/desktop subset, not the full source datasets.

Limitations

n=10 → high run-to-run variance; unsuitable alone for SOTA claims.
Heterogeneous envs (Docker TB/Index vs OSWorld use.computer) → one Harbor job may need adapter support for both.
Tool use is not forced by verifiers; trajectory inspection is required for media/desktop tool quality claims.
Expert/junior minute estimates (where present) are upstream metadata, not measured LibrAgent wall times.

fritzprix/libragent-media-desktop-10

fritzprix/libragent-media-desktop-10

fritzprix/libragent-media-desktop-10

fritzprix/libragent-media-desktop-10

fritzprix/libragent-media-desktop-10

fritzprix/libragent-media-desktop-10

fritzprix/libragent-media-desktop-10

Hub description (short)

Purpose

Motivation

Sampling method

Composition overview

Task notes

1. code-from-image — medium · software-engineering (OCR)

2. chess-best-move — medium · games

3. financial-document-processor — medium · data-processing

4. extract-moves-from-video — hard · file-operations

5. gcode-to-text — medium · file-operations (OCR)

6. skillsbench-ocr-receipts-to-excel — Harbor Index

7. gaia-find-chess-winning-move — Harbor Index (GAIA)

8–10. Custom web visual-bug tasks

What this suite exercises (agent surface)

Harness caveats (read before scoring)

How to run

Harbor Hub

LibrAgent wrappers

Local path (offline / pre-publish edit)

Vision-only slice (skip custom web)

Publishing to Harbor Hub

Initial publish (done: v1.0 + public)

What not to do

Updating the composition later

Versioning & maintenance

Attribution

Limitations

Related links

fritzprix/libragent-media-desktop-10

Hub description (short)

Purpose

Motivation

Sampling method

Composition overview

Task notes

1. code-from-image — medium · software-engineering (OCR)

2. chess-best-move — medium · games

3. financial-document-processor — medium · data-processing

4. extract-moves-from-video — hard · file-operations

5. gcode-to-text — medium · file-operations (OCR)

6. skillsbench-ocr-receipts-to-excel — Harbor Index

7. gaia-find-chess-winning-move — Harbor Index (GAIA)

8–10. Custom web visual-bug tasks

What this suite exercises (agent surface)

Harness caveats (read before scoring)

How to run

Harbor Hub

LibrAgent wrappers

Local path (offline / pre-publish edit)

Vision-only slice (skip custom web)

Publishing to Harbor Hub

Initial publish (done: v1.0 + public)

What not to do

Updating the composition later

Versioning & maintenance

Attribution

Limitations

Related links

fritzprix/libragent-media-desktop-10

Hub description (short)

Purpose

Motivation

Sampling method

Composition overview

Task notes

1. code-from-image — medium · software-engineering (OCR)

2. chess-best-move — medium · games

3. financial-document-processor — medium · data-processing

4. extract-moves-from-video — hard · file-operations

5. gcode-to-text — medium · file-operations (OCR)

6. skillsbench-ocr-receipts-to-excel — Harbor Index

7. gaia-find-chess-winning-move — Harbor Index (GAIA)

1. `code-from-image` — medium · software-engineering (OCR)

2. `chess-best-move` — medium · games

3. `financial-document-processor` — medium · data-processing

4. `extract-moves-from-video` — hard · file-operations

5. `gcode-to-text` — medium · file-operations (OCR)

6. `skillsbench-ocr-receipts-to-excel` — Harbor Index

7. `gaia-find-chess-winning-move` — Harbor Index (GAIA)

1. `code-from-image` — medium · software-engineering (OCR)

2. `chess-best-move` — medium · games

3. `financial-document-processor` — medium · data-processing

4. `extract-moves-from-video` — hard · file-operations

5. `gcode-to-text` — medium · file-operations (OCR)

6. `skillsbench-ocr-receipts-to-excel` — Harbor Index

7. `gaia-find-chess-winning-move` — Harbor Index (GAIA)

1. `code-from-image` — medium · software-engineering (OCR)

2. `chess-best-move` — medium · games

3. `financial-document-processor` — medium · data-processing

4. `extract-moves-from-video` — hard · file-operations

5. `gcode-to-text` — medium · file-operations (OCR)

6. `skillsbench-ocr-receipts-to-excel` — Harbor Index

7. `gaia-find-chess-winning-move` — Harbor Index (GAIA)

1. `code-from-image` — medium · software-engineering (OCR)

2. `chess-best-move` — medium · games

3. `financial-document-processor` — medium · data-processing

4. `extract-moves-from-video` — hard · file-operations

5. `gcode-to-text` — medium · file-operations (OCR)

6. `skillsbench-ocr-receipts-to-excel` — Harbor Index

7. `gaia-find-chess-winning-move` — Harbor Index (GAIA)