Harbor Hub

lamina/product-bench

Product Bench — 3-arm public benchmark (direct / plan / lamina). Thin-slice greenfield product coding with a RewardKit LLM judge. Median of 3 full reruns using Cursor CLI + Composer 2.5.

New Job
harbor run -d lamina/product-bench

Product Bench — 3-arm (direct / plan / lamina)

Harbor dataset: lamina/product-bench
Hub: https://hub.harborframework.com/datasets/lamina/product-bench

Compares three ways to build small product apps with the same agent and judge:

Arm What it means
direct Code straight from the brief
plan Short plan, then code
lamina Lamina skills (/lamina-init + /lamina-design), then code

Initial public Product Bench release. Under this benchmark harness, Lamina’s median score exceeded the direct and plan-first arms on all four tasks. This release covers small greenfield vertical slices only; it does not establish performance on larger applications, brownfield projects, or other models.

What’s in this dataset

12 public tasks = 4 product tasks × 3 arms:

Task Arms
loan-library direct · plan · lamina
review-room direct · plan · lamina
simple-list direct · plan · lamina
toggle-preference direct · plan · lamina
  • Scored on: product source /app/app.mjs + /app/ui.mjs (LLM-as-judge)
  • Published claim (repo): per-cell median of 3 full re-runs
  • Not scored: .lamina/ process notes; no /lamina-verify step in this pilot

How to run

# Needs: CURSOR_API_KEY, OPENAI_API_KEY, REWARDKIT_JUDGE=openai/gpt-5.5, LITELLM_DROP_PARAMS=1

harbor run \
  -d lamina/product-bench@latest \
  -a cursor-cli \
  -m cursor/composer-2.5 \
  --env-file .env \
  -n 1

Pinned tag: @product-bench-median-n3 (same content as latest at publish time).

For the full local 3-seed / median protocol:
benchmarks/lb6/pilot/publication/REPRODUCE.md

Results (what we got)

Task
lamina/dev-loan-library-lamina-v3
lamina/dev-loan-library-direct-v3
lamina/dev-toggle-preference-direct-v3
lamina/dev-review-room-lamina-v3
lamina/dev-toggle-preference-lamina-v3
lamina/dev-simple-list-direct-v3
lamina/dev-review-room-direct-v3
lamina/dev-simple-list-lamina-v3
lamina/dev-toggle-preference-plan-v3
lamina/dev-simple-list-plan-v3
lamina/dev-loan-library-plan-v3
lamina/dev-review-room-plan-v3

Displaying 12 of 12 tasks