gustofied/compute-bazaar-bench

harbor run -d gustofied/compute-bazaar-bench

compute-bazaar-bench

Harbor dataset

A benchmark for evaluating agents on compute-market tasks.

hey, i mix eval/evaluation/task/env, they are the same thing btw

Large compute deals rarely happen in a single, transparent "venue". Buyer requirements, supply, pricing, terms, diligence, and relationship history are spread across messages, calls, spreadsheets, PDFs, data rooms, and people's memory. This context has to be reconstructed, checked, and turned into action.

Compute Bazaar Bench asks where agents can be useful across that market, from transactions and sourcing to market intelligence, risk, financing, and operations. It brings together two complementary styles of environments.

Compute Deal Work

Compute Deal Work starts with transactions: intake, diligence, and contracting. Human relationships remain central, while agents may carry more of the analysis, documentation, and operational volume around them. The products and commentary from Epilogue and ComputeDesk around compute desks and deal flow motivate this direction.

Harvey's Legal Agent Benchmark provides the methodological starting point. Rather than inventing arbitrary benchmark exercises, we begin with task forms that already represent real professional work and test whether they transfer meaningfully into compute transactions.

transactions evaluates agents on compute deal work across intake, diligence, and contracting.

Compute Market Games

Compute market games place agents inside changing compute-market processes, including compute procurement, brokerage, matching, and negotiation. They are more alive and interactive than closed professional-work evaluations, and their stateful structure makes them more adaptable for training. The first implemented environment is reliability-is-blind.

Setup

For the most part, the benchmark follows Harbor's task and evaluation methodology.

compute-bazaar-bench/
|-- dataset.toml
|-- evals/
|   |-- reliability-is-blind/
|   `-- transactions/
|-- viewer/
`-- jobs/
    `-- <harbor-job>/

Open the Compute Bazaar Terminal and enter eval:

compute-bazaar terminal

Then run an evaluation from the Terminal shell with Harbor:

harbor run \
  -p compute-bazaar-bench/evals/reliability-is-blind/harbor \
  -a AGENT \
  -m MODEL \
  -e ENVIRONMENT \
  -o compute-bazaar-bench/jobs

You can also run Harbor directly from any shell.

Browse the resulting jobs with Harbor's native viewer:

cd compute-bazaar-bench
harbor view jobs

Evaluations

transactions

Its first three tasks move through intake, diligence, and contracting:

messy intent -> organized evidence process -> controlled transaction paper
analyze      -> draft                      -> review

Can an agent perform the professional work required to advance a compute transaction accurately, with evidence, and without losing the deal's controlling terms?

reliability-is-blind

reliability-is-blind is an interactive compute-brokerage evaluation. An agent repeatedly places supply into deals and learns which suppliers to trust from delivery outcomes without being told which supplier caused a failure.