harbor run -d oro-agents/smb-benchharbor run -d oro-agents/smb-benchSMB Bench: your own agent runs a simulated small business's buying desk for a 90-day quarter on ORO's hosted service, scored in margin points over a fixed buying rule
harbor run -d oro-agents/smb-benchSMB Bench: your own agent runs a simulated small business's buying desk for a 90-day quarter on ORO's hosted service, scored in margin points over a fixed buying rule
harbor run -d oro-agents/smb-benchSMB Bench gives an agent the buying desk of a simulated small business for one 90-day quarter. Each morning the agent reads the business's status and its own notes, works with the business tools, and ends the day. At the end of the quarter, ORO grades the run against a fixed buying rule on the same world. The simulator and the worlds stay on ORO's hosted lab service; this dataset only connects your Harbor agent to it.
You bring the agent and the model key. Results are runs with your own agent: they are not comparable with the SMB Bench leaderboard, which was produced with ORO's own buyer loop.
| Task | smb-bench-public-1: environment smb-bench-v1, 90 simulated days |
| Agent | any Harbor agent that supports MCP servers (for example codex or claude-code), any model |
| Reward | margin points over the fixed buying rule, from ORO's grader |
| Needs | an SMB Bench grant and your own model API key |
Email team@oroagents.com to request a grant. An operator issues a lab token, typically for 10 episodes with 3 open
at a time, as stated in your grant email; a grant lasts 7 days and can be renewed. Using a grant means accepting
ORO's lab access terms (https://lab-api.oroagents.com/v1/terms). Every Harbor trial uses one episode, including
trials that fail after the episode was created and every attempt (-k) or retry (-r). Keep -n at or below
your grant's open-episode limit.
uv tool install 'harbor[<extra>]==0.25.0' # your environment's extra (see Environments); Docker needs none
set -a; . ./smb-bench-grant.env; set +a # the grant file from your grant email: ORO_ENDPOINT and ORO_LAB_TOKEN
export <ENVIRONMENT_CREDENTIAL>=... # your environment's credentials (see Environments)
export OPENAI_API_KEY=... # your model key, used by your agent as usual
harbor run -y -d oro-agents/smb-bench@1.0.0 -e <environment> -a codex -m openai/gpt-5.4-mini -n 1
With E2B, for example, that is harbor[e2b], E2B_API_KEY and -e e2b.
Only the task's verifier receives ORO_LAB_TOKEN ([verifier.env] in task.toml). Never pass it to your agent
with --ae. ORO_ENDPOINT defaults to https://lab-api.oroagents.com. Set ORO_DECLARED_MODEL to record your
model's name with the episode. A downloaded copy of the dataset runs the same way with -p smb-bench-public-1 in
place of -d.
There is no graded result short of a full 90-day quarter: only the last step carries a reward, and a run that stops early has none.
The task limits your agent's network to ORO's lab service and your model's endpoint (network_mode = "allowlist") and keeps it open while Harbor installs the agent, so the environment must enforce an allowlist of
hostnames and switch it at run time. Harbor runs the task on any environment that can, and refuses it on the rest
before any agent runs, for example with ValueError: The execution phase network policy differs from a broader setup or shared phase policy, but this environment cannot change network policy after start. (harbor run exits
1).
In Harbor 0.25 these environments can run the task. Install Harbor with the environment's extra and set its credentials:
-e |
Install | Needs | Tested by ORO |
|---|---|---|---|
e2b |
harbor[e2b] |
E2B_API_KEY |
Yes: full quarters; during the agent phase the lab service answers and other hosts are blocked |
modal |
harbor[modal] |
MODAL_TOKEN_ID and MODAL_TOKEN_SECRET |
No |
daytona |
harbor[daytona] |
DAYTONA_API_KEY |
No |
beam |
harbor[beam] |
BEAM_TOKEN, or beam login |
No |
tensorlake |
harbor[tensorlake] |
TENSORLAKE_API_KEY |
No |
vercel |
harbor[vercel] |
VERCEL_TOKEN |
No |
novita |
harbor[novita] |
NOVITA_API_KEY |
No |
hyperbrowser |
harbor[hyperbrowser] |
HYPERBROWSER_API_KEY |
No |
islo |
harbor[islo] |
ISLO_API_KEY |
No |
runta |
harbor[runta] |
RUNTA_TOKEN |
No |
sbx (Docker Sandboxes, cloud) |
harbor[sbx] |
SBX_TOKEN |
No |
docker |
harbor |
a Docker host whose kernel supports nftables fib |
No; a host without fib refuses the task (tested) |
"No" means ORO has not played a quarter there. ORO ran the task on each cloud environment above with placeholder
credentials: Harbor accepted it and the run stopped only at the provider's sign-in. Harbor refuses the task on
prime (it needs a prebuilt image; this task builds its own), on runloop, mosaic, blaxel, langsmith and
smol (they cannot switch the allowlist at run time), and on environments without allowlists.
Press Ctrl-C once in the terminal running harbor run and let it stop its sandboxes (it records the trial as
CancelledError and exits 1). The episode stays open on
ORO's service and holds one of your grant's open slots until you abort it, even after its worker stops at the
24-hour limit. Its ID is in
jobs/<job>/<trial>/steps/provision/verifier/test-stdout.txt; to list your episodes and their states, or to abort
one:
curl -sH "Authorization: Bearer $ORO_LAB_TOKEN" "$ORO_ENDPOINT/v1/episodes"
curl -sX POST -H "Authorization: Bearer $ORO_LAB_TOKEN" -H "Content-Type: application/json" -d '{}' \
"$ORO_ENDPOINT/v1/episodes/EPISODE_ID/abort"
An aborted episode still counts toward the grant. A run that fails on its own aborts its episode. To check your
grant's limits (max_episodes, max_active, expires_at), call GET $ORO_ENDPOINT/v1/access. To see what you have
used, list your episodes as above: the ones not FINALIZED, ABORTED or EXPIRED count toward max_active. Every
episode counts toward max_episodes, including ones older than the 14-day trace window, which the list no longer
shows; if your grant was renewed, ask team@oroagents.com how many episodes remain.
Harbor's console doesn't show the simulated day. Each finished step writes its progress to reward.json, so the
latest one tells you where a run is:
ls -t jobs/*/*/steps/*/verifier/reward.json | head -1 | xargs cat
Codex may log failed to refresh available models at the start of each session. That message is harmless: the model
you chose with -m is still used.
The task has a provision step and up to 270 session-NNN steps: up to 3 agent sessions per simulated day. Each step
starts a fresh agent. A day that your agent ends on its first session uses one step, so a quarter normally runs
91 steps; Harbor skips the unused session steps once the quarter is graded. Each step's reward.json shows its progress:
day (the simulated day the session played), nudge (0 for the day's own session, 1 or 2 for a nudge session),
days_completed and running. Only the final step carries reward: the service's grade, which is the trial's
reward.
provision: your agent gets a one-line instruction and replies. The verifier then creates the episode on ORO's
service and opens day 1./var/lib/oro/today/prompt.md), this session's message (/var/lib/oro/today/message.md) and the morning
briefing (/var/lib/oro/today/morning.json: day, status, memory and history). It works through the business
tools on the smb-bench MCP server, at most 60 calls a day (advance_day is exempt), saves notes for tomorrow
with the memory tools, and ends the day with advance_day as the only tool call in its message. The verifier
checks the day and opens the next one.reward.json.The MCP server is a thin connector (oro_environment.mcp_proxy, standard library only) that forwards tool calls
to the hosted service. It holds no prompt, loop or model settings: the service serves the instructions each day.
advance_day; nothing closes a day
for it (see below).If a session ends with the day still open (your agent ended its turn without advance_day, crashed, or hit the
20-minute session limit), the next turn is another fresh session on the same day. Its message is the SMB Bench
leaderboard loop's nudge, "Call a tool, or advance_day to end the session.", with the day and how many of the day's
60 calls are used. A nudge session may still use the business tools within the day's remaining 60-call budget, as
after the leaderboard loop's nudge. A day gets at most 2 nudges, as many as the leaderboard loop's turn-limit tail
has turns (that tail accepts only advance_day; nudge sessions here do not enforce that). Each nudge is recorded in the previous session's verifier/nudge.json, as nudge in the
nudge session's reward.json, and in the final grade.json (nudges). Harbor also records a session's own error
(AgentTimeoutError, NonZeroAgentExitCodeError) on its step; Harbor marks a trial with a
NonZeroAgentExitCodeError step as reward_valid false even when the grade's valid is 1.
The run ends there if a day is still open after its second nudge (DAILY_ADVANCE_NOT_CALLED, the counterpart of
the leaderboard's DAILY_MODEL_TURN_LIMIT; this includes an advance_day the service keeps refusing), if your
agent plays past the end of its day, if the service keeps failing, or if a step cannot start (for example a wrong
token at provision). The verifier aborts the episode if one was created (which frees one of the grant's open
slots), writes early_stop.json, and writes the reason as {"failed": "<reason>"} in that step's reward.json
instead of a reward. Harbor then records the step's exception as VerifierOutputParseError: Non-numeric reward '<reason>', skips the remaining steps, and the trial has no reward.
Look for the reason in the failed step, not in Harbor's summary. Harbor 0.25 records a multi-step trial's failures
on its steps (step_results[].exception_info in the trial's result.json, and harbor view), while its summary
table counts only trial-level exceptions: a failed run prints Trials 0 | Exceptions 0 | Mean 0.000 and
harbor run exits 0. The step's verifier/test-stdout.txt (under jobs/<job>/<trial>/steps/<step>/) holds the
verifier's full output.
Your agent runs as the unprivileged user agent. The step verifier runs as root in the same sandbox with the
grant token ([verifier.env]): it kills every process of the agent user before it starts, creates the episode,
and gives the connector an episode token (/var/lib/oro/session.json), replaced at every step. An episode token can play the episode's
days but cannot finalize, abort, pause or resume it, or open, list or read other episodes. Before Harbor installs
the agent, the task locks root's password and removes set-user-ID programs, so the agent user cannot become root.
reward.json holds the grade's numbers:
| Key | Meaning |
|---|---|
reward |
Profit margin points over the fixed buying rule |
valid |
1 if every server-side check passed (regrade, accounting oracle, replay, public-surface audit) |
net_value_pct_br |
Your run's own margin |
business_return_cents |
Change in the business's net value over the quarter in your run |
reference_business_return_cents |
The same for the fixed buying rule |
gain_cents |
The difference between the two |
Net value is cash, receivables and sellable stock at cost, less bills, debt and the future-sales reserve. Margin
is its growth as a share of the business's sales potential: the revenue it would earn if it never ran out of
stock over the 90 days. A reward of +2.2 means that for every $100 of potential sales, your run finished with
$2.20 more net value than the fixed buying rule. The fixed buying rule buys from the cheapest supplier to cover
estimated demand over the delivery lead time plus about a day and a half, and never reacts to what happens
during the quarter. The full finalize reply is in the last step's verifier/grade.json.
Model cost depends on your agent and model. With codex and gpt-5.4-mini, a day cost about $0.03 and took
about 50 seconds including Harbor's per-step overhead, so a quarter is about $3 and 75 minutes. In ORO's test runs
that model ended its turn without calling advance_day about once in 50 to 60 days; each such day costs one or
two extra nudge sessions. Each day's agent may run for up to 20 minutes. A hosted episode lives at most 24 hours:
a slow agent that needs many nudges can reach that limit, and the run then fails.
The task files in this dataset are licensed under CC BY 4.0, like the published SMB Bench results on Hugging Face. Running the task requires a grant under ORO's lab access terms (https://lab-api.oroagents.com/v1/terms), which forbid training, fine-tuning or distilling any model on the tasks: what the service serves (the buyer instructions, morning briefings, tool results), the traces and the grades. The service keeps traces for 14 days; see the data policy at https://lab-api.oroagents.com/v1/data-policy.
SMB Bench gives an agent the buying desk of a simulated small business for one 90-day quarter. Each morning the agent reads the business's status and its own notes, works with the business tools, and ends the day. At the end of the quarter, ORO grades the run against a fixed buying rule on the same world. The simulator and the worlds stay on ORO's hosted lab service; this dataset only connects your Harbor agent to it.
You bring the agent and the model key. Results are runs with your own agent: they are not comparable with the SMB Bench leaderboard, which was produced with ORO's own buyer loop.
| Task | smb-bench-public-1: environment smb-bench-v1, 90 simulated days |
| Agent | any Harbor agent that supports MCP servers (for example codex or claude-code), any model |
| Reward | margin points over the fixed buying rule, from ORO's grader |
| Needs | an SMB Bench grant and your own model API key |
Email team@oroagents.com to request a grant. An operator issues a lab token, typically for 10 episodes with 3 open
at a time, as stated in your grant email; a grant lasts 7 days and can be renewed. Using a grant means accepting
ORO's lab access terms (https://lab-api.oroagents.com/v1/terms). Every Harbor trial uses one episode, including
trials that fail after the episode was created and every attempt (-k) or retry (-r). Keep -n at or below
your grant's open-episode limit.
uv tool install 'harbor[<extra>]==0.25.0' # your environment's extra (see Environments); Docker needs none
set -a; . ./smb-bench-grant.env; set +a # the grant file from your grant email: ORO_ENDPOINT and ORO_LAB_TOKEN
export <ENVIRONMENT_CREDENTIAL>=... # your environment's credentials (see Environments)
export OPENAI_API_KEY=... # your model key, used by your agent as usual
harbor run -y -d oro-agents/smb-bench@1.0.0 -e <environment> -a codex -m openai/gpt-5.4-mini -n 1
With E2B, for example, that is harbor[e2b], E2B_API_KEY and -e e2b.
Only the task's verifier receives ORO_LAB_TOKEN ([verifier.env] in task.toml). Never pass it to your agent
with --ae. ORO_ENDPOINT defaults to https://lab-api.oroagents.com. Set ORO_DECLARED_MODEL to record your
model's name with the episode. A downloaded copy of the dataset runs the same way with -p smb-bench-public-1 in
place of -d.
There is no graded result short of a full 90-day quarter: only the last step carries a reward, and a run that stops early has none.
The task limits your agent's network to ORO's lab service and your model's endpoint (network_mode = "allowlist") and keeps it open while Harbor installs the agent, so the environment must enforce an allowlist of
hostnames and switch it at run time. Harbor runs the task on any environment that can, and refuses it on the rest
before any agent runs, for example with ValueError: The execution phase network policy differs from a broader setup or shared phase policy, but this environment cannot change network policy after start. (harbor run exits
1).
In Harbor 0.25 these environments can run the task. Install Harbor with the environment's extra and set its credentials:
-e |
Install | Needs | Tested by ORO |
|---|---|---|---|
e2b |
harbor[e2b] |
E2B_API_KEY |
Yes: full quarters; during the agent phase the lab service answers and other hosts are blocked |
modal |
harbor[modal] |
MODAL_TOKEN_ID and MODAL_TOKEN_SECRET |
No |
daytona |
harbor[daytona] |
DAYTONA_API_KEY |
No |
beam |
harbor[beam] |
BEAM_TOKEN, or beam login |
No |
tensorlake |
harbor[tensorlake] |
TENSORLAKE_API_KEY |
No |
vercel |
harbor[vercel] |
VERCEL_TOKEN |
No |
novita |
harbor[novita] |
NOVITA_API_KEY |
No |
hyperbrowser |
harbor[hyperbrowser] |
HYPERBROWSER_API_KEY |
No |
islo |
harbor[islo] |
ISLO_API_KEY |
No |
runta |
harbor[runta] |
RUNTA_TOKEN |
No |
sbx (Docker Sandboxes, cloud) |
harbor[sbx] |
SBX_TOKEN |
No |
docker |
harbor |
a Docker host whose kernel supports nftables fib |
No; a host without fib refuses the task (tested) |
"No" means ORO has not played a quarter there. ORO ran the task on each cloud environment above with placeholder
credentials: Harbor accepted it and the run stopped only at the provider's sign-in. Harbor refuses the task on
prime (it needs a prebuilt image; this task builds its own), on runloop, mosaic, blaxel, langsmith and
smol (they cannot switch the allowlist at run time), and on environments without allowlists.
Press Ctrl-C once in the terminal running harbor run and let it stop its sandboxes (it records the trial as
CancelledError and exits 1). The episode stays open on
ORO's service and holds one of your grant's open slots until you abort it, even after its worker stops at the
24-hour limit. Its ID is in
jobs/<job>/<trial>/steps/provision/verifier/test-stdout.txt; to list your episodes and their states, or to abort
one:
curl -sH "Authorization: Bearer $ORO_LAB_TOKEN" "$ORO_ENDPOINT/v1/episodes"
curl -sX POST -H "Authorization: Bearer $ORO_LAB_TOKEN" -H "Content-Type: application/json" -d '{}' \
"$ORO_ENDPOINT/v1/episodes/EPISODE_ID/abort"
An aborted episode still counts toward the grant. A run that fails on its own aborts its episode. To check your
grant's limits (max_episodes, max_active, expires_at), call GET $ORO_ENDPOINT/v1/access. To see what you have
used, list your episodes as above: the ones not FINALIZED, ABORTED or EXPIRED count toward max_active. Every
episode counts toward max_episodes, including ones older than the 14-day trace window, which the list no longer
shows; if your grant was renewed, ask team@oroagents.com how many episodes remain.
Harbor's console doesn't show the simulated day. Each finished step writes its progress to reward.json, so the
latest one tells you where a run is:
ls -t jobs/*/*/steps/*/verifier/reward.json | head -1 | xargs cat
Codex may log failed to refresh available models at the start of each session. That message is harmless: the model
you chose with -m is still used.
The task has a provision step and up to 270 session-NNN steps: up to 3 agent sessions per simulated day. Each step
starts a fresh agent. A day that your agent ends on its first session uses one step, so a quarter normally runs
91 steps; Harbor skips the unused session steps once the quarter is graded. Each step's reward.json shows its progress:
day (the simulated day the session played), nudge (0 for the day's own session, 1 or 2 for a nudge session),
days_completed and running. Only the final step carries reward: the service's grade, which is the trial's
reward.
provision: your agent gets a one-line instruction and replies. The verifier then creates the episode on ORO's
service and opens day 1./var/lib/oro/today/prompt.md), this session's message (/var/lib/oro/today/message.md) and the morning
briefing (/var/lib/oro/today/morning.json: day, status, memory and history). It works through the business
tools on the smb-bench MCP server, at most 60 calls a day (advance_day is exempt), saves notes for tomorrow
with the memory tools, and ends the day with advance_day as the only tool call in its message. The verifier
checks the day and opens the next one.reward.json.The MCP server is a thin connector (oro_environment.mcp_proxy, standard library only) that forwards tool calls
to the hosted service. It holds no prompt, loop or model settings: the service serves the instructions each day.
advance_day; nothing closes a day
for it (see below).If a session ends with the day still open (your agent ended its turn without advance_day, crashed, or hit the
20-minute session limit), the next turn is another fresh session on the same day. Its message is the SMB Bench
leaderboard loop's nudge, "Call a tool, or advance_day to end the session.", with the day and how many of the day's
60 calls are used. A nudge session may still use the business tools within the day's remaining 60-call budget, as
after the leaderboard loop's nudge. A day gets at most 2 nudges, as many as the leaderboard loop's turn-limit tail
has turns (that tail accepts only advance_day; nudge sessions here do not enforce that). Each nudge is recorded in the previous session's verifier/nudge.json, as nudge in the
nudge session's reward.json, and in the final grade.json (nudges). Harbor also records a session's own error
(AgentTimeoutError, NonZeroAgentExitCodeError) on its step; Harbor marks a trial with a
NonZeroAgentExitCodeError step as reward_valid false even when the grade's valid is 1.
The run ends there if a day is still open after its second nudge (DAILY_ADVANCE_NOT_CALLED, the counterpart of
the leaderboard's DAILY_MODEL_TURN_LIMIT; this includes an advance_day the service keeps refusing), if your
agent plays past the end of its day, if the service keeps failing, or if a step cannot start (for example a wrong
token at provision). The verifier aborts the episode if one was created (which frees one of the grant's open
slots), writes early_stop.json, and writes the reason as {"failed": "<reason>"} in that step's reward.json
instead of a reward. Harbor then records the step's exception as VerifierOutputParseError: Non-numeric reward '<reason>', skips the remaining steps, and the trial has no reward.
Look for the reason in the failed step, not in Harbor's summary. Harbor 0.25 records a multi-step trial's failures
on its steps (step_results[].exception_info in the trial's result.json, and harbor view), while its summary
table counts only trial-level exceptions: a failed run prints Trials 0 | Exceptions 0 | Mean 0.000 and
harbor run exits 0. The step's verifier/test-stdout.txt (under jobs/<job>/<trial>/steps/<step>/) holds the
verifier's full output.
Your agent runs as the unprivileged user agent. The step verifier runs as root in the same sandbox with the
grant token ([verifier.env]): it kills every process of the agent user before it starts, creates the episode,
and gives the connector an episode token (/var/lib/oro/session.json), replaced at every step. An episode token can play the episode's
days but cannot finalize, abort, pause or resume it, or open, list or read other episodes. Before Harbor installs
the agent, the task locks root's password and removes set-user-ID programs, so the agent user cannot become root.
reward.json holds the grade's numbers:
| Key | Meaning |
|---|---|
reward |
Profit margin points over the fixed buying rule |
valid |
1 if every server-side check passed (regrade, accounting oracle, replay, public-surface audit) |
net_value_pct_br |
Your run's own margin |
business_return_cents |
Change in the business's net value over the quarter in your run |
reference_business_return_cents |
The same for the fixed buying rule |
gain_cents |
The difference between the two |
Net value is cash, receivables and sellable stock at cost, less bills, debt and the future-sales reserve. Margin
is its growth as a share of the business's sales potential: the revenue it would earn if it never ran out of
stock over the 90 days. A reward of +2.2 means that for every $100 of potential sales, your run finished with
$2.20 more net value than the fixed buying rule. The fixed buying rule buys from the cheapest supplier to cover
estimated demand over the delivery lead time plus about a day and a half, and never reacts to what happens
during the quarter. The full finalize reply is in the last step's verifier/grade.json.
Model cost depends on your agent and model. With codex and gpt-5.4-mini, a day cost about $0.03 and took
about 50 seconds including Harbor's per-step overhead, so a quarter is about $3 and 75 minutes. In ORO's test runs
that model ended its turn without calling advance_day about once in 50 to 60 days; each such day costs one or
two extra nudge sessions. Each day's agent may run for up to 20 minutes. A hosted episode lives at most 24 hours:
a slow agent that needs many nudges can reach that limit, and the run then fails.
The task files in this dataset are licensed under CC BY 4.0, like the published SMB Bench results on Hugging Face. Running the task requires a grant under ORO's lab access terms (https://lab-api.oroagents.com/v1/terms), which forbid training, fine-tuning or distilling any model on the tasks: what the service serves (the buyer instructions, morning briefings, tool results), the traces and the grades. The service keeps traces for 14 days; see the data policy at https://lab-api.oroagents.com/v1/data-policy.