Loading dataset details…
harbor run -d vibrantlabsai/itsm-benchharbor run -d vibrantlabsai/itsm-benchITSMBench: enterprise ITSM agent tasks graded on database state
harbor run -d vibrantlabsai/itsm-benchITSMBench: enterprise ITSM agent tasks graded on database state
harbor run -d vibrantlabsai/itsm-bench53 multi-turn enterprise IT-service tasks, graded on the database the agent leaves behind — not on what it said.
Most agent benchmarks stop at the answer. Real organizations don't: work only counts once it has been carried out in a system of record, by someone allowed to do it, with a trail that survives an audit. ITSMBench measures that.
Each task drops the agent into a live IT Service Management tenant — around twenty interconnected tables covering incidents, SLA clocks, problems, change requests, a CMDB, a service catalog, knowledge articles, notifications, users, groups, and permissions. The agent gets a written policy it is held to, 93 typed tools, and a colleague on the other end of the conversation who only volunteers what they are asked for.
The agent talks to a simulated operator and works the tenant through tools:
start_conversation returns the operator's opening request — say, onboarding a new
hire today.###STOP###, or the turn budget runs out.The tools behave like the real system: enum gates, required fields, referential integrity, and deterministic ids and timestamps so runs stay comparable.
Binary, and deliberately unforgiving:
reward = db_match × nl_assertions
db_match — the task's gold action sequence is replayed on a fresh copy of the
world, and the resulting database is compared to the agent's, record by record.
Structured fields must match exactly; free-text fields are judged semantically, so
reworded-but-correct prose passes.nl_assertions — an LLM judge grades the task's natural-language assertions
against the transcript, covering what the database cannot express.infra_ok — 0 when the run was degraded by something that is not the agent's
doing (no episode record collected, simulated-user outage, verifier error). Check it
before reading a batch of zeros as model failure.episode_completed — 0 when the episode stopped rather than ended. The operator
exists only behind send_message_to_user, so an agent that asks a question in plain
text is waiting for a reply that cannot arrive, and the run dies where it stands.
That scores 0 like any other failure while having measured much less. A batch whose
mean here is below 1 is a batch to read carefully.This catches things transcript grading misses. In one of our own runs the agent held a flawless conversation and the judge passed every assertion — but it had filed the new hire's laptop at the wrong site, and the database comparison caught it.
harbor dataset download vibrantlabsai/itsm-bench
harbor run -p <task-dir> -a claude-code -m <model>Each task starts two containers: the agent's, and a sidecar that owns the database, the 93 tools, and the simulated operator, offered to the agent over MCP. Grading runs in a third.
Credentials the sidecar and judge need, supplied on the host:
| Variable | Used for |
|---|---|
ITSM_USER_MODEL, ITSM_USER_BASE_URL, ITSM_USER_API_KEY |
the simulated operator |
ITSM_JUDGE_MODEL, ITSM_JUDGE_BASE_URL, ITSM_JUDGE_API_KEY |
the judge |
ITSM_MAX_STEPS |
turn budget (default 30) |
For a judge-free smoke test, ITSM_SKIP_NL_ASSERTIONS=1 ITSM_DB_TEXT_MATCH=fuzzy harbor run -p <task-dir> --agent oracle replays the gold actions and must score 1.0.
episode_completed before you read the rewardsHarbor gives an agent one invocation, and CLI agents end their turn when they answer
without calling a tool. Here the operator exists only behind send_message_to_user, so
an agent that asks "shall I proceed?" as ordinary prose is waiting for a reply that
cannot arrive — the process exits and the episode ends wherever it stood, scoring 0
like any other failure.
Measured on a 15-task run with Claude Opus 4.8 driven by -a claude-code, this ended
8 of 14 episodes early, one after a single step. Driving the same agent so that it
resumes while the episode is still open took completion to 15/15 and the score from
0.00 to 0.33 on the same tasks — so on this agent the effect is most of the score.
episode_completed is reported per trial for exactly this reason. A batch whose mean
is below 1 is measuring how often the agent stopped talking, not how well it did the
work. Whether a given agent is affected depends on how it decides its turn is over;
check the metric rather than assuming.
ITSM_MAX_STEPS), and both tool calls and
messages to the operator spend a turn. It was 12 through v1.2.0, which starved the
longer tasks: itsmbench_single_007 needs 16 gold actions alone, and in one measured
run an agent was cut one call short of finishing. Numbers produced under a different
budget are not comparable to these.Built by Vibrant Labs as part of Enterprise-Worlds. Seed databases and tool surface adapted from ServiceNow's EnterpriseOps-Gym (Apache-2.0); verifier design in the style of τ-bench. Licensed Apache-2.0.
The same benchmark is also published as a Prime Intellect environment and as a NeMo Gym environment. All three share one scoring core and one set of gold data, and replaying a fixed rollout through them produces the same reward — that is what we mean by parity, and we test it.
Scores from live runs are not comparable across the three, and we would rather say so than let a number travel. Each harness drives the agent differently, and how an agent spends its turn budget depends on the scaffolding around it. Compare models within one harness, not one model across two.
53 multi-turn enterprise IT-service tasks, graded on the database the agent leaves behind — not on what it said.
Most agent benchmarks stop at the answer. Real organizations don't: work only counts once it has been carried out in a system of record, by someone allowed to do it, with a trail that survives an audit. ITSMBench measures that.
Each task drops the agent into a live IT Service Management tenant — around twenty interconnected tables covering incidents, SLA clocks, problems, change requests, a CMDB, a service catalog, knowledge articles, notifications, users, groups, and permissions. The agent gets a written policy it is held to, 93 typed tools, and a colleague on the other end of the conversation who only volunteers what they are asked for.
The agent talks to a simulated operator and works the tenant through tools:
start_conversation returns the operator's opening request — say, onboarding a new
hire today.###STOP###, or the turn budget runs out.The tools behave like the real system: enum gates, required fields, referential integrity, and deterministic ids and timestamps so runs stay comparable.
Binary, and deliberately unforgiving:
reward = db_match × nl_assertions
db_match — the task's gold action sequence is replayed on a fresh copy of the
world, and the resulting database is compared to the agent's, record by record.
Structured fields must match exactly; free-text fields are judged semantically, so
reworded-but-correct prose passes.nl_assertions — an LLM judge grades the task's natural-language assertions
against the transcript, covering what the database cannot express.infra_ok — 0 when the run was degraded by something that is not the agent's
doing (no episode record collected, simulated-user outage, verifier error). Check it
before reading a batch of zeros as model failure.episode_completed — 0 when the episode stopped rather than ended. The operator
exists only behind send_message_to_user, so an agent that asks a question in plain
text is waiting for a reply that cannot arrive, and the run dies where it stands.
That scores 0 like any other failure while having measured much less. A batch whose
mean here is below 1 is a batch to read carefully.This catches things transcript grading misses. In one of our own runs the agent held a flawless conversation and the judge passed every assertion — but it had filed the new hire's laptop at the wrong site, and the database comparison caught it.
harbor dataset download vibrantlabsai/itsm-bench
harbor run -p <task-dir> -a claude-code -m <model>Each task starts two containers: the agent's, and a sidecar that owns the database, the 93 tools, and the simulated operator, offered to the agent over MCP. Grading runs in a third.
Credentials the sidecar and judge need, supplied on the host:
| Variable | Used for |
|---|---|
ITSM_USER_MODEL, ITSM_USER_BASE_URL, ITSM_USER_API_KEY |
the simulated operator |
ITSM_JUDGE_MODEL, ITSM_JUDGE_BASE_URL, ITSM_JUDGE_API_KEY |
the judge |
ITSM_MAX_STEPS |
turn budget (default 30) |
For a judge-free smoke test, ITSM_SKIP_NL_ASSERTIONS=1 ITSM_DB_TEXT_MATCH=fuzzy harbor run -p <task-dir> --agent oracle replays the gold actions and must score 1.0.
episode_completed before you read the rewardsHarbor gives an agent one invocation, and CLI agents end their turn when they answer
without calling a tool. Here the operator exists only behind send_message_to_user, so
an agent that asks "shall I proceed?" as ordinary prose is waiting for a reply that
cannot arrive — the process exits and the episode ends wherever it stood, scoring 0
like any other failure.
Measured on a 15-task run with Claude Opus 4.8 driven by -a claude-code, this ended
8 of 14 episodes early, one after a single step. Driving the same agent so that it
resumes while the episode is still open took completion to 15/15 and the score from
0.00 to 0.33 on the same tasks — so on this agent the effect is most of the score.
episode_completed is reported per trial for exactly this reason. A batch whose mean
is below 1 is measuring how often the agent stopped talking, not how well it did the
work. Whether a given agent is affected depends on how it decides its turn is over;
check the metric rather than assuming.
ITSM_MAX_STEPS), and both tool calls and
messages to the operator spend a turn. It was 12 through v1.2.0, which starved the
longer tasks: itsmbench_single_007 needs 16 gold actions alone, and in one measured
run an agent was cut one call short of finishing. Numbers produced under a different
budget are not comparable to these.Built by Vibrant Labs as part of Enterprise-Worlds. Seed databases and tool surface adapted from ServiceNow's EnterpriseOps-Gym (Apache-2.0); verifier design in the style of τ-bench. Licensed Apache-2.0.
The same benchmark is also published as a Prime Intellect environment and as a NeMo Gym environment. All three share one scoring core and one set of gold data, and replaying a fixed rollout through them produces the same reward — that is what we mean by parity, and we test it.
Scores from live runs are not comparable across the three, and we would rather say so than let a number travel. Each harness drives the agent differently, and how an agent spends its turn budget depends on the scaffolding around it. Compare models within one harness, not one model across two.