ITSMBench

53 multi-turn enterprise IT-service tasks, graded on the database the agent leaves behind — not on what it said.

Most agent benchmarks stop at the answer. Real organizations don't: work only counts once it has been carried out in a system of record, by someone allowed to do it, with a trail that survives an audit. ITSMBench measures that.

Each task drops the agent into a live IT Service Management tenant — around twenty interconnected tables covering incidents, SLA clocks, problems, change requests, a CMDB, a service catalog, knowledge articles, notifications, users, groups, and permissions. The agent gets a written policy it is held to, 93 typed tools, and a colleague on the other end of the conversation who only volunteers what they are asked for.

An episode

The agent talks to a simulated operator and works the tenant through tools:

1. start_conversation returns the operator's opening request — say, onboarding a new hire today.
2. The agent asks questions and calls tools. The operator holds private facts (the new hire's email, which site they sit at, the laptop's serial number) and surfaces them only when asked the right question.
3. The episode ends when either side says ###STOP###, or the turn budget runs out.

The tools behave like the real system: enum gates, required fields, referential integrity, and deterministic ids and timestamps so runs stay comparable.

Grading

Binary, and deliberately unforgiving:

reward = db_match × nl_assertions

• db_match — the task's gold action sequence is replayed on a fresh copy of the world, and the resulting database is compared to the agent's, record by record. Structured fields must match exactly; free-text fields are judged semantically, so reworded-but-correct prose passes.
• nl_assertions — an LLM judge grades the task's natural-language assertions against the transcript, covering what the database cannot express.
• infra_ok — 0 when the run was degraded by something that is not the agent's doing (no episode record collected, simulated-user outage, verifier error). Check it before reading a batch of zeros as model failure.
• episode_completed — 0 when the episode stopped rather than ended. The operator exists only behind send_message_to_user, so an agent that asks a question in plain text is waiting for a reply that cannot arrive, and the run dies where it stands. That scores 0 like any other failure while having measured much less. A batch whose mean here is below 1 is a batch to read carefully.

This catches things transcript grading misses. In one of our own runs the agent held a flawless conversation and the judge passed every assertion — but it had filed the new hire's laptop at the wrong site, and the database comparison caught it.

Running it

harbor dataset download vibrantlabsai/itsm-bench
harbor run -p <task-dir> -a claude-code -m <model>

Each task starts two containers: the agent's, and a sidecar that owns the database, the 93 tools, and the simulated operator, offered to the agent over MCP. Grading runs in a third.

Credentials the sidecar and judge need, supplied on the host:

Variable	Used for
ITSM_USER_MODEL, ITSM_USER_BASE_URL, ITSM_USER_API_KEY	the simulated operator
ITSM_JUDGE_MODEL, ITSM_JUDGE_BASE_URL, ITSM_JUDGE_API_KEY	the judge
ITSM_MAX_STEPS	turn budget (default 30)

For a judge-free smoke test, ITSM_SKIP_NL_ASSERTIONS=1 ITSM_DB_TEXT_MATCH=fuzzy harbor run -p <task-dir> --agent oracle replays the gold actions and must score 1.0.

Check episode_completed before you read the rewards

Harbor gives an agent one invocation, and CLI agents end their turn when they answer without calling a tool. Here the operator exists only behind send_message_to_user, so an agent that asks "shall I proceed?" as ordinary prose is waiting for a reply that cannot arrive — the process exits and the episode ends wherever it stood, scoring 0 like any other failure.

Measured on a 15-task run with Claude Opus 4.8 driven by -a claude-code, this ended 8 of 14 episodes early, one after a single step. Driving the same agent so that it resumes while the episode is still open took completion to 15/15 and the score from 0.00 to 0.33 on the same tasks — so on this agent the effect is most of the score.

episode_completed is reported per trial for exactly this reason. A batch whose mean is below 1 is measuring how often the agent stopped talking, not how well it did the work. Whether a given agent is affected depends on how it decides its turn is over; check the metric rather than assuming.

Worth knowing before you cite a number

• The agent has a shell, and that is fine. Nothing benchmark-relevant is reachable from its container: no task data, no scoring code, no credentials, not even the task id. The episode record is written inside the sidecar and collected by Harbor after the agent phase; the verifier runs in its own image. Each of those is a container boundary, and each was verified by running the corresponding attack and confirming it scores 0.
• 93 tools arrive as one MCP surface. That is the honest benchmark surface and it is identical across every port of this benchmark, so a model that does worse here than on a hand-trimmed toolset is telling you something real.
• The operator answers only when asked — it runs inside a tool call and cannot interject.
• The turn budget is 30 by default (ITSM_MAX_STEPS), and both tool calls and messages to the operator spend a turn. It was 12 through v1.2.0, which starved the longer tasks: itsmbench_single_007 needs 16 gold actions alone, and in one measured run an agent was cut one call short of finishing. Numbers produced under a different budget are not comparable to these.

Credits

Built by Vibrant Labs as part of Enterprise-Worlds. Seed databases and tool surface adapted from ServiceNow's EnterpriseOps-Gym (Apache-2.0); verifier design in the style of τ-bench. Licensed Apache-2.0.

The same benchmark is also published as a Prime Intellect environment and as a NeMo Gym environment. All three share one scoring core and one set of gold data, and replaying a fixed rollout through them produces the same reward — that is what we mean by parity, and we test it.

Scores from live runs are not comparable across the three, and we would rather say so than let a number travel. Each harness drives the agent differently, and how an agent spends its turn budget depends on the scaffolding around it. Compare models within one harness, not one model across two.