harbor run -d servicenow-ai-org/enterpriseops-automationbenchharbor run -d servicenow-ai-org/enterpriseops-automationbench100 stateful enterprise workflow tasks.
harbor run -d servicenow-ai-org/enterpriseops-automationbench100 stateful enterprise workflow tasks.
harbor run -d servicenow-ai-org/enterpriseops-automationbenchThis dataset contains 100 stateful workflow tasks: 34 CSM v3, 33 HR EnterpriseOps
v1, and 33 ITSM v7. A task's frozen request and seeded world run in isolated
containers. Agents use the world's MCP tools to search, fetch, and act across
connected apps; the verifier checks the final state. reward gives fractional
credit and task_completed_correctly reports exact completion.
linux/amd64 images.WORK="$HOME/automationbench-public"
mkdir -p "$WORK"
harbor download \
'servicenow-ai-org/enterpriseops-automationbench@1.0.0' \
--output-dir "$WORK"
TASK="$WORK/enterpriseops-automationbench/automationbench-v7-itsm-change_close_review_v7"
harbor run --path "$TASK" --agent nop --env docker \
--n-concurrent 1 --jobs-dir "$WORK/jobs" --job-name smoke-itsm
The no-op agent makes no changes; a reward of zero is expected. Confirm one
graded trial and zero trial errors in $WORK/jobs/smoke-itsm/result.json.
The same downloaded directory contains all 100 tasks.
Provide a supported model key in a restricted env file, then use an MCP-capable
agent. For example, with OPENAI_API_KEY in $WORK/model.env:
harbor run --path "$TASK" --agent codex --model openai/YOUR_MODEL_ID \
--env docker --env-file "$WORK/model.env" \
--agent-setup-timeout-multiplier 5 --n-concurrent 1 \
--jobs-dir "$WORK/jobs" --job-name model-itsm
Start with one task and check the agent trajectory for MCP calls as well as the verifier result. Other MCP-capable Harbor agents can use these tasks, but agent choice affects scores. Increase concurrency only after confirming image pulls, world seeding, tool calls, grading, and Docker network capacity.
This dataset contains 100 stateful workflow tasks: 34 CSM v3, 33 HR EnterpriseOps
v1, and 33 ITSM v7. A task's frozen request and seeded world run in isolated
containers. Agents use the world's MCP tools to search, fetch, and act across
connected apps; the verifier checks the final state. reward gives fractional
credit and task_completed_correctly reports exact completion.
linux/amd64 images.WORK="$HOME/automationbench-public"
mkdir -p "$WORK"
harbor download \
'servicenow-ai-org/enterpriseops-automationbench@1.0.0' \
--output-dir "$WORK"
TASK="$WORK/enterpriseops-automationbench/automationbench-v7-itsm-change_close_review_v7"
harbor run --path "$TASK" --agent nop --env docker \
--n-concurrent 1 --jobs-dir "$WORK/jobs" --job-name smoke-itsm
The no-op agent makes no changes; a reward of zero is expected. Confirm one
graded trial and zero trial errors in $WORK/jobs/smoke-itsm/result.json.
The same downloaded directory contains all 100 tasks.
Provide a supported model key in a restricted env file, then use an MCP-capable
agent. For example, with OPENAI_API_KEY in $WORK/model.env:
harbor run --path "$TASK" --agent codex --model openai/YOUR_MODEL_ID \
--env docker --env-file "$WORK/model.env" \
--agent-setup-timeout-multiplier 5 --n-concurrent 1 \
--jobs-dir "$WORK/jobs" --job-name model-itsm
Start with one task and check the agent trajectory for MCP calls as well as the verifier result. Other MCP-capable Harbor agents can use these tasks, but agent choice affects scores. Increase concurrency only after confirming image pulls, world seeding, tool calls, grading, and Docker network capacity.