orca-bench/orca-bench-verified

ORCA-bench is an agent benchmark for root cause analysis.

This dataset contains the verified split of 40 tasks across 40 incidents (10 easy, 15 medium, 15 hard). For the results on all 1,079 tasks, please see https://arxiv.org/abs/2607.28545.

Quick Start

Harbor needs a one-time patch before the first run, and GradientAI needs a LiteLLM patch for reasoning_effort passthrough. Please follow the steps at Additional setup instructions for Harbor.

Export the OpenAI-specific environment variables, plus any for additional model providers:

# Required: used by the oracle solution and the verifier's LLM judge
export OPENAI_API_KEY=xxx
export OPENAI_BASE_URL=xxx
# Optional: environment variables to support additional model providers for the agent
export GRADIENT_AI_API_KEY=xxx

Pull the snapshot image to a local cache directory (only needs to be done once):

# Stage the snapshot's /app/ to a host-side cache keyed by image id.
SNAPSHOT_IMAGE=orcabench/sre-otel-snapshot:data-0418-harbor-template-v2
docker pull "$SNAPSHOT_IMAGE"
CACHE_DIR="$HOME/.cache/sre-snapshot-cache/$(docker image inspect "$SNAPSHOT_IMAGE" -f '{{.Id}}' | cut -d: -f2 | cut -c1-12)"
if [ -z "$(ls -A "$CACHE_DIR" 2>/dev/null)" ]; then
    mkdir -p "$CACHE_DIR"; CID=$(docker create "$SNAPSHOT_IMAGE"); docker cp "$CID:/app/." "$CACHE_DIR/"; docker rm "$CID"
fi

Run the Harbor trials with the cache directory bind-mounted:

SNAPSHOT_CACHE_HOST_DIR="$CACHE_DIR" uv run harbor run \
    --mounts-json "[{\"type\":\"bind\",\"source\":\"$CACHE_DIR\",\"target\":\"$CACHE_DIR\",\"read_only\":true}]" \
    -d orca-bench/orca-bench-verified \
    -a terminus-2 \
    -m gradient_ai/openai-gpt-5.5 \
    --ak temperature=1 \
    --ak reasoning_effort=medium \
    --ak 'llm_kwargs={"max_tokens": 16384}'

For our experiments in ORCA-bench, we used GradientAI's serverless inference from DigitalOcean. If you use that provider, please see Additional setup instructions for Harbor -> "Using GradientAI's serverless inference from DigitalOcean".

This benchmark has been tested with the following operating system, Docker, Python, and Harbor combination:

• Ubuntu 22.04.5 LTS, Docker 29.1.5, Python 3.13.11, Harbor 0.20.0

License

Shield:

This work is licensed under a Creative Commons Attribution 4.0 International License.