orca-bench/orca-bench-private

harbor run -d orca-bench/orca-bench-private

orca-bench/orca-bench-private

ORCA-bench is an agent benchmark for root cause analysis.

This dataset contains the held-out private (answers removed) split of 324 tasks across 31 incidents (108 easy, 117 medium, 99 hard). For the results on all 1,079 tasks, please see https://arxiv.org/abs/2607.28545.

Quick Start

Harbor needs a one-time patch before the first run, and GradientAI needs a LiteLLM patch for reasoning_effort passthrough. Please follow the steps at Additional setup instructions for Harbor.

Export the OpenAI-specific environment variables, plus any for additional model providers:

# Required: used by the oracle solution and the verifier's LLM judge
export OPENAI_API_KEY=xxx
export OPENAI_BASE_URL=xxx
# Optional: environment variables to support additional model providers for the agent
export GRADIENT_AI_API_KEY=xxx

Pull the snapshot image to a local cache directory (only needs to be done once):

# Stage the snapshot's /app/ to a host-side cache keyed by image id.
SNAPSHOT_IMAGE=orcabench/sre-otel-snapshot:data-0418-harbor-template-v2
docker pull "$SNAPSHOT_IMAGE"
CACHE_DIR="$HOME/.cache/sre-snapshot-cache/$(docker image inspect "$SNAPSHOT_IMAGE" -f '{{.Id}}' | cut -d: -f2 | cut -c1-12)"
if [ -z "$(ls -A "$CACHE_DIR" 2>/dev/null)" ]; then
    mkdir -p "$CACHE_DIR"; CID=$(docker create "$SNAPSHOT_IMAGE"); docker cp "$CID:/app/." "$CACHE_DIR/"; docker rm "$CID"
fi

Run the Harbor trials with the cache directory bind-mounted:

SNAPSHOT_CACHE_HOST_DIR="$CACHE_DIR" uv run harbor run \
    --mounts-json "[{\"type\":\"bind\",\"source\":\"$CACHE_DIR\",\"target\":\"$CACHE_DIR\",\"read_only\":true}]" \
    -d orca-bench/orca-bench-private \
    -a terminus-2 \
    -m gradient_ai/openai-gpt-5.5 \
    --ak temperature=1 \
    --ak reasoning_effort=medium \
    --ak 'llm_kwargs={"max_tokens": 16384}'

For our experiments in ORCA-bench, we used GradientAI's serverless inference from DigitalOcean. If you use that provider, please see Additional setup instructions for Harbor -> "Using GradientAI's serverless inference from DigitalOcean".

This benchmark has been tested with the following operating system, Docker, Python, and Harbor combination:

Ubuntu 22.04.5 LTS, Docker 29.1.5, Python 3.13.11, Harbor 0.20.0

License

Shield:

This work is licensed under a Creative Commons Attribution 4.0 International License.