Loading dataset details…
harbor run -d victorojewale/abstention-actharbor run -d victorojewale/abstention-actCompliance-bias benchmark: safe action, clarification, and abstention
harbor run -d victorojewale/abstention-actCompliance-bias benchmark: safe action, clarification, and abstention
harbor run -d victorojewale/abstention-actA benchmark for compliance bias in tool-using agents.
We define compliance bias as the tendency to follow instructions even when the information, evidence, or authority needed to act safely is missing. AbstentionAct tests whether agents recognise when to pause while still completing tasks where they should proceed.
In simulated HR and DevOps systems, agents face requests that may leave the database target unclear, require authority they do not have, or ask for confirmation of a backup whose status is unavailable. The agent has to use what it finds in the system to decide what it can safely do and report.
The tasks cover three reasons an agent should pause, plus safe tasks where it should proceed:
| Gap | What is missing | Expected behavior |
|---|---|---|
| Specification | ambiguous / incomplete target | ask for clarification |
| Verification | no reliable confirmation of the result | make reasonable checks, then explain what remains unknown |
| Authority | insufficient authorization | hand off / escalate |
| Safe (control) | nothing needed for safe action is missing | proceed and complete |
The safe tasks are included because refusing everything should not count as good performance.
The main scores are:
Action checks use the protected service log, while two verification tasks also use an LLM judge to check unsupported success claims, as described in the scoring guide.
When reporting results, use the summary tool because Harbor's default SR/UR display can include rewards from timed-out trials. The tool excludes failed trials from SR/UR while reporting coverage and retaining their safety verdicts.
There are 22 tasks across HR and DevOps: 16 hazardous tasks and six safe controls.
| Domain | Task count | Tools |
|---|---|---|
| HR operations | 11 | payroll, ad-user, ticketing API, psql |
| DevOps | 11 | dbctl, serverctl, psql |
All tasks run in simulated systems whose tools deliberately allow unsafe actions, so the benchmark tests whether the agent makes a safe decision rather than whether a tool blocks it.
The matched pairs, ag-dev-001/safe-dev-001 and ag-hr-002/safe-hr-003,
keep the request unchanged while varying the agent's authority.
The benchmark is available on Harbor under the versioned dataset:
victorojewale/abstention-act@v0.16-rc1.
You need Docker and Harbor CLI 0.18 or newer.
uv tool install harbor # or: pip install harbor
harbor auth login # GitHub sign-in
mkdir -p ~/harbor-runs && cd ~/harbor-runsYou can use a model provider directly or through a service such as OpenRouter, provided your chosen agent supports it. Configure the credentials for that agent and provider as described in Harbor's setup guide.
# First, export your provider's credentials in this terminal.
# Replace the model placeholder with an identifier supported by your chosen agent.
MODEL="<provider>/<model>"
AGENT="terminus-2" # example agent; choose a compatible Harbor agent
# Configure the claim judge separately from the assistant under test.
export ABSTENTIONACT_JUDGE_MODEL="<judge-model>"
export ABSTENTIONACT_JUDGE_API_BASE="<https-chat-completions-api-base>"
# Set ABSTENTIONACT_JUDGE_API_KEY through your credential setup.
harbor run -d "victorojewale/abstention-act@v0.16-rc1" -a "$AGENT" -m "$MODEL" -k 3MODEL is the full identifier expected by the agent, including any provider or routing
prefix, so changing providers means updating both that identifier and the credentials.
Use -k 3 for three attempts per task and -n <count> to set concurrency to suit your
machine and provider limits, keeping a record of the dataset version, agent, model,
and settings when comparing runs.
Results are saved under ./jobs by default; use --jobs-dir <directory> to change it.
Inspect them with harbor view ./jobs and use the summary tool linked above for SR/UR
with failure handling and coverage.
Run the reference solutions with the same judge configuration, which uses the judge API on two tasks without running an assistant model:
harbor run -d "victorojewale/abstention-act@v0.16-rc1" -a oracleThe tools/ scripts provide optional run and reporting helpers, with OpenRouter support for budget-controlled runs.
Tests check the agent's action log and responses for safety, task completion, and unsupported success claims.
The agent runs without root privileges, while a separate privileged service handles database writes and records the action log to protect the evidence used for scoring.
Each task contains task.toml, instruction.md, an environment/ directory with its
Dockerfile and starting data, scoring tests under tests/, and a reference solution at
solution/solve.sh. Scoring runs in a separate verifier environment.
SR measures safe outcomes rather than whether the agent understood why it should stop and should be read alongside UR, since blanket refusal can score well on SR while failing the safe controls. Coverage is limited to 16 hazardous tasks and six safe controls in two domains, rather than agent safety in general.
Use Python 3.11 or newer for the local checks.
python3 tools/local_test.py # offline code and scoring checks (no Docker)
python3 tools/check_oracle.py --report /tmp/oracle-report.json
python3 tools/summarize_run.py ~/harbor-runs/jobs/* --expected-tasks 22 --expected-repeats 3If you use AbstentionAct, please cite the benchmark:
@software{ojewale_abstentionact_2026,
author = {Ojewale, Victor},
title = {{AbstentionAct}: A Benchmark for Compliance Bias in Tool-Using Agents},
year = {2026},
version = {v0.16-rc1},
url = {https://github.com/victorojewale/abstention-act}
}
For the compliance-bias framework and gap taxonomy, cite Designing for Doubt, published at AIES 2026:
@article{ojewale2026designingfordoubt,
author = {Ojewale, Victor and Venkatasubramanian, Suresh},
title = {Designing for Doubt: The Case for Informed Abstention in Autonomous Agents},
journal = {Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society},
volume = {9},
number = {3},
pages = {1960--1971},
year = {2026},
doi = {10.1609/aies.v9i3.43281},
url = {https://doi.org/10.1609/aies.v9i3.43281}
}
Citation metadata is also available in CITATION.cff.
AbstentionAct is available under the Apache-2.0 license.
Questions or bug reports: open an issue. Maintainer: Victor Ojewale (Brown University).
A benchmark for compliance bias in tool-using agents.
We define compliance bias as the tendency to follow instructions even when the information, evidence, or authority needed to act safely is missing. AbstentionAct tests whether agents recognise when to pause while still completing tasks where they should proceed.
In simulated HR and DevOps systems, agents face requests that may leave the database target unclear, require authority they do not have, or ask for confirmation of a backup whose status is unavailable. The agent has to use what it finds in the system to decide what it can safely do and report.
The tasks cover three reasons an agent should pause, plus safe tasks where it should proceed:
| Gap | What is missing | Expected behavior |
|---|---|---|
| Specification | ambiguous / incomplete target | ask for clarification |
| Verification | no reliable confirmation of the result | make reasonable checks, then explain what remains unknown |
| Authority | insufficient authorization | hand off / escalate |
| Safe (control) | nothing needed for safe action is missing | proceed and complete |
The safe tasks are included because refusing everything should not count as good performance.
The main scores are:
Action checks use the protected service log, while two verification tasks also use an LLM judge to check unsupported success claims, as described in the scoring guide.
When reporting results, use the summary tool because Harbor's default SR/UR display can include rewards from timed-out trials. The tool excludes failed trials from SR/UR while reporting coverage and retaining their safety verdicts.
There are 22 tasks across HR and DevOps: 16 hazardous tasks and six safe controls.
| Domain | Task count | Tools |
|---|---|---|
| HR operations | 11 | payroll, ad-user, ticketing API, psql |
| DevOps | 11 | dbctl, serverctl, psql |
All tasks run in simulated systems whose tools deliberately allow unsafe actions, so the benchmark tests whether the agent makes a safe decision rather than whether a tool blocks it.
The matched pairs, ag-dev-001/safe-dev-001 and ag-hr-002/safe-hr-003,
keep the request unchanged while varying the agent's authority.
The benchmark is available on Harbor under the versioned dataset:
victorojewale/abstention-act@v0.16-rc1.
You need Docker and Harbor CLI 0.18 or newer.
uv tool install harbor # or: pip install harbor
harbor auth login # GitHub sign-in
mkdir -p ~/harbor-runs && cd ~/harbor-runsYou can use a model provider directly or through a service such as OpenRouter, provided your chosen agent supports it. Configure the credentials for that agent and provider as described in Harbor's setup guide.
# First, export your provider's credentials in this terminal.
# Replace the model placeholder with an identifier supported by your chosen agent.
MODEL="<provider>/<model>"
AGENT="terminus-2" # example agent; choose a compatible Harbor agent
# Configure the claim judge separately from the assistant under test.
export ABSTENTIONACT_JUDGE_MODEL="<judge-model>"
export ABSTENTIONACT_JUDGE_API_BASE="<https-chat-completions-api-base>"
# Set ABSTENTIONACT_JUDGE_API_KEY through your credential setup.
harbor run -d "victorojewale/abstention-act@v0.16-rc1" -a "$AGENT" -m "$MODEL" -k 3MODEL is the full identifier expected by the agent, including any provider or routing
prefix, so changing providers means updating both that identifier and the credentials.
Use -k 3 for three attempts per task and -n <count> to set concurrency to suit your
machine and provider limits, keeping a record of the dataset version, agent, model,
and settings when comparing runs.
Results are saved under ./jobs by default; use --jobs-dir <directory> to change it.
Inspect them with harbor view ./jobs and use the summary tool linked above for SR/UR
with failure handling and coverage.
Run the reference solutions with the same judge configuration, which uses the judge API on two tasks without running an assistant model:
harbor run -d "victorojewale/abstention-act@v0.16-rc1" -a oracleThe tools/ scripts provide optional run and reporting helpers, with OpenRouter support for budget-controlled runs.
Tests check the agent's action log and responses for safety, task completion, and unsupported success claims.
The agent runs without root privileges, while a separate privileged service handles database writes and records the action log to protect the evidence used for scoring.
Each task contains task.toml, instruction.md, an environment/ directory with its
Dockerfile and starting data, scoring tests under tests/, and a reference solution at
solution/solve.sh. Scoring runs in a separate verifier environment.
SR measures safe outcomes rather than whether the agent understood why it should stop and should be read alongside UR, since blanket refusal can score well on SR while failing the safe controls. Coverage is limited to 16 hazardous tasks and six safe controls in two domains, rather than agent safety in general.
Use Python 3.11 or newer for the local checks.
python3 tools/local_test.py # offline code and scoring checks (no Docker)
python3 tools/check_oracle.py --report /tmp/oracle-report.json
python3 tools/summarize_run.py ~/harbor-runs/jobs/* --expected-tasks 22 --expected-repeats 3If you use AbstentionAct, please cite the benchmark:
@software{ojewale_abstentionact_2026,
author = {Ojewale, Victor},
title = {{AbstentionAct}: A Benchmark for Compliance Bias in Tool-Using Agents},
year = {2026},
version = {v0.16-rc1},
url = {https://github.com/victorojewale/abstention-act}
}
For the compliance-bias framework and gap taxonomy, cite Designing for Doubt, published at AIES 2026:
@article{ojewale2026designingfordoubt,
author = {Ojewale, Victor and Venkatasubramanian, Suresh},
title = {Designing for Doubt: The Case for Informed Abstention in Autonomous Agents},
journal = {Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society},
volume = {9},
number = {3},
pages = {1960--1971},
year = {2026},
doi = {10.1609/aies.v9i3.43281},
url = {https://doi.org/10.1609/aies.v9i3.43281}
}
Citation metadata is also available in CITATION.cff.
AbstentionAct is available under the Apache-2.0 license.
Questions or bug reports: open an issue. Maintainer: Victor Ojewale (Brown University).