biopharma-bench/biopharma-bench

harbor run -d biopharma-bench/biopharma-bench

Biopharma Bench

This is Havenor Therapeutics, the public company in Biopharma Bench, Raycaster's benchmark of document work inside pharmaceutical and medical-device companies.

Havenor makes sterile injectable drugs. The company is fictional. The products, the people, and the files were written for this benchmark.

There are ten assignments. Each one places an agent in an employee's seat, with that person's files, and asks for one piece of finished work. The date inside the company is the morning of 27 September 2026. Work scheduled after that morning has not been done. The checks used to grade the work are included with each assignment.

Assignments

Prepare the next AVI-401 stability commitment update.
State the technical position on the alternate overwrap change.
Stand behind the AVI-401 sterile-filter validation package.
Draft the response to the health-authority information request.
Check the records before submission.
Check the integrity of the container-closure validation package.
Check the supplier and change records.
Confirm whether the AVI-355 stability results meet the specification in force.
Bring the AVI-401 24-month pull control workbook through QC review.
Issue the QC decision for the AVI-401 24-month campaign.

Run

harbor run -d biopharma-bench/biopharma-bench@v0.1 -a <agent> -m <model>

For example, Claude Code with an Anthropic API key:

harbor run -d biopharma-bench/biopharma-bench@v0.1 -a claude-code -m claude-sonnet-5

Or Claude Code on Amazon Bedrock, with AWS_BEARER_TOKEN_BEDROCK and AWS_REGION set:

harbor run -d biopharma-bench/biopharma-bench@v0.1 -a claude-code -m us.anthropic.claude-sonnet-5 --ae CLAUDE_CODE_USE_BEDROCK=1

To run one assignment, filter by its full name, for example -i 'biopharma-bench/hvn-05'. Harbor asks before it passes your credentials to the grader; add -y to confirm in advance when running from a script.

Tools

Beside the files, each assignment runs an MCP server with the company's systems of record (ERP, QMS, document management, staff directory), its LIMS, and its instant messenger. Some checks need facts that are only in those systems, so an agent without an MCP client cannot pass them. Claude Code and Codex connect to it automatically; Harbor's pi agent has no MCP client.

Colleagues on the messenger answer through a language model. Set these three variables in the shell you run harbor from, pointing at any OpenAI-compatible /chat/completions endpoint:

export PERSONA_BASE_URL=https://api.openai.com/v1
export PERSONA_API_KEY=...
export PERSONA_MODEL=gpt-5.6-sol

Without them, messages to colleagues return an error; the files, records, and LIMS still work.

The same variables reach remote backends such as Daytona. Harbor also passes them into the agent's container, so the agent can read the persona key. Use a dedicated key with a low spending limit.

Grading

The work is graded by an agent judge running in the verifier. It uses whichever credential is in your environment: ANTHROPIC_API_KEY or AWS_BEARER_TOKEN_BEDROCK with AWS_REGION (Claude Code, Claude Opus 5.5), or OPENAI_API_KEY (Codex, GPT-6 Sol). Set GRADER_HARNESS and GRADER_MODEL to choose another judge.

The scored benchmark covers twelve companies. Havenor is the company whose files are public. Results are on the benchmark page. The written account is the announcement.