Alibaba CTF Benchmark

Alibaba CTF Benchmark is a benchmark dataset of 87 Capture-The-Flag tasks curated from the Alibaba CTF (formerly Aliyun CTF) competition series (2023–2026). Sourced from real competition challenges calibrated against human CTF teams, it is used to evaluate agent and model competency on end-to-end security work across Web, Pwn, Reverse, Crypto, and Misc.

Alibaba CTF Benchmark is built on the Harbor framework and will be continuously updated with new tasks from future AlibabaCTF editions.

System requirements

Running evaluations locally requires Docker and an x86_64 Linux host (most CTF binaries are x86_64 ELF). A full run of all 87 tasks with qwen3.8-max at --n-concurrent 4 took 65 hours in total, consuming 1484M tokens.

Resource	Recommended (--n-concurrent 4)
Docker	Installed and running
harbor cli	Installed and running
Free disk space	100 GB (images + agent workspaces + logs)
System RAM	64 GB
CPU cores	8 cores
Architecture	x86_64 Linux
Network	Required (images pulled from ACR on first run)

Dataset composition

The dataset contains 87 tasks sourced from the 2023–2026 AlibabaCTF editions, each packaged as a self-contained Docker environment with a reference oracle solution and an automated flag verifier.

By year and category

Year	Crypto	Misc	Pwn	Reverse	Web	Total
2023	2	3	3	2	6	16
2024	3	3	2	—	3	11
2025	3	4	2	2	4	15
2026	5	4	12	12	12	45
Total	13	14	19	16	25	87

By difficulty

Difficulty	Crypto	Misc	Pwn	Reverse	Web	Total	Share
Easy	—	3	3	—	6	12	14%
Medium	12	9	15	13	18	67	77%
Hard	—	2	1	3	1	7	8%
Very Hard	1	—	—	—	—	1	1%
Total	13	14	19	16	25	87	100%

Technical coverage

Web — 25 tasks SQL/command/LDAP injection, deserialization (Java/Python/PHP), SSTI (Jinja2/Freemarker/Velocity), JWT forgery, SSRF, XXE, path traversal, XSS, CSP bypass, TOCTOU race

Pwn — 19 tasks Stack overflow, ROP, use-after-free, double-free, tcache poisoning, heap feng shui, format string, kernel Pwn, Rust/Go binaries, embedded CGI

Reverse — 16 tasks VM obfuscation, control-flow flattening, anti-debug, packing, symbolic execution, APK reversing, firmware, graphics/pixel recovery

Crypto — 13 tasks RSA (Coppersmith/low-exponent/common-modulus), ECC (ECDLP/invalid-curve/PRNG), Diffie-Hellman, AEAD, block cipher modes, ZKP forgery, side-channel

Misc — 14 tasks Forensics (memory/disk/traffic), steganography (image/audio/protocol), blockchain/smart contracts, AI adversarial, CUDA reverse, hardware

Quality assurance

Every task ships with an oracle solution/solve.sh that achieves reward = 1.0, ensuring each challenge is solvable. Scoring is fully automated and objective — tests/test.sh reads the agent's submitted flag from /logs/agent/flag.txt and compares it against the ground-truth flag, writing 1 (pass) or 0 (fail) to /logs/verifier/reward.txt, with no subjective rubrics. The flag is injected into the target container via the FLAG environment variable and never exposed to the agent directly.

Quickstart

You can run any agent and model supported by Harbor. The agent is selected via -a and the model via -m in provider/model format. For example, to run with the opencode agent and a custom OpenAI-compatible provider (such as ModelStudio), you also pass the provider configuration in --ak. Run a single task evaluation locally:

timeout 22200 harbor run \
   -d alibaba-aaig/alibabactf-2023-crypto-BabyPRNG \
   -a opencode \
   -m modelstudio/qwen3.8-max \
   --ak "opencode_config={\"provider\":{\"modelstudio\":{\"npm\":\"@ai-sdk/openai-compatible\",\"options\":{\"baseURL\":\"$MODELSTUDIO_BASE_URL\",\"apiKey\":\"$MODELSTUDIO_API_KEY\"},\"models\":{\"qwen3.8-max\":{\"name\":\"Qwen3.8 Max\",\"limit\":{\"context\":202752,\"output\":16384}}}}}}" \
   -y 2>&1

Or run the full dataset in parallel running Claude Code with a Claude model. :

export ANTHROPIC_API_KEY=<YOUR-KEY> 
harbor run -d alibaba-aaig/alibaba-ctf \
  --agent claude-code \
  -m anthropic/claude-sonnet-4 \
  --n-concurrent 4 \
  --env docker 

To upload results to Harbor Hub (e.g. for leaderboard submission), include the --upload flag or run harbor upload "<job/path>".

Contribute

Tasks are sourced from the AlibabaCTF competition series. We also encourage the community to reach out when they find bugs in tasks. Building benchmarks for current model capabilities is hard — community feedback helps keep quality high.

⚠️ Disclaimer

The Alibaba CTF Benchmark is intended to facilitate the establishment of a security governance framework for large models and to accelerate their safe and controllable application. It may contain offensive, upsetting, or otherwise sensitive content, including but not limited to vulnerability exploitation, privilege escalation, reverse engineering, cryptography, and other security-related techniques. The benchmark is provided solely for research and lawful purposes, including but not limited to security evaluation, red-teaming, and safety alignment of large language models. The views, flags, solutions, and any content expressed in the data are not related to the organizations, authors, and affiliated entities involved in this project. Users assume full responsibility for the final behavior and compliance of downstream systems. This project is not liable for any direct or indirect losses resulting from the use of this dataset.

📄 License

This project is licensed under the Apache-2.0 License.