Loading dataset details…
harbor run -d birdsql/bird-mini-devharbor run -d birdsql/bird-mini-devBIRD Mini-Dev: 500 aligned questions across SQLite, MySQL and PostgreSQL (1,500 dialect-specific tasks).
harbor run -d birdsql/bird-mini-devBIRD Mini-Dev: 500 aligned questions across SQLite, MySQL and PostgreSQL (1,500 dialect-specific tasks).
harbor run -d birdsql/bird-mini-devBIRD evaluates agents' ability to answer
real-world questions over relational databases. This Harbor edition provides
500 aligned questions across SQLite, MySQL and PostgreSQL: 1,500 interactive
tasks covering 11 databases. Agents can explore the database and submit their
final query to /app/answer.sql. An independent verifier evaluates execution
accuracy against a fresh database snapshot.
This release revises the last update following community feedback and a
human-in-the-loop review. We corrected reference SQL, clarified questions and
evidence, and replaced selected instances. Annotations, fixed-prompt baselines
and setup instructions are available on
Hugging Face.
Solver and verifier environments have network access disabled. For local
Docker execution, use the included local_provider.py with Harbor's
environment.import_path set to local_provider:LocalDockerEnvironment.
If you find this benchmark useful, please consider citing:
@article{li2024can,
title={Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls},
author={Li, Jinyang and Hui, Binyuan and Qu, Ge and Yang, Jiaxi and Li, Binhua and Li, Bowen and Wang, Bailin and Qin, Bowen and Geng, Ruiying and Huo, Nan and others},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2024}
}
BIRD evaluates agents' ability to answer
real-world questions over relational databases. This Harbor edition provides
500 aligned questions across SQLite, MySQL and PostgreSQL: 1,500 interactive
tasks covering 11 databases. Agents can explore the database and submit their
final query to /app/answer.sql. An independent verifier evaluates execution
accuracy against a fresh database snapshot.
This release revises the last update following community feedback and a
human-in-the-loop review. We corrected reference SQL, clarified questions and
evidence, and replaced selected instances. Annotations, fixed-prompt baselines
and setup instructions are available on
Hugging Face.
Solver and verifier environments have network access disabled. For local
Docker execution, use the included local_provider.py with Harbor's
environment.import_path set to local_provider:LocalDockerEnvironment.
If you find this benchmark useful, please consider citing:
@article{li2024can,
title={Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls},
author={Li, Jinyang and Hui, Binyuan and Qu, Ge and Yang, Jiaxi and Li, Binhua and Li, Bowen and Wang, Bailin and Qin, Bowen and Geng, Ruiying and Huo, Nan and others},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2024}
}