harbor run -d pranalabs-ai-team/3dscenebench-miniharbor run -d pranalabs-ai-team/3dscenebench-mini3DSceneBench mini: seven scenes of 3DSceneBench, one per category, each with its reference room, so a run ends with the full score (appearance, geometry, functionality). 7 tasks.
harbor run -d pranalabs-ai-team/3dscenebench-mini3DSceneBench mini: seven scenes of 3DSceneBench, one per category, each with its reference room, so a run ends with the full score (appearance, geometry, functionality). 7 tasks.
harbor run -d pranalabs-ai-team/3dscenebench-miniCan a coding agent build a simulated world that supports a scene's many uses?
3DSceneBench gives an agent pictures of a scene and asks for a world of it as a MuJoCo scene. The world is scored by simulating it, not only by looking at it: verifiers load it, render it, measure it, push its objects, open its doors and drawers, and run everyday tasks in it.
3DSceneBench mini is seven of the benchmark's 49 scenes, one from each category. Each task carries its reference room, so a run ends with the full score on your own machine. The other 42 scenes are coming soon.
docs/ENTRANTS.md in that repository| Scene | Name | Category | Objects |
|---|---|---|---|
| 003 | Shared Office | Offices and recreation | 216 |
| 019 | Pipe Maintenance Room | Industry and infrastructure | 333 |
| 022 | Outdoor Gear Store | Retail | 980 |
| 027 | Rehabilitation Center | Healthcare and assisted living | 83 |
| 034 | Chemistry Lab | Science and research | 366 |
| 040 | Bakery | Hospitality and food service | 1,127 |
| 048 | Home Office | Homes | 238 |
There are seven tasks, one per scene, named pranalabs-ai-team/3dscenebench-<scene number>-l0. Each gives the agent 8 main pictures and up to 16 extra pictures of the scene, with their cameras, and the list of tasks that should be possible in it.
Each task holds the instruction, the pictures, the closed container the agent works in, and the grader with the scene's reference room. A reference room is the simulated room that the scene's pictures were rendered from. The agent never sees it: Harbor brings the grader in only after the agent has finished, in a separate container.
One world in /output/scene/:
scene.xml: the room, its furniture and every rigid object, for MuJoCo 3.14.nonrigid.json: fluids, soft bodies and thin shells, in scenes that have them, for Genesis.The reward is the mean of three parts, each from 0 to 100.
| Part | Asks |
|---|---|
| Appearance | Does the world look like the scene from 16 cameras the agent never saw? |
| Geometry | Are the right objects there, with the right shape and size, in the right places? |
| Functionality | Does it behave like the reference room when objects are pushed, tilted, opened, loaded and handled by a robot? |
A world must first load in MuJoCo and run without blowing up.
You need Docker, Harbor 0.23.0 and the grader image, which you build once from the repository. Its README has the steps.
harbor download pranalabs-ai-team/3dscenebench-mini@v1.0.0
harbor run -p 3dscenebench-mini -a claude-code -m anthropic/<model> --ak disallowed_tools=WebFetch,WebSearch -n 1
This runs the seven scenes one after another. Each run ends with reward.json: the reward, its three parts and the four functionality levels. Keep -n 1: the tests are timed, and two graders running at once can change a result.
| Time | 3 hours |
| Output tokens | 10,000,000 |
| Agent's machine | 2 CPUs, 16 GB of memory, no GPU |
| Grader's machine | 4 to 8 CPUs and 16 to 48 GB of memory, by scene; no GPU |
| Network | Only the agent's own model API. No web search, no web fetch, no outside files |
v1.0.0. The scenes, tasks and scoring rules are frozen for this version.
Can a coding agent build a simulated world that supports a scene's many uses?
3DSceneBench gives an agent pictures of a scene and asks for a world of it as a MuJoCo scene. The world is scored by simulating it, not only by looking at it: verifiers load it, render it, measure it, push its objects, open its doors and drawers, and run everyday tasks in it.
3DSceneBench mini is seven of the benchmark's 49 scenes, one from each category. Each task carries its reference room, so a run ends with the full score on your own machine. The other 42 scenes are coming soon.
docs/ENTRANTS.md in that repository| Scene | Name | Category | Objects |
|---|---|---|---|
| 003 | Shared Office | Offices and recreation | 216 |
| 019 | Pipe Maintenance Room | Industry and infrastructure | 333 |
| 022 | Outdoor Gear Store | Retail | 980 |
| 027 | Rehabilitation Center | Healthcare and assisted living | 83 |
| 034 | Chemistry Lab | Science and research | 366 |
| 040 | Bakery | Hospitality and food service | 1,127 |
| 048 | Home Office | Homes | 238 |
There are seven tasks, one per scene, named pranalabs-ai-team/3dscenebench-<scene number>-l0. Each gives the agent 8 main pictures and up to 16 extra pictures of the scene, with their cameras, and the list of tasks that should be possible in it.
Each task holds the instruction, the pictures, the closed container the agent works in, and the grader with the scene's reference room. A reference room is the simulated room that the scene's pictures were rendered from. The agent never sees it: Harbor brings the grader in only after the agent has finished, in a separate container.
One world in /output/scene/:
scene.xml: the room, its furniture and every rigid object, for MuJoCo 3.14.nonrigid.json: fluids, soft bodies and thin shells, in scenes that have them, for Genesis.The reward is the mean of three parts, each from 0 to 100.
| Part | Asks |
|---|---|
| Appearance | Does the world look like the scene from 16 cameras the agent never saw? |
| Geometry | Are the right objects there, with the right shape and size, in the right places? |
| Functionality | Does it behave like the reference room when objects are pushed, tilted, opened, loaded and handled by a robot? |
A world must first load in MuJoCo and run without blowing up.
You need Docker, Harbor 0.23.0 and the grader image, which you build once from the repository. Its README has the steps.
harbor download pranalabs-ai-team/3dscenebench-mini@v1.0.0
harbor run -p 3dscenebench-mini -a claude-code -m anthropic/<model> --ak disallowed_tools=WebFetch,WebSearch -n 1
This runs the seven scenes one after another. Each run ends with reward.json: the reward, its three parts and the four functionality levels. Keep -n 1: the tests are timed, and two graders running at once can change a result.
| Time | 3 hours |
| Output tokens | 10,000,000 |
| Agent's machine | 2 CPUs, 16 GB of memory, no GPU |
| Grader's machine | 4 to 8 CPUs and 16 to 48 GB of memory, by scene; no GPU |
| Network | Only the agent's own model API. No web search, no web fetch, no outside files |
v1.0.0. The scenes, tasks and scoring rules are frozen for this version.