Loading dataset details…
harbor run -d hcltech-robotics/simreadybenchharbor run -d hcltech-robotics/simreadybenchSimReadyBench NV100
harbor run -d hcltech-robotics/simreadybenchSimReadyBench NV100
harbor run -d hcltech-robotics/simreadybenchSimReadyBench measures reconstruction of articulated USD assets from one image and nominal dimensions. This set contains 100 tasks: 100 public observation images and 400 comparison images. Each agent receives one observation. The four comparison views and reference USD are used by the verifier.
Use Linux x86_64 with Docker, Compose and Harbor 0.24.0:
python -m pip install 'harbor==0.24.0'
harbor run --dataset hcltech-robotics/simreadybench --agent YOUR_AGENT --model YOUR_MODELUse the dataset content digest to reproduce a specific release. Agent containers have public outbound internet access. Public inputs are included in each task package. Verifier references are fetched from pinned HF files during setup; final evaluation runs offline.
Write a self-contained USD package to /output/submission, with asset.usd as
its entrypoint. The package limit is 1 GiB of expanded files and 10000 files.
The task's nominal dimensions use canonical metres in XYZ order. Blender is
reserved for the verifier renderer and must not be used for agent authoring.
Up to five candidate-only formal feedback calls are available:
python -m simready_eval.feedback validate_candidate --package /output/submission --request-id attempt-1 --url http://127.0.0.1:8080Use a new request ID for each changed candidate; retry a transport interruption with the same ID. Final verification independently scores the submitted files.
Formal, visual and kinematic fidelity are separate scores on a 0–100 scale; Harbor rewards normalize each to 0–1. There is no combined overall score.
95 * passed / 46, plus 5 when all 46 mandatory requirements pass.
The 51 optional requirements are diagnostic and do not change this score.sqrt(G * R). G measures full 3D surface agreement in both
directions after translation/yaw alignment, without scale fitting. R is the
mean foreground SSIM across the four fixed comparison views, using the union
of reference and candidate foreground masks. Cameras and lighting are fixed.
Material-property correspondence and its coverage are reported as diagnostics.Known scores, coverage and unavailable dimensions are reported separately. The dataset metric averages each dimension over its own known results and reports coverage and score bounds; infrastructure-unknown dimensions are not model zeros.
The canonical dataset is hcltech-robotics/simreadybench
at immutable revision c56694fe51cda7833bea4a0e4b6e6ae255f72182. Task manifests pin each file's
size and SHA-256. NVIDIA source: nvidia/simready-assets@f4860aa0c75607adb50a78bf96e77510bffc040f.
Evaluator code is MIT licensed; see LICENSE. Assets and rendered references retain their asset-specific upstream licences and provenance in the pinned dataset. Dependency licences are listed in THIRD_PARTY_NOTICES.md.
nv100_04ed00317ccc5a90): preview/download
nv100_0d5a045d08832558): preview/download
nv100_0ee38dbeb8dccf6d): preview/download
nv100_0fdbdbbf0914a091): preview/download
nv100_142ac642d87da17c): preview/download
nv100_15d6296b43e32e18): preview/download
nv100_17a2f7ce1c73e24b): preview/download
nv100_17b98c24982e1657): preview/download
nv100_19607166b9604d89): preview/download
nv100_198a9e9c4f7ce3f2): preview/download
nv100_1d6ca6d90cd58961): preview/download
nv100_2124cde602dbdcb7): preview/download
nv100_2442a6f885eb3cc3): preview/download
nv100_247345f2a336cc5f): preview/download
nv100_277538c0752a1186): preview/download
nv100_29738d03bc5d9a68): preview/download
nv100_2999ac8a4e7af7e7): preview/download
nv100_2a6a57f1f660128a): preview/download
nv100_2ab1c8d84ebe4eb9): preview/download
nv100_2af97ae24a8d7fde): preview/download
nv100_3084bc1f12b79ce8): preview/download
nv100_36db64b13e6813e1): preview/download
nv100_381f0169f4c4fbd1): preview/download
nv100_382b02d058a48b6a): preview/download
nv100_3b37b9f58a70ca40): preview/download
nv100_47447ae933c565c5): preview/download
nv100_4afa194354477c47): preview/download
nv100_4b024c41c6d4c2bb): preview/download
nv100_4e10782bf98f9405): preview/download
nv100_525b65ccbe56ffea): preview/download
nv100_56025f338315d66a): preview/download
nv100_58b1427ce07ba99a): preview/download
nv100_58fba5a2a836bcef): preview/download
nv100_5b8085baf6c090ed): preview/download
nv100_5c3d3461fa840d9d): preview/download
nv100_5d12c190ef7457d1): preview/download
nv100_600fc2ad768f1930): preview/download
nv100_6627b65375e93e5d): preview/download
nv100_69ecd52a60671756): preview/download
nv100_6f6f63bdb94a83b4): preview/download
nv100_70a1243908f64ead): preview/download
nv100_70c417d940a526ee): preview/download
nv100_7777d0b26e69f86a): preview/download
nv100_7876a753434cf7d3): preview/download
nv100_78da45d42fdf538c): preview/download
nv100_791e79d947ee35ae): preview/download
nv100_7ac8dcd5266e9924): preview/download
nv100_80416c2bc0a430eb): preview/download
nv100_8574b6f4aa22c8e3): preview/download
nv100_8817a4bd528f5549): preview/download
nv100_8cfd1e56a0b504ea): preview/download
nv100_8d81b2987b196041): preview/download
nv100_8dd6e1f1cbb8b462): preview/download
nv100_918619c7fe40bdc5): preview/download
nv100_9499de2220b36cbe): preview/download
nv100_968cd3dfe5ad21c1): preview/download
nv100_977905a7f68dc323): preview/download
nv100_9ba70a69917d2305): preview/download
nv100_9e3f7dbd969acac5): preview/download
nv100_9e6234280413cdcb): preview/download
nv100_a194e402f44013d1): preview/download
nv100_a2939a17e3444d52): preview/download
nv100_a3fe4697f3e4c15e): preview/download
nv100_a7a336d818a3d9d3): preview/download
nv100_abb1b470da2eb131): preview/download
nv100_adb626df31bf7509): preview/download
nv100_af46aeafd295d310): preview/download
nv100_b4208a0e4f762579): preview/download
nv100_b6aeeb666424ae0d): preview/download
nv100_b83324e831779e19): preview/download
nv100_bff10216da262e6a): preview/download
nv100_c2f7f1f03e91af9b): preview/download
nv100_c35d376f74c032a5): preview/download
nv100_c6c60a8c7081041a): preview/download
nv100_c844b5efec42e5fa): preview/download
nv100_c87b9d6690f9e687): preview/download
nv100_c89ec640cbfbd302): preview/download
nv100_c8a8bac26c7b4874): preview/download
nv100_c92d6c8478d7758f): preview/download
nv100_ca63914168dce803): preview/download
nv100_cb8c72f166e90540): preview/download
nv100_d152ce4f60b45ea9): preview/download
nv100_d37c0067281cdd26): preview/download
nv100_d5ddbfeda2fe7ad9): preview/download
nv100_d9edaf956545bd67): preview/download
nv100_dea081707992a622): preview/download
nv100_df1e96e685d276d3): preview/download
nv100_e006be66e4cc28f9): preview/download
nv100_e7a7d6fbf4f2326f): preview/download
nv100_e948d0fd08624353): preview/download
nv100_eadc1038f3bfd872): preview/download
nv100_eb55c976d6cd528f): preview/download
nv100_eb6800ce857dc2eb): preview/download
nv100_ec0dd39731982fa2): preview/download
nv100_ef3a3efa5b480cd0): preview/download
nv100_f33ae88933be01fa): preview/download
nv100_f7f391c5b6b8e44d): preview/download
nv100_fb8ddbbe3dbc2900): preview/download
nv100_fbb5620efff99eee): preview/download
nv100_fd477c46fa524292): preview/downloadEach attempt must include a complete, reviewable runner-recorded trajectory of agent tool calls and results. Do not remove or conceal calls or results.
Retrieving or importing any external OpenUSD asset is an automatic protocol failure, whether or not it is a benchmark reference. This includes USD, USDA, USDC and USDZ files, and USD assets contained in downloaded archives.
You may author, load, render and validate your own USD locally and use the task-provided inputs. Ordinary software-tool installation is allowed; it does not permit importing externally supplied USD assets into the reconstruction.
Protocol validity is recorded separately from reconstruction scores. A missing or unreviewable trajectory is unverified, not a clean integrity pass. Ambiguous cases require review.
Geometry is measured with 65,536 deterministic area-weighted surface samples and graded distance similarity. The exact alignment, 46 formal requirements, four-view rendering quality and articulation measurements are retained. Reports identify the new geometry scoring protocol. The verifier includes SciPy 1.18.0; actor and formal-feedback environments do not require it.
SimReadyBench measures reconstruction of articulated USD assets from one image and nominal dimensions. This set contains 100 tasks: 100 public observation images and 400 comparison images. Each agent receives one observation. The four comparison views and reference USD are used by the verifier.
Use Linux x86_64 with Docker, Compose and Harbor 0.24.0:
python -m pip install 'harbor==0.24.0'
harbor run --dataset hcltech-robotics/simreadybench --agent YOUR_AGENT --model YOUR_MODELUse the dataset content digest to reproduce a specific release. Agent containers have public outbound internet access. Public inputs are included in each task package. Verifier references are fetched from pinned HF files during setup; final evaluation runs offline.
Write a self-contained USD package to /output/submission, with asset.usd as
its entrypoint. The package limit is 1 GiB of expanded files and 10000 files.
The task's nominal dimensions use canonical metres in XYZ order. Blender is
reserved for the verifier renderer and must not be used for agent authoring.
Up to five candidate-only formal feedback calls are available:
python -m simready_eval.feedback validate_candidate --package /output/submission --request-id attempt-1 --url http://127.0.0.1:8080Use a new request ID for each changed candidate; retry a transport interruption with the same ID. Final verification independently scores the submitted files.
Formal, visual and kinematic fidelity are separate scores on a 0–100 scale; Harbor rewards normalize each to 0–1. There is no combined overall score.
95 * passed / 46, plus 5 when all 46 mandatory requirements pass.
The 51 optional requirements are diagnostic and do not change this score.sqrt(G * R). G measures full 3D surface agreement in both
directions after translation/yaw alignment, without scale fitting. R is the
mean foreground SSIM across the four fixed comparison views, using the union
of reference and candidate foreground masks. Cameras and lighting are fixed.
Material-property correspondence and its coverage are reported as diagnostics.Known scores, coverage and unavailable dimensions are reported separately. The dataset metric averages each dimension over its own known results and reports coverage and score bounds; infrastructure-unknown dimensions are not model zeros.
The canonical dataset is hcltech-robotics/simreadybench
at immutable revision c56694fe51cda7833bea4a0e4b6e6ae255f72182. Task manifests pin each file's
size and SHA-256. NVIDIA source: nvidia/simready-assets@f4860aa0c75607adb50a78bf96e77510bffc040f.
Evaluator code is MIT licensed; see LICENSE. Assets and rendered references retain their asset-specific upstream licences and provenance in the pinned dataset. Dependency licences are listed in THIRD_PARTY_NOTICES.md.
nv100_04ed00317ccc5a90): preview/downloadnv100_0d5a045d08832558): preview/downloadnv100_0ee38dbeb8dccf6d): preview/downloadnv100_0fdbdbbf0914a091): preview/downloadnv100_142ac642d87da17c): preview/downloadnv100_15d6296b43e32e18): preview/downloadnv100_17a2f7ce1c73e24b): preview/downloadnv100_17b98c24982e1657): preview/downloadnv100_19607166b9604d89): preview/downloadnv100_198a9e9c4f7ce3f2): preview/downloadnv100_1d6ca6d90cd58961): preview/downloadnv100_2124cde602dbdcb7): preview/downloadnv100_2442a6f885eb3cc3): preview/downloadnv100_247345f2a336cc5f): preview/downloadnv100_277538c0752a1186): preview/downloadnv100_29738d03bc5d9a68): preview/downloadnv100_2999ac8a4e7af7e7): preview/downloadnv100_2a6a57f1f660128a): preview/downloadnv100_2ab1c8d84ebe4eb9): preview/downloadnv100_2af97ae24a8d7fde): preview/downloadnv100_3084bc1f12b79ce8): preview/downloadnv100_36db64b13e6813e1): preview/downloadnv100_381f0169f4c4fbd1): preview/downloadnv100_382b02d058a48b6a): preview/downloadnv100_3b37b9f58a70ca40): preview/downloadnv100_47447ae933c565c5): preview/downloadnv100_4afa194354477c47): preview/downloadnv100_4b024c41c6d4c2bb): preview/downloadnv100_4e10782bf98f9405): preview/downloadnv100_525b65ccbe56ffea): preview/downloadnv100_56025f338315d66a): preview/downloadnv100_58b1427ce07ba99a): preview/downloadnv100_58fba5a2a836bcef): preview/downloadnv100_5b8085baf6c090ed): preview/downloadnv100_5c3d3461fa840d9d): preview/downloadnv100_5d12c190ef7457d1): preview/downloadnv100_600fc2ad768f1930): preview/downloadnv100_6627b65375e93e5d): preview/downloadnv100_69ecd52a60671756): preview/downloadnv100_6f6f63bdb94a83b4): preview/downloadnv100_70a1243908f64ead): preview/downloadnv100_70c417d940a526ee): preview/downloadnv100_7777d0b26e69f86a): preview/downloadnv100_7876a753434cf7d3): preview/downloadnv100_78da45d42fdf538c): preview/downloadnv100_791e79d947ee35ae): preview/downloadnv100_7ac8dcd5266e9924): preview/downloadnv100_80416c2bc0a430eb): preview/downloadnv100_8574b6f4aa22c8e3): preview/downloadnv100_8817a4bd528f5549): preview/downloadnv100_8cfd1e56a0b504ea): preview/downloadnv100_8d81b2987b196041): preview/downloadnv100_8dd6e1f1cbb8b462): preview/downloadnv100_918619c7fe40bdc5): preview/downloadnv100_9499de2220b36cbe): preview/downloadnv100_968cd3dfe5ad21c1): preview/downloadnv100_977905a7f68dc323): preview/downloadnv100_9ba70a69917d2305): preview/downloadnv100_9e3f7dbd969acac5): preview/downloadnv100_9e6234280413cdcb): preview/downloadnv100_a194e402f44013d1): preview/downloadnv100_a2939a17e3444d52): preview/downloadnv100_a3fe4697f3e4c15e): preview/downloadnv100_a7a336d818a3d9d3): preview/downloadnv100_abb1b470da2eb131): preview/downloadnv100_adb626df31bf7509): preview/downloadnv100_af46aeafd295d310): preview/downloadnv100_b4208a0e4f762579): preview/downloadnv100_b6aeeb666424ae0d): preview/downloadnv100_b83324e831779e19): preview/downloadnv100_bff10216da262e6a): preview/downloadnv100_c2f7f1f03e91af9b): preview/downloadnv100_c35d376f74c032a5): preview/downloadnv100_c6c60a8c7081041a): preview/downloadnv100_c844b5efec42e5fa): preview/downloadnv100_c87b9d6690f9e687): preview/downloadnv100_c89ec640cbfbd302): preview/downloadnv100_c8a8bac26c7b4874): preview/downloadnv100_c92d6c8478d7758f): preview/downloadnv100_ca63914168dce803): preview/downloadnv100_cb8c72f166e90540): preview/downloadnv100_d152ce4f60b45ea9): preview/downloadnv100_d37c0067281cdd26): preview/downloadnv100_d5ddbfeda2fe7ad9): preview/downloadnv100_d9edaf956545bd67): preview/downloadnv100_dea081707992a622): preview/downloadnv100_df1e96e685d276d3): preview/downloadnv100_e006be66e4cc28f9): preview/downloadnv100_e7a7d6fbf4f2326f): preview/downloadnv100_e948d0fd08624353): preview/downloadnv100_eadc1038f3bfd872): preview/downloadnv100_eb55c976d6cd528f): preview/downloadnv100_eb6800ce857dc2eb): preview/downloadnv100_ec0dd39731982fa2): preview/downloadnv100_ef3a3efa5b480cd0): preview/downloadnv100_f33ae88933be01fa): preview/downloadnv100_f7f391c5b6b8e44d): preview/downloadnv100_fb8ddbbe3dbc2900): preview/downloadnv100_fbb5620efff99eee): preview/downloadnv100_fd477c46fa524292): preview/downloadEach attempt must include a complete, reviewable runner-recorded trajectory of agent tool calls and results. Do not remove or conceal calls or results.
Retrieving or importing any external OpenUSD asset is an automatic protocol failure, whether or not it is a benchmark reference. This includes USD, USDA, USDC and USDZ files, and USD assets contained in downloaded archives.
You may author, load, render and validate your own USD locally and use the task-provided inputs. Ordinary software-tool installation is allowed; it does not permit importing externally supplied USD assets into the reconstruction.
Protocol validity is recorded separately from reconstruction scores. A missing or unreviewable trajectory is unverified, not a clean integrity pass. Ambiguous cases require review.
Geometry is measured with 65,536 deterministic area-weighted surface samples and graded distance similarity. The exact alignment, 46 formal requirements, four-view rendering quality and articulation measurements are retained. Reports identify the new geometry scoring protocol. The verifier includes SciPy 1.18.0; actor and formal-feedback environments do not require it.