Loading dataset details…
harbor run -d userbench/UserBench-train400harbor run -d userbench/UserBench-train400UserBench train400 twin: the same 620 held tasks plus 400 leak-safe prior turns per developer. Composer 2.5 multi-label acts score by set Jaccard/IoU.
harbor run -d userbench/UserBench-train400UserBench train400 twin: the same 620 held tasks plus 400 leak-safe prior turns per developer. Composer 2.5 multi-label acts score by set Jaccard/IoU.
harbor run -d userbench/UserBench-train400Twin of the public UserBench eval with leak-safe prior-session context.
Same 620 held tasks / 62 developers as userbench/UserBench@v2 (history, gold, verifier unchanged). Each task additionally ships earlier sessions for that developer under /sim/train/, totaling 400 human user-turns sampled with sqrt two-stage.
| Path | Role |
|---|---|
/sim/history.md |
Held conversation so far (identical to baseline) |
/sim/train/_index.json |
Index of packed prior sessions (ids, timestamps, repos, turn takes, paths) |
/sim/train/<sid>.md |
Contiguous prefixes of those sessions (same > DEVELOPER / > AGENT format) |
/sim/answer.txt |
Where the agent writes the next developer message |
The instruction is count-agnostic: it points at /sim/train/ without hardcoding “400”. The turn budget is a publish-time sampler parameter.
For each developer let T* = min(held_sessions[].ts) (manifest session timestamp).
Every packed train session satisfies:
session.ts < T*max(turn.ts) < T*This is stricter than the cohort’s session-start train/held split alone. Sessions that wall-clock-overlap the earliest held session are never included.
QC on this package: 0 / 620 tasks violate max(turn.ts) < T*.
sqrt_two_stage (time-stratified session pick, then contiguous-prefix waterfill).The full pools (not just the 400-turn subsample) live in the GitHub repo under train_pools/ so future Hub variants (e.g. train1000) can resample from the same T*-filtered corpus without changing the leakage rule.
| Developers | 62 |
| Tasks | 620 (10 per developer; same selection as baseline) |
| Condition | train400 |
| Train budget | 400 human-turns / developer (shared across that developer’s 10 tasks) |
Same short IDs as baseline (Harbor org/name):
userbench/<username>__<hash>Versioning is via Harbor tags on this package (e.g. @v2). Digests differ from the noprofile baseline; consume via the dataset ref below so baseline UserBench@v2 stays pinned to zero-train digests.
userbench · user-simulation · coding-agents · move-prediction · train400
| What | Ref |
|---|---|
| This twin | userbench/UserBench-train400@v2 (also latest) |
| Zero-train baseline | userbench/UserBench@v2 |
| Hub | https://hub.harborframework.com/datasets/userbench/UserBench-train400 |
| Full train pools (GitHub) | https://github.com/AlienKevin/user-simulator/tree/main/train_pools |
harbor run -d userbench/UserBench-train400@v2 -a <agent> -m <model>UserBench (noprofile, no /sim/train/)Twin of the public UserBench eval with leak-safe prior-session context.
Same 620 held tasks / 62 developers as userbench/UserBench@v2 (history, gold, verifier unchanged). Each task additionally ships earlier sessions for that developer under /sim/train/, totaling 400 human user-turns sampled with sqrt two-stage.
| Path | Role |
|---|---|
/sim/history.md |
Held conversation so far (identical to baseline) |
/sim/train/_index.json |
Index of packed prior sessions (ids, timestamps, repos, turn takes, paths) |
/sim/train/<sid>.md |
Contiguous prefixes of those sessions (same > DEVELOPER / > AGENT format) |
/sim/answer.txt |
Where the agent writes the next developer message |
The instruction is count-agnostic: it points at /sim/train/ without hardcoding “400”. The turn budget is a publish-time sampler parameter.
For each developer let T* = min(held_sessions[].ts) (manifest session timestamp).
Every packed train session satisfies:
session.ts < T*max(turn.ts) < T*This is stricter than the cohort’s session-start train/held split alone. Sessions that wall-clock-overlap the earliest held session are never included.
QC on this package: 0 / 620 tasks violate max(turn.ts) < T*.
sqrt_two_stage (time-stratified session pick, then contiguous-prefix waterfill).The full pools (not just the 400-turn subsample) live in the GitHub repo under train_pools/ so future Hub variants (e.g. train1000) can resample from the same T*-filtered corpus without changing the leakage rule.
| Developers | 62 |
| Tasks | 620 (10 per developer; same selection as baseline) |
| Condition | train400 |
| Train budget | 400 human-turns / developer (shared across that developer’s 10 tasks) |
Same short IDs as baseline (Harbor org/name):
userbench/<username>__<hash>Versioning is via Harbor tags on this package (e.g. @v2). Digests differ from the noprofile baseline; consume via the dataset ref below so baseline UserBench@v2 stays pinned to zero-train digests.
userbench · user-simulation · coding-agents · move-prediction · train400
| What | Ref |
|---|---|
| This twin | userbench/UserBench-train400@v2 (also latest) |
| Zero-train baseline | userbench/UserBench@v2 |
| Hub | https://hub.harborframework.com/datasets/userbench/UserBench-train400 |
| Full train pools (GitHub) | https://github.com/AlienKevin/user-simulator/tree/main/train_pools |
harbor run -d userbench/UserBench-train400@v2 -a <agent> -m <model>UserBench (noprofile, no /sim/train/)