Git.branch for git.branch (wrong case)| Set | n | MCP | UTCP | code mode | Notebook | Δ |
|---|---|---|---|---|---|---|
| Public sets | ||||||
| API-Bank L1 | 79 | 27.4 ± 2.6 | 31.6 ± 4.6 | 32.9 ± 8.9 | 25.7 ± 1.5 | −0.07 |
| API-Bank L2 | 27 | n.f. | n.f. | n.f. | 25.9 ± 3.7 | +0.39 |
| M3ToolEval | 16 | 4.2 ± 7.2 | 8.3 ± 3.6 | n.f. | 8.3 ± 3.6 | +0.04 |
| NESTFUL coding | 89 | 10.5 ± 1.7 | 6.0 ± 1.3 | n.f. | 4.1 ± 1.7 | −0.31 |
| When2Call | 723 | 40.0 ± 3.0 | 34.2 ± 0.1 | 23.4 ± 0.3 | 34.1 ± 0.5 | −2.37 |
| Our tasks and generated sets | ||||||
| Real tools | 97 | 16.5 ± 2.1 | 7.9 ± 5.7 | n.f. | 11.3 ± 3.6 | −0.28 |
| Generated tasks, test | 200 | 29.2 ± 2.1 | 25.3 ± 1.9 | 11.5 ± 1.3 | 26.0 ± 1.0 | −0.35 |
| Generated, new domains | 200 | 14.7 ± 0.6 | 11.7 ± 1.3 | 6.8 ± 0.8 | 9.0 ± 0.5 | −0.62 |
| Generated, new wording | 200 | 19.7 ± 1.3 | 16.3 ± 1.6 | 8.0 ± 1.0 | 14.3 ± 0.8 | −0.59 |
| Floors (API-Bank L3, BFCL multi-turn, ToolHop): every protocol that fits scores below 5% | ||||||
| Pool B, mirrors the locked parts | 214 | 14.3 | 14.0 | 11.5 | 13.6 | −0.78 |
| Pool A, all 12 sets (post hoc) | 1,818 | 25.6 ± 1.1 | 21.7 ± 0.3 | 13.7 ± 0.6 | 21.4 ± 0.4 | −4.20 |
| Pool A, no When2Call | 1,095 | 16.0 ± 0.3 | 13.4 ± 0.6 | 7.2 ± 1.1 | 13.0 ± 0.4 | −3.04 |
| Pool A, reference tool required | 1,818 | 23.7 ± 0.9 | 18.6 ± 0.4 | 12.3 ± 0.2 | 16.4 ± 0.4 | −7.32 |
A seed-aware test: item-sampling variance plus the spread between fine-tune seeds, with near-copy items clustered (CR2 variance) and each set its own stratum.
The Notebook's three comparisons are corrected by Holm's method within each pool. A difference is shown only if Holm p ≤ 0.05. Otherwise the result is "no difference shown", with its 95% interval and MDD, never "equal".
In simulation with a seed sd of 2.1 points, item-level tests (McNemar, sign, item z) called two equal protocols different 13.5–21.2% of the time.
Pool B, the 214 items that mirror the locked parts: Notebook minus MCP −0.8 [−8.0, +6.4] points. MDD about 10.5 to 11.9. No difference shown.
Pool A, 1,818 items, post hoc: −4.2 [−6.0, −2.4], Holm p < 0.001, negative at every seed (−3.6 / −5.3 / −3.7). No difference from UTCP (Holm p = 0.71).
Locked test, 861 items, one run in semester 2: MDD about 11.5 points at 80% power, interval about ±6.3. A gap to MCP is unlikely to be shown there.
| Tried | Result | What it changed |
|---|---|---|
| Python or JSON calls | BFCL (a public function-calling benchmark) calling, Python minus JSON +1.0 and +0.0 at two seeds, with 29% fewer tokens | Token cost reported beside success |
| Refusal rows for requests no tool fits | Declining rose 22 and 18 points (two-seed means). Identical runs differed by up to 25 | Three seeds per protocol |
| Generated multi-step data | +62.5 on its own test, +1.5 on BFCL multi-turn. Two runs of one recipe differed by 12.4 on the real-tools set | Public test sets, a seed-aware test |
| On-policy distillation, reinforcement learning | No gain. Four RL tries reached 41.0% from 38.5%, within run-to-run noise | One supervised recipe |
| 25% Python in pretraining | HumanEval 0.6% → 15.2%. When2Call cannot-answer recall fell by 4.5 to 8.1 points | The Python-heavy base is the shared base |
| A 4,096-token window (Phase 2) | Passed 5 of 6 readiness checks. Failed to choose among 8 tool descriptions far back in the context | Deferred to semester 2 |
| Risk | L / I | Mitigation |
|---|---|---|
| The locked test can only detect large gaps (MDD 11.5 points) | Known / H | Propose d′ on When2Call's 2,929 held-out items as a co-primary test, power and multiplicity set before the lock |
| Requiring the reference tool on When2Call changes two conclusions | Known / H | Settle the scorer rule in weeks 1–2, before the lock, and report both scorings |
| The single locked run is spoiled by a later change to the scorer, labels or comparisons | L / H | Fix scorer rules and every label first, freeze the comparisons before the run |
| Compute: about 186 GPU-hours on a shared cluster | M / M | Priority order: scorer rules, locked test, declining and data fix, the GitPython demo, goal-based sets, 4,096 tokens |
| WorkBench's tasks do not fit the 2,048-token window | M / M | A free fit check first. The git game levels remain as a goal-based set |
| No 4,096-token extension passes the readiness checks | M / L | Last in priority. Findings stay stated for 2,048 tokens |