Training a small language model to use tools

Does it matter how the tools are shown to it?
Arnav Goel CP4101 Final Year Project · NUS School of Computing
Supervisor Prof Eric Han · Evaluator Dr Henry Chia
20 November 2026

Small models can't know everything, so they use tools.

The goal: many tools in a small memory.

Web text
plus code
My own model
pretrained from scratch
SoC cluster
1.38B
parameters
Taught to use tools
this semester
A library it never saw
semester 2 demo
GitPython

The model writes a call. The harness runs it.

notes task
Change the tag of my "Quarterly review" note to urgent.
reference episode, written by code · not model output

Its working memory is tiny, and the menu eats it.

working memory (the context window) · 2,048 tokens, about three pages of text
the request,
the calls, the answer
about
70%
gone to MCP's tool list before you speak

The same tools look very different to the model.

the start of each prompt, to scale · search-first protocols pay later, when they look tools up

My idea: a table of contents, opened when needed.

Notebook
WHAT THE MODEL SEES · ONE LINE PER GROUP
files: list, read, write, search and move ...
calc: evaluate arithmetic and convert units
notes: create, list, read, update, tag and ...
what the model saw most recently
notes.list_notes(tag=""): List notes (id, title, ...
notes.update_note(note_id, title=None, body=None, ...
and 3 more tools
In [1]
In [2]
 
 
My bet: less to read, better decisions.

A fair test changes only one thing.

24models

Before trusting any score, I had to rule out cheating and luck.

A cheating model

On 29 tasks where a tool fails on purpose, it just says "done".
29 of 29 passed
caught

Luck

The same model, trained twice. Only the shuffle changed.

It worked: about half the memory.

759
Notebook · tokens per task, in its own runs

But it didn't win.

% of tasks solved
solved: the task's check passes
white marks: seeds
Pool picked after the fact. Items picked in advance: no difference shown.

Ask it for a salon, and it checks your alarms.

I would like to find a Unisex Salon in San Jose
When2Call · seed 42 · verbatim · the right answer is to decline
the Notebook's call, verbatim
tools.Alarm_1_GetAlarms(user_id='Unisex Salon')

full transcripts · When2Call item, seed 42 · MCP and Notebook recorded from the model, UTCP and code mode from the report

It doesn't know when to say no.

When2Call items with tools offered · split chosen after the results

Most of the gap sits at one decision.

points of the 4.2-point gap, Notebook minus MCP · by the stage where the failing run first went wrong · post hoc

It reads the right tool, then calls a name that was never offered.

lookup fix
githug · real repo · seed 42, recorded from the model
what it wrote, six times over, until the step captittle(name='test_code')
also common: Git.branch for git.branch (wrong case)
1 of 84
episodes solved

full transcript · githug, level branch · seed 42, recorded from the model

Changing the shared data didn't fix it.

Four suspects for why it can't say no.

The test bed is built. Semester 2 uses it to find the cause.

664GPU-hours so far,
failed runs included
Cheaper?Yes.
Better?No.
The gap?Mostly one decision.
For a small model like mine, the hard part isn't reading the menu.
It's knowing when not to order.
Backup B1

Success by set, standard data mix, three seeds (%).

SetnMCPUTCPcode modeNotebookΔ
Public sets
API-Bank L17927.4 ± 2.631.6 ± 4.632.9 ± 8.925.7 ± 1.5−0.07
API-Bank L227n.f.n.f.n.f.25.9 ± 3.7+0.39
M3ToolEval164.2 ± 7.28.3 ± 3.6n.f.8.3 ± 3.6+0.04
NESTFUL coding8910.5 ± 1.76.0 ± 1.3n.f.4.1 ± 1.7−0.31
When2Call72340.0 ± 3.034.2 ± 0.123.4 ± 0.334.1 ± 0.5−2.37
Our tasks and generated sets
Real tools9716.5 ± 2.17.9 ± 5.7n.f.11.3 ± 3.6−0.28
Generated tasks, test20029.2 ± 2.125.3 ± 1.911.5 ± 1.326.0 ± 1.0−0.35
Generated, new domains20014.7 ± 0.611.7 ± 1.36.8 ± 0.89.0 ± 0.5−0.62
Generated, new wording20019.7 ± 1.316.3 ± 1.68.0 ± 1.014.3 ± 0.8−0.59
Floors (API-Bank L3, BFCL multi-turn, ToolHop): every protocol that fits scores below 5%
Pool B, mirrors the locked parts21414.314.011.513.6−0.78
Pool A, all 12 sets (post hoc)1,81825.6 ± 1.121.7 ± 0.313.7 ± 0.621.4 ± 0.4−4.20
Pool A, no When2Call1,09516.0 ± 0.313.4 ± 0.67.2 ± 1.113.0 ± 0.4−3.04
Pool A, reference tool required1,81823.7 ± 0.918.6 ± 0.412.3 ± 0.216.4 ± 0.4−7.32
n.f.: the reference fits fewer than 5 items. Δ: Notebook minus MCP, set rows weighted by share of Pool A. Source: report Table 5.2.
Backup B2

How a difference counts as shown.

The test

A seed-aware test: item-sampling variance plus the spread between fine-tune seeds, with near-copy items clustered (CR2 variance) and each set its own stratum.

The Notebook's three comparisons are corrected by Holm's method within each pool. A difference is shown only if Holm p ≤ 0.05. Otherwise the result is "no difference shown", with its 95% interval and MDD, never "equal".

In simulation with a seed sd of 2.1 points, item-level tests (McNemar, sign, item z) called two equal protocols different 13.5–21.2% of the time.

What it says here

Pool B, the 214 items that mirror the locked parts: Notebook minus MCP −0.8 [−8.0, +6.4] points. MDD about 10.5 to 11.9. No difference shown.

Pool A, 1,818 items, post hoc: −4.2 [−6.0, −2.4], Holm p < 0.001, negative at every seed (−3.6 / −5.3 / −3.7). No difference from UTCP (Holm p = 0.71).

Locked test, 861 items, one run in semester 2: MDD about 11.5 points at 80% power, interval about ±6.3. A gap to MCP is unlikely to be shown there.

Source: report Section 4.3, Appendix H.
Backup B3

What could make these results wrong.

Exploratory only. Every test is on development items. Pool A is post hoc. When2Call and the generated sets, which have no locked part, give most of MCP's lead.
The design is not neutral. The Notebook is mine and tuned to this window. But every training task must fit all four protocols, so the weakest one selects the data (44,567 of 121,145 candidates), and 72% of tasks come from MCP servers.
The base is not neutral. The base's extra Python may interact with Python calling. Cannot-answer recall fell 4.5 to 8.1 points with it in Phase 1.
One model, one window. 1.38B and 2,048 tokens may favour the Notebook. My MCP lists every tool, against MCP's client guide, so MCP may be understated.
When2Call is 40% of Pool A. Its call items pass on any call. Requiring the reference tool changes two conclusions.
Our own items. The generated and real-tools items are mine, vetted against a checklist. Flagged items don't explain the generated sets' part of the lead.
Success is low. 57.4% of items pass in no main-mix fine-tune. Model size and training length are confounded.
Source: report Section 6.2.
Backup B4

Alternatives tried before the comparison was fixed.

TriedResultWhat it changed
Python or JSON callsBFCL (a public function-calling benchmark) calling, Python minus JSON +1.0 and +0.0 at two seeds, with 29% fewer tokensToken cost reported beside success
Refusal rows for requests no tool fitsDeclining rose 22 and 18 points (two-seed means). Identical runs differed by up to 25Three seeds per protocol
Generated multi-step data+62.5 on its own test, +1.5 on BFCL multi-turn. Two runs of one recipe differed by 12.4 on the real-tools setPublic test sets, a seed-aware test
On-policy distillation, reinforcement learningNo gain. Four RL tries reached 41.0% from 38.5%, within run-to-run noiseOne supervised recipe
25% Python in pretrainingHumanEval 0.6% → 15.2%. When2Call cannot-answer recall fell by 4.5 to 8.1 pointsThe Python-heavy base is the shared base
A 4,096-token window (Phase 2)Passed 5 of 6 readiness checks. Failed to choose among 8 tool descriptions far back in the contextDeferred to semester 2
Phase 1 used its own earlier scorer, so compare within a row only. Source: report Table 5.1, Appendix I.
Backup B5

Risks to the semester 2 plan.

RiskL / IMitigation
The locked test can only detect large gaps (MDD 11.5 points)Known / HPropose d′ on When2Call's 2,929 held-out items as a co-primary test, power and multiplicity set before the lock
Requiring the reference tool on When2Call changes two conclusionsKnown / HSettle the scorer rule in weeks 1–2, before the lock, and report both scorings
The single locked run is spoiled by a later change to the scorer, labels or comparisonsL / HFix scorer rules and every label first, freeze the comparisons before the run
Compute: about 186 GPU-hours on a shared clusterM / MPriority order: scorer rules, locked test, declining and data fix, the GitPython demo, goal-based sets, 4,096 tokens
WorkBench's tasks do not fit the 2,048-token windowM / MA free fit check first. The git game levels remain as a goal-based set
No 4,096-token extension passes the readiness checksM / LLast in priority. Findings stay stated for 2,048 tokens
L: likelihood, I: impact. Source: report Appendix K.
1
recording