arkey researchRSS
All posts / Research
seen once2026-10-09tool calling

A tool makes the action the same every time

Given one play-on-Spotify tool, GameTerm's models play well-known songs the same way every time. A niche artist breaks it. Asked for Mike Sherm, the models typed Mike Sherman. Nemotron 3 Nano 4B and Qwen3.5 4B needed the name spelled out; Qwen 3.8 27B fixed the name itself with a web search in one turn.

tool-callingdeterminismspotifyweb-searchBy Julian AbeledaProject gametermModels nemotron-3-nano-4b, qwen3.5-4b, qwen3.8-27bCreated 2026-10-09Edited 2026-10-09
Expected

the tool plays the song

Observed

3 of 3 sessions typed Sherman first1 fixed it alone

The question

A model can play a song many ways: the Spotify API, AppleScript on the mouse, AppleScript on the app. If the harness gives it one tool that plays a song, does the action become the same every time? And where does it still break?

Said before the run

No prediction was written down before these sessions. The idea was: less thinking, more tool. A model that calls one tool does the same thing whichever model it is.

What happened

Well-known songs played through the one tool, media_play. Lil Baby's "My Dawg" played on 6 of 6 calls in one session.

The niche artist broke it. Every session below asked for "Asshole" by Mike Sherm.

SessionModelFirst callWhat fixed the namePlayed
1Nemotron 3 Nano 4B (hybrid Mamba-2, text only)Mike Sherman (the first ask said Sherman)nothing; it found Mike Sherm, then lost itno
2Qwen3.5 4B (dense, multimodal)Mike Shermanthe person spelled it outyes, 4th turn
3Qwen 3.8 27BMike Shermanits own web searchyes, 1st turn

These are the logs, one line per event. > is what the person typed, call is the tool the model chose, = is what the tool answered.

Session 1, Nemotron 3 Nano 4B, a hybrid Mamba-2 model that reads text only:

> Can you play "Astral" by Mike Sherman on Spotify?
call media_play  artist "Mike Sherman"  title "Astral"
=    Rejected: Spotify search failed
> can you play the song "Asshole" by Mike Schirm?
call media_play  artist "Mike Schirmer"  title "Asshole"
=    Rejected: Spotify search failed
> use web search to figure out who Mike Schirm is
call web_search  "Mike Schirm musician Asshole song"
=    summary: the musician you are likely thinking of is Mike Sherm
> can you play this song on Spotify?
call media_play  artist "Mike Sherm"  title "Asshole"
=    Rejected: Spotify search failed
> can you play asshole by mike sherm on spotify
call media_play  artist "Mike Sherman"  title "Asshole"
=    Rejected: Spotify search failed

Session 2, Qwen3.5 4B, a dense multimodal model:

> Can you play Asshole by Mike and his last name is S-H-E-R-M.
call media_play  artist "Mike Sherman"  title "Asshole"
=    Rejected: no song by that title was found
> You put sherman. its mike sherm
call media_play  artist "Mike Sherman"  title "Sherman"
=    Rejected: no song by that title was found
> so sherm - isn't short for sherman. its just mike sherm
call media_play  artist "Mike Sherm"  title "Sherman"
=    Rejected: no song by that title was found
> So the song is Asshole by Mike Sherm
call media_play  artist "Mike Sherm"  title "Asshole"
=    Succeeded: playing

Session 3, Qwen 3.8 27B, one turn, 14 seconds:

> can you play mike sherm - asshole
call media_play  artist "Mike Sherman"  title "Asshole"
=    Rejected: no song by that title was found
call web_search  "\"Mike Sherman\" \"Asshole\" song artist"
=    1. everybodywiki.com  https://en.everybodywiki.com/Mike_Sherm
call media_play  artist "Mike Sherm"  title "Asshole"
=    Succeeded: playing

In its next turn it did the same for "anti by sob rbe": rejected, web search, then SOB X RBE played.

What it cannot tell you

These are 3 sessions, not a frozen test. The logs do not record the model; the models named here come from the author. One recovery is one recovery, not a rate. Mike Sherm is one artist. The tool choice itself was measured separately, on 320 frozen requests, in the DayCare record below.

Method

GameTerm writes every turn to turns-<start>-<pid>.jsonl: the request, each tool call with its arguments, each result, and the outcome. The files below are every turn in those sessions that mentions Mike Sherm, cut from the full session files with nothing else changed. The logs above are shortened from them. Times are at_ms, milliseconds since 1970.

Sources