A tool makes the action the same every time
Given one play-on-Spotify tool, GameTerm's models play well-known songs the same way every time. A niche artist breaks it. Asked for Mike Sherm, the models typed Mike Sherman. Nemotron 3 Nano 4B and Qwen3.5 4B needed the name spelled out; Qwen 3.8 27B fixed the name itself with a web search in one turn.
the tool plays the song
3 of 3 sessions typed Sherman first1 fixed it alone
The question
A model can play a song many ways: the Spotify API, AppleScript on the mouse, AppleScript on the app. If the harness gives it one tool that plays a song, does the action become the same every time? And where does it still break?
Said before the run
No prediction was written down before these sessions. The idea was: less thinking, more tool. A model that calls one tool does the same thing whichever model it is.
What happened
Well-known songs played through the one tool, media_play. Lil Baby's "My Dawg" played on 6 of 6 calls in one session.
The niche artist broke it. Every session below asked for "Asshole" by Mike Sherm.
| Session | Model | First call | What fixed the name | Played |
|---|---|---|---|---|
| 1 | Nemotron 3 Nano 4B (hybrid Mamba-2, text only) | Mike Sherman (the first ask said Sherman) | nothing; it found Mike Sherm, then lost it | no |
| 2 | Qwen3.5 4B (dense, multimodal) | Mike Sherman | the person spelled it out | yes, 4th turn |
| 3 | Qwen 3.8 27B | Mike Sherman | its own web search | yes, 1st turn |
These are the logs, one line per event. > is what the person typed, call is the tool the model chose, = is what the tool answered.
Session 1, Nemotron 3 Nano 4B, a hybrid Mamba-2 model that reads text only:
> Can you play "Astral" by Mike Sherman on Spotify?
call media_play artist "Mike Sherman" title "Astral"
= Rejected: Spotify search failed
> can you play the song "Asshole" by Mike Schirm?
call media_play artist "Mike Schirmer" title "Asshole"
= Rejected: Spotify search failed
> use web search to figure out who Mike Schirm is
call web_search "Mike Schirm musician Asshole song"
= summary: the musician you are likely thinking of is Mike Sherm
> can you play this song on Spotify?
call media_play artist "Mike Sherm" title "Asshole"
= Rejected: Spotify search failed
> can you play asshole by mike sherm on spotify
call media_play artist "Mike Sherman" title "Asshole"
= Rejected: Spotify search failed
Session 2, Qwen3.5 4B, a dense multimodal model:
> Can you play Asshole by Mike and his last name is S-H-E-R-M.
call media_play artist "Mike Sherman" title "Asshole"
= Rejected: no song by that title was found
> You put sherman. its mike sherm
call media_play artist "Mike Sherman" title "Sherman"
= Rejected: no song by that title was found
> so sherm - isn't short for sherman. its just mike sherm
call media_play artist "Mike Sherm" title "Sherman"
= Rejected: no song by that title was found
> So the song is Asshole by Mike Sherm
call media_play artist "Mike Sherm" title "Asshole"
= Succeeded: playing
Session 3, Qwen 3.8 27B, one turn, 14 seconds:
> can you play mike sherm - asshole
call media_play artist "Mike Sherman" title "Asshole"
= Rejected: no song by that title was found
call web_search "\"Mike Sherman\" \"Asshole\" song artist"
= 1. everybodywiki.com https://en.everybodywiki.com/Mike_Sherm
call media_play artist "Mike Sherm" title "Asshole"
= Succeeded: playing
In its next turn it did the same for "anti by sob rbe": rejected, web search, then SOB X RBE played.
What it cannot tell you
These are 3 sessions, not a frozen test. The logs do not record the model; the models named here come from the author. One recovery is one recovery, not a rate. Mike Sherm is one artist. The tool choice itself was measured separately, on 320 frozen requests, in the DayCare record below.
Method
GameTerm writes every turn to turns-<start>-<pid>.jsonl: the request, each tool call with its arguments, each result, and the outcome. The files below are every turn in those sessions that mentions Mike Sherm, cut from the full session files with nothing else changed. The logs above are shortened from them. Times are at_ms, milliseconds since 1970.