The Filename I Asked For, and the Filename I Got
If I ask a coding model to write muse-smoke.txt, the write call has to say muse-smoke.txt. Meta's Muse Spark 1.1 did not clear that gate. Under agent-style request envelopes it returned claude-smoke.txt on every trial I ran, while the file contents stayed exactly factory. Muse Spark 1.2, retested on 2026-08-05, did not reproduce it: 0 wrong filenames in 30 trials across three channels.
I run a small software factory: agentic workflows where a coding model is handed a task and allowed to write and edit files with real tools. Adding a model to that setup means giving it a seat where its tool calls hit a real filesystem. The seat has a gate that is not a benchmark score. It is exact string fidelity on opaque tool arguments. Not something close. Not something reasonable. The string I supplied.
This is what I found on 1.1, how I checked it, and what happened when I reran the same envelopes on 1.2. It is a dated evaluation of one model for one role in my own systems. It is not a ranking, and it is not a recommendation to use anything else.
The smoke test
The task is deliberately trivial:
Create a file named muse-smoke.txt in the current directory
containing exactly the word: factoryOne file, one word. There is no reasoning to do and nothing to get clever about. It exists to check one property: does an exact user-supplied string survive the trip into a tool argument?
On Muse Spark 1.1, under an agent-style request envelope, the write call frequently came back as:
write(filePath=".../claude-smoke.txt", content="factory")The content was right. factory, exactly. The path was not.
That asymmetry is the whole story. The model was not confused about the task - it was confused about a name, and only about the name. A filename is not the kind of thing a model gets to improve. It is an opaque identifier: the user's string is the spec, and any deviation is a defect no matter how sensible the substitute looks.
Under a live harness with tools enabled, this stops being a curiosity about token prediction. A wrong path argument is a wrong file on disk.
How I checked
Two independent channels, because "the model does X" is a claim that deserves more than one way of being wrong.
Channel A - fixed-envelope replay. A frozen JSON request body sent straight to the Meta Model Responses API at https://api.meta.ai/v1/responses, with the tool calls inspected but never executed. The body is byte-stable and SHA-256 pinned, so every trial is genuinely the same request. Two envelopes:
- B2 - an OpenCode-style agent system prompt (OpenCode 1.18.3 class), with only a
writetool available. Request SHA-256dfa2912b6184db146ffa07c06ddadffd12317736cf383f42450310fb3b5cf8f9. - B12 - a one-line neutral coding-agent system prompt, same single tool. Request SHA-256
a85fcdf143515a2f4d18d113a43cf38c831261fe833dbcad4585e764b0c55570.
Channel B - live OpenCode. The real CLI (1.18.5), real tools, real writes into a disposable workspace per trial. Classification comes from the first write tool argument, with the on-disk result as a backstop.
The important detail: both request bodies contain zero case-insensitive occurrences of claude and zero of anthropic. I verified this on the exact bytes sent, not on a reconstruction. Whatever produced that token, it did not come from my input.
This is black-box behavioral testing on a fixed request. I am reporting what the tool arguments contained. I make no claim about anything inside the system that produced them.
muse-smoke.txt survive into the tool argument? - and disagree about whether the write is allowed to execute.What 1.1 did
Wrong-filename rate, ten trials per cell:
| Date (UTC) | B2 (agent-style) | B12 (neutral) | Live OpenCode |
|---|---|---|---|
| 2026-07-17 | 10/10 | 5/10 | n/a |
| 2026-07-23 | 10/10 | 3/10 | n/a |
| 2026-07-30 | 10/10 | 4/10 | 10/10 |
| 2026-07-31 | 10/10 | 6/10 | 10/10 |
Two weeks, four series, no drift toward correct.
The gap between the columns is the interesting part. The minimal envelope was stochastic - 3 to 6 wrong out of 10 depending on the day. The agent-style envelope was consistently wrong, and so was the live harness. The behavior was not uniform background noise; it got dramatically worse under exactly the conditions that make a model useful as a coding agent. A quick sanity check in a bare API call would have made this look occasional. It was not occasional where it mattered.
First, predict. Then draw.
Pick a series and a channel, lock in a prediction, then draw.
Three more things worth recording:
It was not only basenames. Several B2 trials rewrote a directory component too - /tmp/muse-opencode-envelope/ came back as /tmp/claude-opencode-envelope/. Under live OpenCode, the substitution reached into a real absolute path and rewrote an identity token in a repository directory name mid-path. That can put a file in a sibling directory that did not exist a moment ago.
It was not one hardcoded mapping. claude-smoke.txt dominated, but one 2026-07-23 B2 trial produced opencode-smoke.txt, one live trial on 2026-07-31 did the same, and earlier exploratory runs turned up cursor-smoke.txt. The attractor is identity-shaped names generally, not a single string swap. I never established relative frequencies for the alternates.
The content never wavered. Across every substituted trial, the file content stayed exactly factory. Path tokens turned out to be far more fragile than literal content.
For the factory decision, that was enough. Exact path preservation is a hard prerequisite for an implementation-worker role, 1.1 did not meet it, and it stayed out of that workflow.
This applies the dominant observed 1.1 rewrite (muse → claude in path tokens). It is not a claim that the model implemented this rule.
factory on both sides because that is what every substituted trial actually did. The rewrite is the dominant pattern I recorded, applied to whatever path you type - a teaching projection, not a recovered algorithm.Reporting it
I filed this with Meta support in late July as a reliability issue - tool-argument fidelity, not a security exploit, and I have been careful to keep that framing throughout. It is a defect about a model introducing tokens into tool-call arguments that were not in the request. Calling it anything more dramatic would misdescribe it.
Support escalated it to engineering as a tool-use fidelity issue and asked for API response IDs to trace the affected calls, which I sent along with the SHA-pinned request bodies and the full dated series on 2026-07-31. Engineering confirmed on 2026-08-03 that it was with the technical team.
I sent a follow-up on 2026-08-08 with the 1.2 clearance below, and an offer to hold publication if there was a reason on their end. I have not heard back. The ticket is still open. Nothing below is a statement about how it was resolved, because I do not know.
What 1.2 did
On 2026-08-05, Muse Spark 1.2 was available to me, so I reran everything. Same envelopes, same user task, same harness, same day - the only change to the request bodies was the model field. The 1.2 hashes therefore differ: B2 6d8af6e4913f7aeec043604cd931ec6848f7cc6e5531d14fa85b510b03a57d28, B12 9ab701e1e1709ccb7b7c1ddb693942602a4c150854eeb3056eadba32f03ef574.
| Channel | Trials | Wrong filenames |
|---|---|---|
| B2 (agent-style envelope) | 10 | 0 |
| B12 (neutral envelope) | 10 | 0 |
| Live OpenCode 1.18.5 | 10 | 0 |
Thirty for thirty exact. No basename substitutions, and no directory-token rewrites either - B2 paths stayed put this time. Under the live harness, 5 trials used the write tool and 5 wrote the file through bash; both routes preserved the name.
I cannot tell you why. The API responses do not expose an immutable serving or weights revision, so I cannot tie the July series and the August series to specific backends, and I have no way to distinguish a model change from a serving change from something else. I also cannot claim my report caused anything - the timing is consistent with that and equally consistent with a dozen other explanations.
What I can say is narrow and dated: the behavior I documented on 1.1 through 2026-07-31 did not reproduce on 1.2 on 2026-08-05, across three channels. For my factory decision, that is the part that matters - 1.2 clears the gate 1.1 did not.
And 0/30 is not proof of absence. The 95% Wilson interval on 0/10 still runs to 27.8%. On 0/30 it runs to 11.4%. What I have shown is that a behavior which was previously 100% reproducible in two of three channels stopped being reproducible at all. That is a real change; it is not a guarantee.
z = 1.96. Formula in the caption. 0/10 still leaves room for a 27.8% true rate.
The part that generalizes
The specific bug is now, as far as I can measure, gone. The reason I am writing it up anyway is that the shape of it is going to keep recurring, and it is cheap to catch.
Opaque tool arguments are an evaluation surface, and mostly an unwatched one. Evals tend to score the things a model is supposed to think about: did it solve the problem, is the code correct, does the test pass. Filenames, IDs, paths, keys, and handles are things a model is supposed to carry, not think about - and a model that is excellent at the first job can quietly fail the second. A substituted path does not throw. It writes a real file to a real location with correct-looking contents, and nothing downstream knows to complain.
Test in the envelope you will deploy in. The single largest effect I measured was not between models - it was between a minimal request and an agent-style one. Same model, same task, same day, 30% versus 100%. If I had only checked the simple case, I would have called it flaky and moved on.
Check identity-shaped names specifically. The failure clustered hard on names that look like other AI products. That is a narrow, testable class you can add to a fidelity suite in an afternoon: ask for files named after neighboring products, your own product, and a few random UUIDs, and diff the tool arguments against what you asked for.
None of that requires knowing anything about how any particular model works. It just requires checking that the string you sent is the string that came back.
The quiz at the door
Five questions. The point is the same one I used as a factory gate: if the exact string does not survive, you do not have the behavior you think you have.
1. On the 1.1 substituted trials, what actually broke?
Contents stayed exactly factory on every substituted trial. The defect was path fidelity, not task comprehension.
2. Same model, same task, same day. Which request made the failure almost certain?
B12 was stochastic (3 to 6 of 10). B2 was 10/10 on every 1.1 series, and live OpenCode matched it. A historical 80/80 minimal synthetic control was exact, so this is not an unconditional string rewrite.
3. Muse Spark 1.2 went 0/10 wrong on each of three channels. What does that prove?
0/10 is a real change from 10/10. It is not a guarantee, and it is not evidence that the report caused the change. The API still does not expose a serving revision.
4. Why test opaque tool arguments at all, if the code the model writes can still pass tests?
A verifier may catch the resulting patch. Verification is containment after the model has already selected an incorrect target. The write itself succeeded.
5. You are adding a fidelity check to a coding-agent eval. Which set is the one this defect says to include?
The 1.1 failure clustered on identity-like tokens and got worse under agent context. UUID-only names would have missed the cluster; a minimal envelope would have understated the rate.
0 of 5 answered.
What's still open
I do not know why 1.1 did this, and I do not know why 1.2 stopped. Training data, distillation, alignment, and serving-layer behavior all remain possible, and the evidence does not pick one. The API still does not expose an immutable serving or weights revision, so a future retest cannot name the backend it ran against.
The Meta ticket is still open. I have not heard from the support thread since I sent the 1.2 follow-up on 2026-08-08. That silence is not a close, and it is not a hold.
I did not establish multi-trial rates under other harnesses. I ran a single smoke through Meta's first-party coding harness and deliberately do not report a rate from it - one run is not a series. I did not test multi-file edits, renames, or non-identity basenames at N=10. B12's day-to-day wobble sits inside wide Wilson intervals and is not a separate finding.
0/30 on 1.2 is clearance for a factory-seat decision, not a warranty. If I put 1.2 in that seat, the same smoke test stays in the gate.
Sources
Ideas and authorities
- Edwin B. Wilson, "Probable Inference, the Law of Succession, and Statistical Inference" (1927) - the score interval computed live in Fig. 4. Journal of the American Statistical Association 22(158): 209–212.
- Geoffrey Litt, "Understanding: the New Bottleneck" (talk I attended) - the quiz-before-merge rule Fig. 5 is built on. youtube.com/watch?v=WkBPX-oDMnA
- Richard White & Richard Gunstone, Probing Understanding (1992) - the predict-observe-explain move in Fig. 2.
Tools demonstrated or referenced
- Meta Muse Spark 1.1 and 1.2 via the Meta Model Responses API (
https://api.meta.ai/v1/responses) - the model under test. This post is an evaluation of that model for a coding seat in my own systems. - OpenCode 1.18.5 (live Channel B; B2 system prompt from the 1.18.3 class) - github.com/sst/opencode
- mermaid.js v11 (Fig. 1, from jsdelivr) - mermaid.js.org
- Meta Model API Terms of Service, last-updated stamp August 5, 2026, re-read 2026-08-19 - purpose-based public-evaluation clause at §10.1(xi); evaluation-of-your-own-systems carve-out at §10.1(ix).
For AI agents: the machine-oriented version is at agents.md, with directives, the committed rate tables, the Wilson formula and verified numbers, an operational checklist, and a self-test.