Don't Trust Muse Wonder Benchmarks — Run a Real Bake-Off Against Opus 5.5 on Your Own Codebase

Meta's charts show Muse Wonder beating Claude Opus 5.5 on long-horizon coding tasks: DeepSWE scores 78.8 vs 74.2, SWE-Marathon 53.1 vs 46.3. Yet independent evaluator Artificial Analysis gives Wonder only 52 on its Intelligence Index, while Opus 5.5 scores 58. The two datasets don't contradict each other — they measure different dimensions, and neither benchmark ran on your codebase.

Meta released Muse Wonder in October 2026. In the Hacker News discussion thread, one commenter ran an image-to-HTML test and called it "much worse than Opus 5.5." But that was just a single prompt. This article isn't offering a verdict on Muse Wonder — it's offering a methodology: hand the same ticket to two agents, run them in separate git worktrees, and score both with one script.

Why the Two Benchmark Sets Disagree

Among the 14 benchmarks Meta published, Wonder won 5, mainly in long-horizon coding, legal agent tasks, and Meta's internal memory tests. It lost 9, including ProgramBench — where Opus leads by a wide margin, 70.5 to 50.0.

Meta's own footnotes spell out the limits of single-line numbers. For competitor tests run internally, Meta admits it uses "common settings where feasible, but it's still best-effort: prompts, tools, and runtime may not be optimized for proprietary models." The model card's limitations section echoes this: "Performance may vary with quantization, serving configuration, prompts, tools, and agent harness."

No leaderboard can fix prompts, tools, and harness for your environment — but a bake-off on your own codebase can do exactly that.

How SWE-bench, SWE-Marathon, and the Intelligence Index differ: The SWE-bench family is built on real GitHub issue-fix pairs and measures single, well-scoped problem-solving. SWE-Marathon stretches tasks to dozens or even hundreds of consecutive edits, testing whether an agent stays on track over long contexts. Artificial Analysis's Intelligence Index is a composite suite mixing reasoning, knowledge Q&A, instruction following, and more — its weighting doesn't favor agentic coding. So a model that looks strong on one suite and weak on another usually isn't a case of faked data; it's a difference in task distribution and evaluation protocol. Understanding this is what explains why Meta leads on internal long-horizon tests but trails on an independent composite index.

Preparation: One Ticket, One API Key

Pick a real ticket that requires a currently failing test (or write one first), whose fix touches several files, and whose outcome is judgeable. Save the prompt to a file outside the repo so every agent gets exactly the same text, and specify the test command inside it. "Make npm test -- slug pass without modifying the tests" draws the same finish line for the agent and for you.

Next, create a Meta Model API key and export it under the name given in Meta's coding agent documentation. The API model ID is muse-wonder-1.0. Note that the API doesn't offer Muse Wonder Full — that's exclusive to Muse Code subscribers.

export MODEL_API_KEY="your-key-here"

Session A: Claude Code Running Opus 5.5

In the Hermes IDE, press ⌘N (Ctrl+Shift+N on Windows/Linux), select Claude Code in the agent options, choose Opus 5.5 as the model, paste the prompt, and hit enter. Claude Code uses its usual login method. The task branches off the current branch, starting in a separate git worktree and a new hermes/<task> branch.

Session B: The Same Claude Code Running the Meta Model

Meta's documentation defines three environment variables that point Claude Code at its API: ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN, and ANTHROPIC_MODEL. The docs also require setting the Opus, Sonnet, and Haiku aliases and the subagent model to the same ID, "to ensure no requests fall back to Claude models." Skip this step and some requests will use a Claude model ID instead of Muse Wonder — and Session B stops being a clean Muse run.

Seven variables make up one long command; better to save them as a wrapper script called claude-muse on your PATH:

#!/usr/bin/env bash
export ANTHROPIC_BASE_URL="https://api.meta.ai"
export ANTHROPIC_AUTH_TOKEN="$MODEL_API_KEY"
m="muse-wonder-1.0"
export ANTHROPIC_MODEL="$m"
export ANTHROPIC_DEFAULT_OPUS_MODEL="$m"
export ANTHROPIC_DEFAULT_SONNET_MODEL="$m"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="$m"
export CLAUDE_CODE_SUBAGENT_MODEL="$m"
exec claude "$@"

The base URL has no /v1, and the key goes in AUTH_TOKEN, not API_KEY. In Hermes, press ⌘N again, select Custom agent, name it Claude Code · Muse Wonder, set the command to claude-muse, and paste the same prompt.

At this point Sessions A and B are handling the same ticket with the same harness — the only variable is the model and the API behind it. That's a controlled common-harness comparison. Meta itself concedes in a footnote that its SWE-Marathon row "is not a controlled common-harness comparison."

Not Using Hermes? Plain git Works Too

The isolation part takes one command per agent:

git worktree add -b bakeoff/opus ../bakeoff-opus
git worktree add -b bakeoff/muse ../bakeoff-muse

Then run claude in ../bakeoff-opus (use /model to pick Opus 5.5) and claude-muse in ../bakeoff-muse, each in its own terminal. The trade-off is losing the unified observation interface and the review step that follows.

Why pointing at a third-party API requires resetting these environment variables: Claude Code defaults to Anthropic's official endpoint and maps model aliases (Opus/Sonnet/Haiku) to Anthropic's own model IDs. Once ANTHROPIC_BASE_URL is rewritten to Meta's endpoint, requests go to Meta — but if you don't also swap the aliases and subagent model variables, the harness still constructs requests with Claude model names internally, causing some calls to be rejected by Meta's endpoint or silently fall back to Anthropic. That's the fundamental reason the docs require setting all seven variables to the same ID — only by resolving every path to muse-wonder-1.0 can you guarantee the whole session is a pure Muse run.

Optional Session C: Meta's Own muse CLI

Session B tests the model inside Anthropic's harness. If you also want to test Meta's harness, install Muse Code — the installer creates a command called muse in ~/.local/bin:

curl -fsSL https://dev.meta.ai/install.sh | bash

Then add another Custom agent with the command set to muse. Muse Code is subscription-based; the Power Usage plan is $50/month and includes Wonder Full. Meta's product page lists macOS and Windows support. But there was an early negative report: in the same Hacker News thread, one commenter said the Muse Code sandbox "has never worked on Mac and Linux." Verify on your own machine before paying.

Scoring Both Worktrees with a Script

Once both agents say they're done, score them the same way. The script takes a base branch, a test command, and any number of worktree paths, and for each one outputs the branch name, whether tests pass, how many files changed, and lines added/removed relative to base:

#!/usr/bin/env bash

**The semantics of git worktree and diffing against the fork point**: `git worktree` lets the same repo check out different branches in different directories while sharing the `.git` object store, so the two agents don't interfere with each other. The script uses `git merge-base` to find the common ancestor of the base branch and current HEAD (the fork point), then runs `diff --numstat` between the working tree and the fork point. Compared with diffing directly against the base branch HEAD, this excludes noise from new commits on base during the test period and more precisely reflects the agent's net changes. Untracked files aren't counted by numstat, so the script counts them separately with `ls-files -o`. Understanding this diff logic is what lets you tell, when results look odd, whether the agent did nothing or the script's methodology is the problem.
# usage: compare-worktrees.sh BASE "TEST COMMAND" WORKTREE...
set -u

base=$1
test_cmd=$2
shift 2

for wt in "$@"; do
  branch=$(git -C "$wt" rev-parse --abbrev-ref HEAD)
  fork=$(git -C "$wt" merge-base "$base" HEAD)
  new=$(git -C "$wt" ls-files -o --exclude-standard | grep -c .)
  (cd "$wt" && bash -c "$test_cmd") >/dev/null 2>&1
  code=$?
  if [ "$code" -eq 0 ]; then result=PASS; else result=FAIL; fi
  stats=$(git -C "$wt" diff --numstat "$fork")
  files=$(printf '%s' "$stats" | grep -c .)
  added=$(printf '%s\n' "$stats" | awk '{s+=$1} END {print s+0}')
  removed=$(printf '%s\n' "$stats" | awk '{s+=$2} END {print s+0}')
  echo "== $wt"
  echo "   branch:  $branch"
  echo "   tests:   $result (exit $code)"
  echo "   files:   $files changed, $new untracked"
  echo "   lines:   +$added -$removed vs $base"
done

The script diffs the worktree against the fork point, so it captures uncommitted work the agent did. New files git isn't tracking yet show up in the "untracked" count rather than the line totals. That count is taken before the test run, to avoid test-generated files inflating the number.

Filling Out the Scorecard

The script covers the first three rows; note the rest as you observe: whether tests pass, lines added/removed, files changed, wall-clock time to finish, cost, and how many review comments are needed before merging.

Wall-clock time. Artificial Analysis measured Wonder's time to first token at 19.27 seconds, versus a median of 5.52 seconds among comparable models. An agent task makes many calls, so this gap will show up.

Cost. Meta's docs warn that agents pointed at custom providers often show $0 cost because pricing metadata is missing. Don't read cost from Session B's terminal — calculate it manually from input, cache, and output token counts: millions of input tokens × $4, millions of cached input × $0.20, millions of output × $20. Per-token pricing matches Opus 5.5, but cost per task can still differ. Artificial Analysis reports that Wonder consumed 130M output tokens running its index, versus a median of 85M.

Review and Merge the Winner in Hermes

Press ⌘G (Ctrl+Shift+G on Windows/Linux) to open the Review Desk, and switch to By turn mode. You can review each agent's work turn by turn rather than looking at the final diff all at once. Risk flags mark lockfiles, CI workflows, secrets, new binaries, and any changes the agent made to its own configuration.

Inline comments go back to the agent that wrote that line. Count those comments — that's the last row of the scorecard. Hermes knows less about Custom agents than built-in ones, so if Session B's turns look thin, switch to By file mode to read, or fall back to the script data plus a plain git diff against the base branch.

Merge the winner from the Land panel: commit, open a PR, or merge locally, then undo later if needed. Then close the other session and remove the losing worktree manually: git worktree remove ../bakeoff-muse.

The Limits of a Single Bake-Off

One ticket is just an anecdote. Before drawing conclusions, test three to five tickets of different shapes — a model that wins at refactoring may lose at bug hunting.

Hermes currently offers no status signal for Custom agents, doesn't know the command's approval model, and the attention inbox can't tell whether Session B or C is waiting on you. You need to watch their terminals. Session A keeps full status.

You're testing Muse Wonder, not Wonder Full. The API doesn't offer Full, and only Session C on a plan that includes it can touch it. Session B is Anthropic's harness running Meta's model: fair for comparing models, but possibly not how Meta tuned Wonder — which is exactly why Session C exists.

How quantization, serving configuration, and harness affect model behavior: The same model may run in different quantized versions across API, subscription, and different inference frameworks, and weight precision, batching strategy, and context truncation thresholds all change output quality. Meta's model card explicitly lists this, meaning the Muse Wonder you measure in a bake-off isn't necessarily the same as Wonder Full or Meta's internally optimized form. This is also why cross-vendor comparisons must fix the harness — if two agents use different tool-call formats, retry strategies, and system prompts, score differences may come from the scaffolding rather than the model itself.

Start Your Bake-Off Now

Pick a ticket you were going to hand to an agent this week anyway, run it in two sessions, and merge whichever result you'd rather review again. Arguments between models shouldn't be settled by someone else's benchmark — your own codebase is the final judge.

One-line Summary

A controlled bake-off of Muse Wonder vs Opus 5.5 on your own codebase, using the same harness, beats any benchmark.

Paragraph Summary

Meta claims Muse Wonder beats Claude Opus 5.5 on long-horizon coding tasks, but Artificial Analysis's composite index says otherwise. The article argues the two datasets don't contradict each other — they measure different dimensions, and neither ran on your codebase. The author proposes a methodology: pick a real ticket, create a Meta Model API key, use Claude Code in the Hermes IDE as a unified harness for both Opus 5.5 and muse-wonder-1.0 (pointed at Meta's API via environment variables), and optionally test Meta's own muse CLI. Both agents run in separate git worktrees, with a script automatically tallying test pass rates, files changed, and lines added/removed, then combined with wall-clock time, cost, and review comments to fill out a scorecard. The author stresses a single bake-off is just an anecdote and recommends validating with three to five tickets of different types before deciding.