The AI Model Scorecard – Mathew's Library

01 Why the price and the chart can't choose

On October 12, Meta released its first frontier models. On Meta's own launch chart, Muse Wonder Full and Claude Opus 5.5 win six rows each.

The version sold by the token is plain Muse Wonder. $4 per million tokens you send and $20 per million it writes back. Same as Opus, to the cent. The version winning the chart, Muse Wonder Full, isn't sold by the token. It's only in Muse Code, Meta's AI coding app, on plans that start at $50 a month. They're not the same model. On the Artificial Analysis Intelligence Index, an independent test, Full scores 56 and the per-token version 52. Opus scores 58.

When the price is the same, the price can't choose for you. A launch chart, read the usual way, can't either. The three checks below can.

The only chart with your work on it is the one you make.

02 Layer 1: the column

Before you read a single score, read the column headers.

Find the column that wins most rows. Write its exact name, including words like "Full", "max" or "preview".

Find that exact name on the pricing page. Is it sold per token, or only inside a subscription or an app?

Find the column you would actually buy. Read its scores, not the winner's. That's the model you're paying for.

03 Layer 2: the footnotes

A model card is the document that explains how a company tested its model. Look for two things in it.

Where each number comes from. Meta's card says that where no comparable outside result exists, it uses the highest of its own validated run, a self-reported score, or the official leaderboard score. Who ran the rivals. Meta calls its own runs of other companies' models best effort, and says the prompts, tools and setup may not be optimally tuned for them.

Then mark any row that is the maker's own test. On Meta's chart, the one row the per-token model wins is MemoryBench, a test Meta built: 85.6 against Opus at 81.5. An internal test isn't a wrong test. It's worth knowing whose ruler it is.

04 Layer 3: the row names, and the data setting

A launch chart is a list of the jobs the maker decided to aim at. Nobody's chart has a row for your monthly client report, your product pages, or your support emails in your brand's voice.

So read each row name and ask: does this look like my work? One row worth finding: Artificial Analysis runs a test of factual knowledge that also counts how often a model makes things up. There, Opus scores 46 and Wonder Full 30. If your AI writes client copy with facts in it, that row matters more than any coding score.

Check the data setting first. Meta's documentation says Muse Code subscriptions default to Contributor mode: more usage, in exchange for Meta using your prompts, code, files and answers to train future models. You can turn it off at any time. On Meta's per-token platform, standard mode content isn't used for training. Check this before anyone pastes in a client's work.

05 The five-job test

When people test on real work, the gap flattens. One developer ran four top models on 105 real bugs: 45, 44, 43 and 41.7. Four models within about three points. The gap between models is smaller than the gap between your jobs.

Take five real jobs from last month. Give two or three models the same instructions for each job. Judge the answers yourself, job by job, and pick per job.

My model scorecard

MODEL CHECK: [MODEL NAME AS SOLD]
Column: the version I can buy is [NAME], at [PRICE / PLAN]
Footnotes: rows the maker ran or built itself: [LIST]
Rows that look like my work: [LIST, OR NONE]
Data setting: [CHECKED ON / OFF] in [PLAN]

FIVE-JOB TEST (same instructions for every model)
Job 1: [JOB] Model A: [GOOD / FIX / FAIL] Model B: [ ] Model C: [ ]
Job 2: [JOB] Model A: [ ] Model B: [ ] Model C: [ ]
Job 3: [JOB] Model A: [ ] Model B: [ ] Model C: [ ]
Job 4: [JOB] Model A: [ ] Model B: [ ] Model C: [ ]
Job 5: [JOB] Model A: [ ] Model B: [ ] Model C: [ ]
The job where a wrong answer costs most: [JOB]
Pick for that job: [MODEL]

Sources

Meta: Muse Wonder announcement Meta: Muse Wonder model card Meta Model API: pricing Meta Model API: Muse Code subscriptions Artificial Analysis: Muse Wonder Anthropic: Claude pricing

Mathew's LibraryLibrary card

Free access

Keep reading free with your library card

One email opens this template and all 86 resources in Mathew's library. No account, no password.