Mathew PaquetTemplate · 4 min read
The Twenty-Case AI Model Switch Test
A fill-in sheet to test two AI models on twenty real cases of your own before you switch: what good looks like, pass or fail, and the cost per approved result.
On October 16, 2026, TechCrunch reported that the CEO of Replit, a platform where you describe an app and an AI agent builds it, said onstage that ChatGPT is now the default model behind Replit Agent, over Anthropic's Claude. An insider told the same article the partnership with Anthropic is still in place. Either way, the default moved, and TechCrunch pointed out that most customers stick with the default rather than picking a model themselves.
So start with a list. For each workflow, write which model it uses, and whether you pinned it (stable until the model retires) or follow the tool's default (it moves with the vendor). Both are fine. Just know which you chose.
Tool: [THE APP OR CODE THAT RUNS IT]
Model today: [MODEL NAME]
Pinned or default: [PINNED / DEFAULT]
Who hears when it changes: [NAME]
If a tool picks the model for you, ask the vendor two things: which model is it using today, and how will you hear when that changes. If the answer is nothing, that tells you how much of the setting you own.
When you can skip this. If you call one model by name from your own code, at low volume, with a handful of prompts, staying put and re-checking when the vendor announces a retirement can be the right call.
The harness is the code and instructions around a model. Replit's blog put it this way on September 29, 2026: "Every model release invalidates assumptions baked into the harness." So keep each workflow's prompt in one file, with one line saying which model it was tuned on. When a new model fails the test, you fix one file, not twelve automations.
Take twenty real cases, including the awkward ones, and write one line on what good looks like before you run anything. For product descriptions that line is: the right length, the real materials, no invented claims. That's an afternoon. Then run both models on the same twenty and mark each pass or fail.
Case | Input | Model A | Model B | Why it failed
1 | [A NORMAL CASE] | [P/F] | [P/F] |
2 | [AN AWKWARD CASE] | [P/F] | [P/F] |
... | ... | | |
20 | [THE HARDEST ONE] | [P/F] | [P/F] |
Passed: Model A [ ]/20 Model B [ ]/20
Same cases failing on both: [LIST] -> prompt problem, not model problem
Worked example (a made-up shop, invented numbers): the first model passes 18, the second 15, and the same three fail on both because each invented a material. That's the prompt, not the model. Add one line to the prompt, and the second passes 17. Without the test, a customer would have found it.
Replit compares cost per task, not price per token. Do the same: divide what the twenty tries cost by the results that passed.
The cheaper one wins, unless the miss is expensive.
A switch can be one line of settings and still cost you. Anthropic's own documents for using Claude through OpenAI's code library say you change the address, the key and the model name, then list what changes: a setting that makes tool calls follow your exact format is ignored, and prompt caching isn't supported through that layer. Replit's blog adds that on some models a switch rebuilds the cache, which means paying full price again for context you already sent.
Sources
Mathew's LibraryLibrary card
Keep reading free with your library card
One email opens this template and all 86 resources in Mathew's library. No account, no password.
Pick one AI workflow and write down which model it uses today, and whether you pinned it or follow the tool's default.
Saving works best in your phone's browser. A copy is also in your inbox.