What Its 90% Is Worth: Test an AI Decision Before It Touches Money

01 A number is only useful if 90 means 90

OpenAI's Decisions API opened to every developer in October 2026, as a public beta. It doesn't answer in sentences. You give it text or an image plus a question you've written, and it gives back one of three things:

A probability that something is true (a yes or no question, from 0 to 1). One pick from a list you wrote. A score on a scale you define.

It's cheap. OpenAI charges $0.10 per million tokens you send in, and nothing for the answer. Judging one long customer email costs about a hundredth of a cent. A hundred thousand emails, roughly $10. So the real question is whether your business can act on its numbers.

One developer tested it with a coin loaded to land heads 70% of the time, asked 1,000 times. As a yes or no question: heads 70% of the time. Exactly right. As a choice, heads or tails: heads 98% of the time. Changing the order of the options changed the numbers too.

Why it matters. Say 1,000 refund photos come back at 90%. If 90 really means 90, about 100 are wrong, and you can plan for that. If its 90 behaves like a 70, 300 are wrong. Every answer still looks clean and confident.

The early outside tests are small, and none of them is your data. OpenAI's guide doesn't give a cutoff either. It says to set your thresholds from labeled examples, based on what a wrong yes and a wrong no cost you. Here is how.

02 The 200-case test

Take 200 real cases where you already know the right answer. "Labeled" just means that: each one has the correct answer written beside it. For a returns desk, 200 refund photos you've already judged. Include the hard ones. The blurry photo. The crushed box with a perfect item inside. The cases that make your team argue. Anthus, which benchmarks these models, says easy problems flatter everyone: on the easiest problems the models it tested were within a few points of each other, and the gaps grew on the harder ones. Run them through. Ask the same question you'll ask in real use, worded the same way. Sort them by the number it gave. Count. Of everything it called 90 or higher, how many were really damaged? 18 out of 20: its 90 is a 90. 14 out of 20: you just learned what its 90 is worth, before it cost you anything.

My test result

Decision: [THE QUESTION, e.g. Is this item visibly damaged?]
Labeled cases run: [NUMBER], hard ones included: [NUMBER]
Called 90 or higher: [A]. Actually right: [B].
What its 90 is worth: [B] out of [A]
Cost of a wrong yes: [$] Cost of a wrong no: [$]

03 Two lines, and the price of the middle

Above the top line, it acts on its own. The refund goes out. Below the bottom line, no automatic refund. The claim goes the normal way. In between, a person looks.

That middle zone is your staffing bill. Worked example: 1,000 photos a day, 200 land in the middle, half a minute each. That's 100 minutes, almost two hours of someone's day, every day.

Middle-zone calculator

Cases a day: [A]
Cases between the two lines: [B]
Minutes a person spends on each: [C]
Minutes a day of someone's time: [B] x [C] = [TOTAL]

Pull the top line down and fewer cases need a person, but more wrong refunds go out. Push it up and fewer go out wrong, but someone spends the afternoon on photos. You're trading money against hours. Set the lines by what each mistake costs: a forty-dollar refund and a forty-thousand-dollar contract shouldn't share a line.

04 Where to start

Not with refunds. Start where a wrong answer is cheap and easy to undo.

Sending a support ticket to a team. A wrong one costs a few minutes, and a person bounces it back. Tagging a lead, like "pricing question" versus "demo request". A wrong tag costs almost nothing.

Run it there first, watch its numbers against what really happened, and move toward money and customers only once its 90 has earned your trust.

05 Five habits from the early tests

Ask yes or no when you can. That's the version that got the coin right.

For a pick from a list, keep the order fixed and add "other". Odd cases land with a person, not in the wrong box.

Plan for no answer. It can refuse. In one blogger's test it gave nothing back for "what is my card PIN?"

Keep a reason with every decision. A probability shows how confident it was, not why. "0.91" doesn't answer a customer.

Run the 200-case test again every month or so. The cases people send you will change.

The model gives you the number. You decide what the number is allowed to do.

Sources

OpenAI: Decisions API guide OpenAI Developer Community: Decisions API is now available in public beta StackToHeap: How does Jev's big-lab competitor fare? Anthus: The OpenAI Decisions API needs a confidence you can trust InfoWorld: TypeSafe AI's new models work with machines, not humans

Mathew's LibraryLibrary card

Free access

Keep reading free with your library card

One email opens this guide and all 53 resources in Mathew's library. No account, no password.