4 Numbers to Check Before You Cut Jobs for AI

01 The number your boss is watching

In June, Meta's chief technology officer, Andrew Bosworth, posted a number to staff that looked like a win. Weeks after Meta laid off about 8,000 people, code changes were up 220% from a year earlier. The same post had two more numbers:

Changes that led to new or better features reaching users: up 36%. Major technical and security incidents, like outages and possible data leaks: up 40%.

The first number is the one most bosses watch. It counts what was made, not what reached anyone. Once a number becomes the target, people make the number. At Meta, an employee built a leaderboard that ranked staff by AI tokens burned (tokens are the units AI tools measure and bill use in), and some ran agents on many tasks at once to climb it.

Andrew Bosworth, in an April memo. "All motion is not progress, and token usage alone is not a measure of impact of any kind."

To be fair: 36% more improvements reaching users is real growth, and Meta hasn't quit AI. These numbers come from internal posts Reuters reported in August. The question isn't whether AI worked. It's where the rest of the output went.

02 Where the extra output goes

It goes to the people who check it. Everything AI makes still has to be read, tested and kept running. Make three times as much, and somebody has to look at three times as much. Meta had just cut the people who look. Its major incidents rose 40%, and the time staff spent firefighting them rose 70%. Meta's own posts tied the warnings to the AI coding surge, though nobody has proven the tools caused every fire.

It isn't only Meta. The developer research firm DX looked at more than 500 companies: over four quarters, code changes per week rose 37%, while the share of engineering time spent building new things stayed about flat.

A worked example: the twelve-person agency

What the dashboard shows What actually happened

Ads made: 40 a week becomes 120 One senior strategist now reviews 120 ads instead of 40

Output tripled 46 of them actually run

A big week One goes live with last month's price

03 The four numbers

Count these for a few weeks before you change anything. Otherwise you have nothing to compare to.

What reached the customer. Not ads made. Ads approved, running, and what they did. This is Meta's 36%. The checking load. Human hours of review per thing that ships, and whose hours they are. If output triples and review time doesn't move, nobody is really reviewing. If your senior person's calendar is full of review, nobody is doing the strategy. The fires. Mistakes that reached a customer, and the time to fix them: rejected ads, corrections, complaints, refunds. This is Meta's 40% and 70%. Cost per thing that shipped, AI bill included. Total cost of the work, AI bill included, divided by what shipped. In June, Meta told about 6,000 employees it would cap and track AI token spending. Even Meta now has an AI budget.

Weekly scorecard

Week of: [DATE]
1. Reached the customer: [APPROVED] approved, [LIVE] live. Result: [WHAT THEY DID]
2. Checking load: [HOURS] review hours, by [WHO], per [UNIT] shipped
3. Fires: [COUNT] mistakes reached a customer. Fix time: [HOURS]
4. Cost per thing shipped: ([PEOPLE COST] + [AI BILL]) / [THINGS SHIPPED] = [COST]

The number not to manage by. Volume, or AI use on its own. That's how Meta got a leaderboard title called Token Legend.

04 The order to change things in

Prove one task. For example, first drafts of ad copy, with a person checking every one. Watch the four numbers for a month.

If they hold, widen what the AI is trusted with. Decide how much review it still gets.

Only then change the team. Peter Bell, who is writing a book on scaling AI in engineering teams: shrink teams near the end of an AI change, not the beginning.

Move people before you remove them. Meta kept the option to transfer people back. Most businesses only get to make that call once.

The real cost of AI isn't the subscription or the tokens. It's the checking, the fixing, and the people who know when the output is wrong.

Sources

CIO: Meta's plans to replace workers with AI fell flat, report says LeadDev: Meta tried to shrink engineering teams around AI GIGAZINE: How did Project OT end NPR Illinois: Meta slashes 8,000 jobs as it pivots towards AI The Pragmatic Engineer: Tokenmaxxing as a weird new trend Michael Parekh: Meta steps back from AI tokenmaxxing AI Weekly: Meta caps employee AI token use IANS: Zuckerberg admits mistakes in AI workforce overhaul

Mathew's LibraryLibrary card

Free access

Keep reading free with your library card

One email opens this template and all 45 resources in Mathew's library. No account, no password.