The AI Crawler Checklist
Follow
Mathew PaquetChecklist · 4 min read

The AI Crawler Checklist

Every crawler name Meta, OpenAI, Anthropic, Google and Perplexity publish, what each one does, and the robots.txt lines to copy before Meta's new business terms take effect on November 20.

01 What Meta's terms cover

Meta's updated business terms (its Commercial Terms) take effect on November 20, 2026. Read any terms update with three questions: who is covered, what can they take, and what can they do with it.

Who: anyone using ads or marketing messages, selling or listing products, creating a product catalog, or managing a Page or a Group for business. You don't need an ad budget: a Page you run for your company is inside it. What: Meta, or anyone acting for it, may use bots and scrapers on the domains, web pages and URLs associated with your products or services. That's your website. What for: the first version said provide, develop, improve, research, train, and maintain. The revised version says provide, develop, improve, research, and maintain advertising and business products, plus the integrity and security of Meta's products.

Nieman Lab reported there is no option for business accounts to opt out. Meta says the update does not change how it crawls the web, and that robots.txt preferences are respected; the terms themselves don't say so.

02 Every bot has a name

A crawler is a program that reads web pages automatically. Each one announces a name (its user agent), and robots.txt rules use that name. There are three kinds: training, search, and fetching a page because a person asked.

Company Name What it does, in the company's words
Meta facebookexternalhit Builds the preview when someone shares your link
Meta meta-webindexer Improves Meta AI search results, including citing and linking to sites
Meta meta-externalads Supports advertising and other business products
Meta meta-externalagent Trains foundation AI models or improves products by indexing content directly
Meta meta-externalfetcher Fetches a single link when a user asks, for features like agentic AI
OpenAI GPTBot / OAI-SearchBot / ChatGPT-User Training / search / when a person asks
Anthropic ClaudeBot / Claude-SearchBot / Claude-User Training / search / when a person asks
Google Google-Extended A token for Gemini; Google says it doesn't affect your place in Google Search
Perplexity PerplexityBot Search, not training

OpenAI says each setting is independent: you can block one kind and not the others.

03 Read your own robots.txt
Open it. Type your domain followed by /robots.txt into a browser, for example yourbakery.com/robots.txt.
No file? Then nothing is blocked. The standard says a crawler may read everything when the file isn't there.
A file? Read every "User-agent" line. Someone may have put rules in years ago that you never chose. Ahrefs says many sites block AI by accident through rules they inherited.
Decide by kind. Training, search, fetching: write down what you want for each.
04 The lines to copy

For most businesses the choice isn't block everything. It's block training if you want to, and keep the bots that send people to you. OpenAI says a site that blocks its search bot won't be shown in ChatGPT's search answers, and Anthropic says turning off its search bot may reduce your visibility. These lines ask the training crawlers named above not to read your site and leave search and link previews alone:

robots.txt: block training
User-agent: meta-externalagent
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

How you edit the file depends on your platform; your host or developer can do it. Meta says to allow up to 24 hours for a change. Don't block facebookexternalhit out of habit: it builds the preview when your page is shared on Facebook, a different job from the AI crawlers.

05 A sign, not a lock

Robots.txt is a request. Law professor Benjamin Sobel compared it to a sign that says "don't look at my house". The standard behind it says these rules are not a form of access authorization. Meta's own page says meta-externalfetcher may bypass robots.txt because a user asked, and facebookexternalhit might bypass it for security checks. Perplexity says the same about its user bot.

Check your server logs. Search them for the names in section 2 to see who actually visits.
Ask for a lock if you need one. A lock is your server or your content delivery network refusing requests by name. Ask your host or developer what they can enforce.
Put November 20 in your calendar. Meta can change this text again. This checklist reads the version as of late October 2026.

Sources

Nieman Lab: Meta's new commercial terms will allow the scraping of news sites (October 15, 2026) Nieman Lab: Legal experts warn Meta's new terms still grant broad scraping permissions (October 17, 2026) Meta Commercial Terms Meta Web Crawlers OpenAI: Overview of OpenAI Crawlers Anthropic: how site owners can block the crawler Google: common crawlers (Google-Extended) Perplexity Crawlers RFC 9309: Robots Exclusion Protocol
Mathew's LibraryLibrary card
Free access

Keep reading free with your library card

One email opens this checklist and all 3 resources in Mathew's library. No account, no password.

It opens right here, and a copy goes to your inbox. You'll also get The Operator Notes. Unsubscribe in one click.

Thanks
You finished it
Now do one thing

Type your domain followed by /robots.txt into a browser and write down what it says, or that there is no file.

Template The Bot Cruelty Plan
Up next in Trust & RiskThe Bot Cruelty Plan: Three Decisions for a Support Bot Built on Claude
Want it set up for you?Message Mathew's team

Saving works best in your phone's browser. A copy is also in your inbox.