Template
Template
The Bot Cruelty Plan
The Bot Cruelty Plan: Three Decisions for a Support Bot Built on Claude
4 min
Trust & Risk
Checklist
Free
Checklist
The AI Crawler Checklist
Mathew PaquetTrust & Risk · Checklist
The AI Crawler Checklist
Every crawler name Meta, OpenAI, Anthropic, Google and Perplexity publish, what each one does, and the robots.txt lines to copy before Meta's new business terms take effect on November 20.
4 min
7 checks
1 copy-ready template
9 sources cited
One email gets you a library card. It opens this checklist and all 86 resources in Mathew's library, now and later.
Open the checklist
Your library card opens everything here
I'm the CEO of myMarketing and Ryv AI, and I make a video every day on what new AI changes for marketers and founders. Every resource here is built from those videos and fact-checked.
Swipe through the preview
What's inside
What Meta's terms coverWho is a business customer, what Meta's bots may crawl, and what for.
Every bot has a nameThe crawler names five companies publish, sorted into training, search and fetching.
Read your own robots.txtWhere to find it, and what to look for in two minutes.
The lines to copyBlock training, keep the bots that send people to you.
A sign, not a lockWhat robots.txt can't do, and what to ask your host.
Quick win you can do today. Type your domain followed by /robots.txt into a browser and write down what it says, or that there is no file.
You might also want
Trust & Risk
Checklist
Checklist
The Last Valve Check
The Last Valve: Four Checks to Keep Your Website Patched
Trust & Risk
AI Agents
Template
Template
The Real Yes Scripts
The Real Yes Scripts for AI in Meetings
4 min
Trust & Risk
Checklist
Checklist
The Three Doors Checklist
The Three Doors Exit Checklist
Trust & Risk
AI Agents
Checklist
The AI Crawler Checklist