Check That AI Can Reach Your Site

The one thing that can undo everything else

Every AI engine reaches your website through an automated visitor called a crawler or bot. Your site can quietly tell those bots to go away, often without anyone intending it. Security plugins, a nervous web developer, a hosting provider's default setting, or a well-meaning “block AI scrapers” checkbox can all shut the door. It is worth twenty minutes to check, because if the door is shut, the best-written page in your industry will never be quoted.

How to find your robots.txt

Type your website address into a browser and add /robots.txt at the end, like yourbusiness.com/robots.txt. It's a plain text file that every site has, and it lists which automated visitors are welcome. Select all the text and copy it. If you get a “not found” page, that usually means nothing is being blocked, which is fine, though the prompt below will tell you what's worth adding.

The three families of AI bots

• •Retrieval crawlers build the index the AI searches when answering. These include OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, plus Bingbot and Googlebot. These are your citation pipeline.
• •User-triggered fetchers show up when a real person asks an assistant to read a specific page: ChatGPT-User, Perplexity-User, Claude-User. Someone is actively looking at you when these visit.
• •Training collectors gather text used to train future models: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended. These feed the model's long-term background knowledge of your brand, not today's answer.

Which ones to allow

For a small business whose goal is to be found, the answer for the first two families is straightforward: allow them. Blocking a retrieval crawler or a user fetcher removes you from the fastest-growing discovery channel there is, which is precisely the opposite of what you're here to do. Big news publishers often block these bots deliberately, because their content is their product and they want licensing leverage. That calculation does not apply to a business that wants customers to find it.

Training collectors are a genuine business decision, not a visibility requirement. Allowing them means your content may inform how future models describe your industry and your brand, even without a link. Blocking them is a reasonable stance if you'd rather not contribute your writing to model training. Either way, it will not remove you from tomorrow's ChatGPT answer, and blocking is forward-looking only: it does nothing about content that has already been collected.

Don't forget Bing

Most owners haven't thought about Bing in a decade, and that's a mistake now. ChatGPT and Microsoft Copilot lean on Microsoft's index, so if Bing hasn't properly indexed your site, a huge slice of AI search simply can't see you. Bing Webmaster Tools is free, takes a few minutes to set up, and lets you submit your sitemap directly. Do the same with Google Search Console if you haven't already.

What about llms.txt?

You may run into advice telling you to add a file called llms.txt to help AI understand your site. As of today, no major AI engine uses it. Google has said plainly that it does not support it and has no plans to, and server logs show the engines don't even check whether the file exists. One study of about 300,000 domains found roughly one in ten had added it, with no measurable effect on citations. It costs almost nothing to add, but it will not help, so don't let anyone sell it to you as an AEO service.

Step by step

• •Visit yourbusiness.com/robots.txt and copy everything you see.
• •Run the AI Crawler Access Check prompt below, pasting the file in where it asks.
• •Read the bot-by-bot table it gives back, and decide your position on the training bots.
• •If changes are needed, send the corrected file it produces to whoever manages your website. This is usually a two-minute edit.
• •Set up Bing Webmaster Tools and submit your sitemap while you're at it.