Estimated read time: 5 minutes

A website is crawlable by AI agents when the AI search crawlers are allowed in, nothing at the network layer turns them away, and the important facts sit in plain text. In 2026 one easy gap to miss is a blanket block aimed at training bots that also stops search bots. Fix access first, then content.

A site can look polished to people and still be closed to the systems that build AI answers. The owner may never notice, because the site loads fine in a browser. This piece explains which crawlers do which job, where the blocks hide, and how to open the door to AI search while keeping a say over training.

AI companies run separate bots for separate purposes. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for model training, and says the two controls are independent. ChatGPT-User visits a page when a person asks a question, and OpenAI notes robots.txt rules may not apply to it.

Anthropic follows a similar split. ClaudeBot collects content that may be used for training, Claude-SearchBot works to improve search results, and Claude-User fetches pages for a person's question. Perplexity says PerplexityBot surfaces and links sites in its search results and is not used to crawl content for AI foundation models.

Google points site owners to Google-Extended to limit AI training and grounding in some of its other systems. Eligibility for AI Overviews still runs through ordinary Google Search indexing.

The action: list every bot your robots.txt names today, and mark each one as search, training, or user fetch.

robots.txt is one gate among several. Hosting, CDN, and security tools can turn a bot away before it ever reads that file. Google's AI features guidance says to make sure crawling is allowed, including by any CDN or hosting infrastructure.

Cloudflare is a clear example. On July 1, 2025, it made blocking AI crawlers the default for every new domain. On July 1, 2026, it announced new defaults starting September 15, 2026: for new domains, Training and Agent bots are blocked on pages that display ads, while Search bots stay allowed.

The same announcement says customers who choose to block Training will also block multi-purpose crawlers such as Googlebot, Applebot, and Bingbot. A setting meant to protect content from training can also turn away the crawlers behind search.

The action: open your host or CDN dashboard, find the bot or AI crawler settings, and write down what each category is set to.

A crawler reads text. Services listed inside an image, a slideshow, or a downloadable PDF may never reach it. Google lists keeping important content in text form among its best practices for AI features.

Pages that build their content with JavaScript carry a second risk. Google renders JavaScript with a headless browser, but its own documentation notes that not all bots can run JavaScript, and it recommends server-side rendering or pre-rendering.

A quick test: try to select your service list with the cursor. If it will not highlight, it is probably an image. Then view the page source. If your key sentences are missing there, a script is building them.

The action: run that test on your services page and your contact page first.

A clean robots.txt names each bot on purpose. Allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Decide separately on GPTBot, ClaudeBot, and Google-Extended, based on how you view training use.

Skip blocking by IP address. Anthropic notes that blocking the IP addresses its bots use may not work correctly, because it stops them from reading robots.txt, where the opt-out lives. OpenAI says robots.txt changes can take about 24 hours to take effect.

Then check the result. Open yourdomain.com/robots.txt and read it. Ask an AI assistant to summarize a specific page, and note whether the answer matches the page. Repeat after each change and keep the notes dated.

Being recommended by AI starts with being readable by AI. Open the door to search crawlers, keep the facts in text, and make training a separate, deliberate choice. Once the machines can get in, the work on Authority, Recommendability, and Citation has something to build on.

This article contains affiliate links. I may earn a commission if you purchase through them, at no additional cost to you.

Sources checked October 24, 2026: OpenAI: crawler controls, Anthropic: crawler controls, Perplexity: crawlers, Google: AI features, Google: JavaScript SEO basics, Cloudflare: 2025 AI crawler default, Cloudflare: 2026 AI traffic defaults.