Skip to content
createwithlogic

Free tool · no signup

Can AI crawlers
read your website?

Enter your website. The checker loads your home page once without running JavaScript, the way many AI crawlers fetch pages, and reads your robots.txt for eleven crawlers from OpenAI, Perplexity, Anthropic, Google and Microsoft. You see what a crawler gets, and for each crawler the line that decides. It uses the same rules as our studies of 200 CPA firm websites and 197 real estate agent websites.

We load your home page and robots.txt once and send back the results, never a copy of your page.

robots.txt, crawler by crawler

The scan fills in your robots.txt here. You can also paste a file by hand, or change a line and see what it would do.

The example lists the crawlers it allows and then blocks every other one, the pattern that kept PerplexityBot out of 24 of 184 CPA firm websites in our study.

Result

Paste a robots.txt to see which crawlers may read the page.

Crawler This page Decided by

Want the whole picture, not just your website?

The free AI visibility check also covers your Google profile and what ChatGPT, Gemini and Perplexity say about you. For US accounting firms, CPA firms and real estate agents. Leave your email, and I will ask for your firm or brokerage name, city and website.

One reply to ask for your details, then the check. No newsletter.

Tim Muenzel

Tim Muenzel

builds and runs the systems himself

Which crawlers decide what AI assistants can read

Each AI company runs more than one crawler, and they do different jobs. Search crawlers build the index an assistant searches: OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, Claude-SearchBot for Claude, Googlebot for Google Search with its AI Overviews and AI Mode, and bingbot for Bing and Copilot. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

User agents open a single page when someone asks about it: ChatGPT-User, Perplexity-User and Claude-User. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user started the request, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says its bots honor robots.txt.

Training crawlers collect data for future models: GPTBot for OpenAI and ClaudeBot for Anthropic. Blocking them is a separate decision that does not take you out of search. Google-Extended is not a crawler but a robots.txt token: Google says it controls whether your content may be used to train Gemini models and to ground answers in Gemini Apps, and that it does not affect Google Search.

What we found in two studies

In October 2026 we read the robots.txt files of 200 US CPA firm websites and 197 US real estate agent websites from Google Maps in the same 10 metros. None of the 184 CPA firm sites we could reach blocked OAI-SearchBot or ChatGPT-User. 24 of them (13%) blocked PerplexityBot, all on the same website platform for accounting firms: its robots.txt lists the crawlers it allows and ends with a rule that blocks every other crawler, and PerplexityBot is not on the list. 25 (13.6%) blocked GPTBot.

Of the 163 agent sites that answered with a page, 3 blocked OAI-SearchBot, two of them not live websites, 4 (2.5%) blocked PerplexityBot and 10 (6.1%) blocked GPTBot. Agent sites showed no cluster like the one among CPA firms.

robots.txt is only the first check

A file that lets every crawler in does not help if the firewall in front of the site turns them away. 23 of 189 real estate agent websites (12%) turned away a request without JavaScript, 16 of them with a Cloudflare page that asks the browser to prove itself by running JavaScript. Among CPA firm websites it was 13 of 200 (6.5%). Cloudflare announced on July 1, 2025 that it blocks AI crawlers on new domains by default and that owners can allow them at any time.

Vercel and MERJ reported on December 17, 2024 that none of the major AI crawlers rendered JavaScript, so a page whose text only appears through JavaScript was empty for them. In our own server logs on October 14, 2026, OpenAI’s search crawler ran the JavaScript on 4 of the 7 pages it fetched, and ChatGPT-User, PerplexityBot and GPTBot showed no sign of it. 6 agent websites and 3 CPA firm websites in our studies showed their text only through JavaScript. None of this shows when you open your own site in a normal browser. The scan at the top of this page checks both for your home page.

How the checker reads your site and your file

The website scan loads the address you enter once, from a data center in the eastern US, with a common browser user agent and without running JavaScript, and follows up to five redirects. An answer counts as turned away if it is not HTTP 200, or if it is a firewall check page with fewer than 50 words. The scan counts the visible words, reads the schema.org types in the page’s JSON-LD, including types written as a list, and loads robots.txt from the address where the page ended up. These are the rules of our two studies.

For robots.txt the checker follows the standard, RFC 9309, published by the IETF in September 2022. Rules are grouped under User-agent lines. A crawler obeys the group that names it, or the group for * if none does. Within that group the most specific rule wins, measured by its length, and Allow wins a tie. A * in a rule matches any characters, and a $ at the end ties the rule to the end of the address.

The checker is a JavaScript version of Protego, the open-source robots.txt parser behind the Scrapy crawler, which we used for both studies. We compared the two on the 377 robots.txt answers from both studies and on 4,000 generated files: 412,516 decisions, all identical. One difference is on purpose: a byte order mark at the very start of a file is ignored. Each AI company runs its own parser, so rare edge cases can read differently.

Common questions, answered straight

Does blocking GPTBot keep my site out of ChatGPT?

No. OpenAI uses GPTBot to collect data for training its models. ChatGPT search uses OAI-SearchBot, and OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. To appear in ChatGPT search, allow OAI-SearchBot.

Which crawlers should a CPA firm or real estate agent allow?

For AI search: OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, Claude-SearchBot for Claude, Googlebot for Google’s AI Overviews and AI Mode, and bingbot for Bing and Copilot. Allow ChatGPT-User, Perplexity-User and Claude-User as well, so a page opens when a client asks about it. Training crawlers such as GPTBot and ClaudeBot are a separate decision that does not affect search. Google-Extended does not affect Google Search, but Google says it also decides whether Gemini Apps may use your pages to ground their answers.

Do ChatGPT-User and Perplexity-User follow robots.txt?

Not reliably. Both open a page because a user asked for it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. The checker still shows what your file says for them. Whether your site can appear in their search results is decided by OAI-SearchBot and PerplexityBot.

What if my site has no robots.txt?

Then robots.txt restricts nothing. Under the robots.txt standard (RFC 9309), if the file is unavailable, for example with a 404 answer, crawlers may access any page. If the server answers with an error such as 500 instead, crawlers must assume the whole site is blocked.

My robots.txt allows everything. Why can an assistant still miss my site?

Because robots.txt is only the first check. In our study of 197 real estate agent websites, 23 of 189 (12%) turned away a request without JavaScript, most of them with a Cloudflare or Vercel check page, and among 200 CPA firm websites 13 (6.5%) did the same. 6 agent websites and 3 CPA firm websites showed their text only through JavaScript, which many AI crawlers do not run. And a readable site is only the start: what assistants say also depends on the sources they cite, such as your Google Business Profile and its reviews.

My site opens fine in my browser. Why does the checker see a check page?

Your browser runs JavaScript, and firewalls such as Cloudflare use that to tell people from bots: a request that does not run JavaScript gets a check page instead of the site. Crawlers that do not run JavaScript get past such a check only if the firewall is set to let them through, and nobody outside can see that setting. Firewalls can let known crawlers such as Googlebot through, so a check page for our request does not prove that every AI crawler is stopped. It shows where to look: the bot settings of your firewall or hosting.

What does the checker send, and what does it keep?

A robots.txt you paste stays in your browser. When you enter a website, our server loads that page and its robots.txt once and sends back only the results and the robots.txt, never a copy of the page. It keeps the result for ten minutes, so a repeat check does not load your site again. Measurement on this site is cookieless and records only which page was viewed.

Who made this checker?

Tim Muenzel, a systems engineer who runs createwithlogic, a US company (CREATEWITHLOGIC LLC, New Mexico). It uses the rules of our published studies of 200 CPA firm websites and 197 real estate agent websites, and gives the same answer on every robots.txt file from them. It is free and needs no signup.

Sources

Related reading

Want the whole picture?

The free AI visibility check covers your robots.txt, your firewall, how much text a crawler sees, your Google profile and what ChatGPT, Gemini and Perplexity answer for clients in your city. For US accounting firms, CPA firms and real estate agents, by email.

Get the free check Book a free 20-minute call