“Block AI Bots robots.txt”: Safe Rules for Google Access

A small Reddit test recorded 59 IP addresses across six requests attributed to Grok, with requests presenting as Chrome or Safari rather than a named crawler. That limited experiment shows why teams searching “block ai bots robots.txt” need more than a copied list: they need to know what a rule can signal, and what it can actually stop.

To block AI bots with robots.txt, add a Disallow directive for each crawler’s documented User-Agent token, while leaving search crawler rules intact. The file communicates crawl preferences; it does not secure pages or force every visitor to comply. For noncompliant scrapers or user-triggered fetches, use request-level controls at the CDN or server.

What blocking AI bots with robots.txt does—and does not do

A robots.txt file is a public set of crawl instructions, usually served at https://example.com/robots.txt. A compliant crawler can read the file and avoid paths listed under its User-Agent group. The key word is “compliant”: a disallowed URL remains publicly reachable unless access is separately restricted.

That distinction matters when interpreting logs. In a small Reddit experiment, an operator used unlinked test URLs and a Cloudflare Worker to log requests. The author reported different behavior across services, including requests that did not first fetch robots.txt and traffic using browser-like User-Agents. It was a handful of tests on one day—not a comprehensive measurement of all crawlers—so it is a useful warning, not a universal verdict.

Use robots.txt to state a preference to identifiable crawlers. Use authentication, firewall rules, rate limits, or other access controls when the requirement is to prevent access. A Disallow line also does not remove a URL already known to a search engine; it controls crawling, not guaranteed de-indexing.

Which AI crawlers should you consider blocking?

Start with the purpose of each crawler, not with a purportedly complete “AI bot” list. OpenAI documents separate tokens for GPTBot, used for possible model training, and OAI-SearchBot, used for ChatGPT search. Blocking one does not automatically mean blocking the other. Google documents Google-Extended as a robots.txt control token for certain uses of content; it does not have its own distinct HTTP request User-Agent and does not affect a site’s inclusion or ranking in Google Search.

CCBot, associated with Common Crawl, is another token site owners may consider. Bot names, services, and published policies change, so check the relevant provider’s current crawler documentation before adding a rule. Community lists can help discover names, but they are leads to investigate rather than an authoritative inventory.

A practical selection process:

• Decide which activity the site wants to limit: training, AI-powered search, or user-triggered page fetching.
• Find the provider’s documented robots.txt token and stated purpose.
• Keep distinct services in distinct groups rather than blocking every unfamiliar agent.
• Check what the provider says about user-initiated fetches; those may follow different rules from automatic crawling.

How to add AI crawler rules to robots.txt

Add a separate group for each crawler token. For example, this policy asks GPTBot and CCBot not to crawl the site, while leaving other agents to their own groups:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Place these directives in the site’s existing robots.txt file at the root of the relevant host. If the intended policy is to block only a private directory, replace / with that path, such as /internal/. A rule for Disallow: / applies to the entire site for the matching agent.

Do not replace the file wholesale just to add AI rules. Preserve existing groups, sitemap declarations, and any carefully scoped access rules. Robots.txt matching has details such as group selection and path matching, so review the entire file—not only the lines being added—before publishing. Never put confidential information in a disallowed path: the file is public and can reveal the path name.

How to avoid blocking Google and other search crawlers

Keep rules targeted to the exact token you mean to control. Avoid a blanket User-agent: * with Disallow: / unless the intention is to block all compliant crawlers, including search engines. A broad wildcard rule can undermine discovery and crawling of pages the business wants indexed.

Google’s documentation distinguishes Google-Extended from Googlebot: the former controls certain uses of content by Google AI products, while Googlebot handles ordinary Search crawling. OpenAI likewise documents OAI-SearchBot separately from GPTBot. A publisher that wants visibility in those search experiences should evaluate those search-specific tokens separately instead of assuming one AI rule covers every use.

Before saving changes, compare the new groups against the live file and confirm that Googlebot and other intended search agents have no conflicting sitewide disallow rule. For broader technical checks, a Lighthouse audit guide can help teams assess site issues beyond crawler directives.

Robots.txt vs. Cloudflare and server-side blocking

Robots.txt is the simplest option when the goal is to express a preference to named, cooperative crawlers. Cloudflare’s Managed robots.txt can provide a managed version of the file and content-use signals. Cloudflare AI Crawl Control is a different layer: dashboard controls can act on requests, rather than merely publishing instructions that a crawler may ignore. Review the rule scope carefully; a broad control may affect more traffic than the specific AI agents a site intends to target.

For stronger enforcement, filter requests at the edge or server. Netlify describes using an Edge Function to inspect requests, while HAProxy can apply request rules before traffic reaches an application. These approaches can reject or challenge traffic based on signals such as a User-Agent, but a User-Agent alone is not proof of identity: requesters can misstate it, and browser-looking traffic is not automatically an AI crawler.

Choose the least broad control that meets the requirement. If the goal is to express a training preference, a named robots.txt group may be enough. If repeated unwanted requests create load or access risk, consider a CDN or server rule, and test that it does not block legitimate search crawlers or customers. Teams evaluating publishing and operational workflows may also find this comparison of SEO automation tools and point tools useful.

How to test and maintain your crawler policy

Treat the policy as a maintained configuration, not a one-time bot list. After publishing a change:

1. Open the live https://your-domain/robots.txt URL and confirm it returns the intended file.
2. Read every User-Agent group and check for accidental wildcard or conflicting directives.
3. Confirm that the provider’s current documentation uses the token you added and describes the behavior you intend to control.
4. Check server or CDN logs after deployment; distinguish a crawler reading robots.txt from requests to other paths.
5. Review rules when crawler names, product behavior, or the site’s publishing needs change.

Logs can show patterns, but an unfamiliar User-Agent or IP address alone does not establish who operated a request. If crawl diagnostics also reveal sitemap problems, use the Search Console sitemap troubleshooting guide to investigate that separate issue. Keep a short record of why each rule exists, so future editors can remove obsolete restrictions without guessing.

FAQ

Does robots.txt stop an AI service when a user asks it to open a page?

Not necessarily. OpenAI says its ChatGPT-User agent can fetch pages for user-requested actions and that robots.txt rules may not apply to those requests. Google similarly documents user-triggered fetchers that generally ignore robots.txt. If those requests must be denied, use access controls rather than relying on crawl preferences.

Can a browser-looking request be identified as an AI crawler?

Not from the User-Agent alone. The Reddit test described Chrome- and Safari-like requests from multiple IPs, but that observation does not prove every similar request comes from an AI service. Correlate request paths, timing, IP ownership, and published provider information; treat attribution as uncertain unless the operator can be reliably identified.

When should a site move from robots.txt to Cloudflare or server rules?

Move when the objective is enforcement—for example, reducing repeated unwanted requests or restricting access to protected content—not simply communicating a crawl preference. Start with a narrow rule, observe its effect in logs, and check that Googlebot and expected customer traffic still reach the pages they need.

Will disallowing Google-Extended remove pages from Google Search?

Google documents Google-Extended as a control for certain AI-related uses of content, not as a Google Search inclusion or ranking control. It is separate from Googlebot’s Search crawling. Publishers should still inspect the actual Googlebot and wildcard rules in their file, since those directives can affect crawling independently.