# Block bots with robots.txt, and know its limits

A `robots.txt` file asks well-behaved crawlers which parts of your site they may
visit. It does not stop bad bots, hide private pages, or protect your site.

Think of it as a public contract. Crawlers can follow it, ignore it, or lie
about who they are.

**tl;dr;** Use `robots.txt` to state your preference, and use server or CDN
controls when you need that preference enforced.

## What robots.txt can and cannot do

Put the file at the root of your site:

```
https://example.com/robots.txt
```

The file contains rules for one or more crawlers. This example lets crawlers
visit most of the site, but asks them to stay out of two paths:

```txt
User-agent: *
Disallow: /preview/
Disallow: /search

Sitemap: https://example.com/sitemap.xml
```

`User-agent: *` means any crawler that does not have its own group. `Disallow`
contains a path the crawler should not fetch. An empty `Disallow` allows
everything, so you usually do not need an `Allow: /` rule.

The current standard is [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html).
It says plainly that these rules are not access control. The file is public, so
listing a path can even draw attention to it.

If a page is private, put it behind a login or another real access check. Do not
rely on `robots.txt`. A tool such as `curl` does not need to read the file at
all:

```shell
$ curl https://example.com/preview/secret-page
```

`robots.txt` controls crawling, not search indexing. A search engine may still
show the URL of a blocked page if it finds a link to it elsewhere. To keep a
public page out of search results, use a `noindex` meta tag or an `X-Robots-Tag`
header and allow the search crawler to read it. For private content, require a
login. Google has a good guide to
[robots.txt and indexing](https://developers.google.com/search/docs/crawling-indexing/robots/intro).

## Blocking one crawler

To ask one crawler to stay out of the whole site, name it and disallow the root
path:

```txt
User-agent: ExampleBot
Disallow: /
```

To ask all crawlers to stay out:

```txt
User-agent: *
Disallow: /
```

That second example can remove your content from search and other useful
services. Use it only when that is what you want. It still does nothing against
a crawler that ignores the file.

## AI crawlers

AI companies run different crawlers for different jobs. One may collect pages
for model training while another powers search results or fetches a page when a
person asks a question. Blocking every AI-related crawler may also stop your
pages from appearing with links in AI search.

If you want to allow search but opt out of some training crawlers, a file could
look like this:

```txt
# OpenAI model training
User-agent: GPTBot
Disallow: /

# Anthropic model training
User-agent: ClaudeBot
Disallow: /

# Use of Google-crawled content by Gemini
User-agent: Google-Extended
Disallow: /

# Keep OpenAI search discovery enabled
User-agent: OAI-SearchBot
Allow: /
```

These names and their uses can change. Check the crawler owner's current docs
before copying a rule. OpenAI explains the difference between
[GPTBot and OAI-SearchBot](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq),
Anthropic documents
[ClaudeBot](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler),
and Google documents
[Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended).

A giant copied list of bot names is hard to check and becomes stale quickly.
Small, chosen rules are easier to understand and maintain.

## Detecting bots in Node

If your application needs to recognize likely bots, use a maintained package
instead of pasting a fixed list into your code. The
[`isbot` package](https://www.npmjs.com/package/isbot) checks a request's user
agent:

```shell
$ pnpm add isbot
```

```ts
import { isBot } from 'isbot'

const userAgent = request.headers.get('user-agent')

if (isBot(userAgent)) {
  // Log the request, limit its rate, or skip expensive work
}
```

This is detection, not proof. A bot can send a normal browser's user agent, and
`isbot` also finds useful bots such as search crawlers and link preview bots. Do
not block every match without checking what your site needs.

## Actually blocking unwanted bots

To enforce a block, handle the request before serving the page. You can do that
in a CDN, firewall, reverse proxy, or application. Useful controls include:

- rate limits for clients making too many requests
- rules based on request patterns, known bot details, or verified crawler data
- challenges for suspicious traffic
- login checks for content that is not public

If your site uses Cloudflare, its
[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) shows
which known AI crawlers visit your site and lets you allow or block them.
Cloudflare also has a maintained
[Block AI bots setting](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/).
Those products enforce rules at Cloudflare's edge. Cloudflare's managed
`robots.txt` feature only publishes your preference, so it has the same limits
as a file you write yourself.

Start by looking at your request logs. Block or slow the traffic that causes a
real problem, then check that search engines, uptime checks, feeds, and link
previews still work. Bot rules need regular review because both crawlers and
your site's needs change.