A robots.txt file asks well-behaved crawlers which parts of your site they may visit. It does not stop bad bots, hide private pages, or protect your site.

Think of it as a public contract. Crawlers can follow it, ignore it, or lie about who they are.

tl;dr; Use robots.txt to state your preference, and use server or CDN controls when you need that preference enforced.

What robots.txt can and cannot do

Put the file at the root of your site:

https://example.com/robots.txt

The file contains rules for one or more crawlers. This example lets crawlers visit most of the site, but asks them to stay out of two paths:

User-agent: *
Disallow: /preview/
Disallow: /search
 
Sitemap: https://example.com/sitemap.xml

User-agent: * means any crawler that does not have its own group. Disallow contains a path the crawler should not fetch. An empty Disallow allows everything, so you usually do not need an Allow: / rule.

The current standard is RFC 9309. It says plainly that these rules are not access control. The file is public, so listing a path can even draw attention to it.

If a page is private, put it behind a login or another real access check. Do not rely on robots.txt. A tool such as curl does not need to read the file at all:

$ curl https://example.com/preview/secret-page

robots.txt controls crawling, not search indexing. A search engine may still show the URL of a blocked page if it finds a link to it elsewhere. To keep a public page out of search results, use a noindex meta tag or an X-Robots-Tag header and allow the search crawler to read it. For private content, require a login. Google has a good guide to robots.txt and indexing.

Blocking one crawler

To ask one crawler to stay out of the whole site, name it and disallow the root path:

User-agent: ExampleBot
Disallow: /

To ask all crawlers to stay out:

User-agent: *
Disallow: /

That second example can remove your content from search and other useful services. Use it only when that is what you want. It still does nothing against a crawler that ignores the file.

AI crawlers

AI companies run different crawlers for different jobs. One may collect pages for model training while another powers search results or fetches a page when a person asks a question. Blocking every AI-related crawler may also stop your pages from appearing with links in AI search.

If you want to allow search but opt out of some training crawlers, a file could look like this:

# OpenAI model training
User-agent: GPTBot
Disallow: /
 
# Anthropic model training
User-agent: ClaudeBot
Disallow: /
 
# Use of Google-crawled content by Gemini
User-agent: Google-Extended
Disallow: /
 
# Keep OpenAI search discovery enabled
User-agent: OAI-SearchBot
Allow: /

These names and their uses can change. Check the crawler owner's current docs before copying a rule. OpenAI explains the difference between GPTBot and OAI-SearchBot, Anthropic documents ClaudeBot, and Google documents Google-Extended.

A giant copied list of bot names is hard to check and becomes stale quickly. Small, chosen rules are easier to understand and maintain.

Detecting bots in Node

If your application needs to recognize likely bots, use a maintained package instead of pasting a fixed list into your code. The isbot package checks a request's user agent:

$ pnpm add isbot

import { isBot } from 'isbot'
 
const userAgent = request.headers.get('user-agent')
 
if (isBot(userAgent)) {
  // Log the request, limit its rate, or skip expensive work
}

This is detection, not proof. A bot can send a normal browser's user agent, and isbot also finds useful bots such as search crawlers and link preview bots. Do not block every match without checking what your site needs.

Actually blocking unwanted bots

To enforce a block, handle the request before serving the page. You can do that in a CDN, firewall, reverse proxy, or application. Useful controls include:

• rate limits for clients making too many requests
• rules based on request patterns, known bot details, or verified crawler data
• challenges for suspicious traffic
• login checks for content that is not public

If your site uses Cloudflare, its AI Crawl Control shows which known AI crawlers visit your site and lets you allow or block them. Cloudflare also has a maintained Block AI bots setting. Those products enforce rules at Cloudflare's edge. Cloudflare's managed robots.txt feature only publishes your preference, so it has the same limits as a file you write yourself.

Start by looking at your request logs. Block or slow the traffic that causes a real problem, then check that search engines, uptime checks, feeds, and link previews still work. Bot rules need regular review because both crawlers and your site's needs change.