SEO Reference
Robots.txt Directives Reference
Robots.txt is a small file with a large blast radius, and most of the damage I have been called in to fix came from a single misplaced slash. This page covers what Google supports today, what it ignores, and the distinction between blocking a crawl and removing a page from the index.
| Directive | Purpose | Google support |
|---|---|---|
| User-agent | Names the crawler the rules apply to | Supported |
| Disallow | Blocks crawling of a path | Supported |
| Allow | Creates an exception inside a disallowed path | Supported |
| Sitemap | Declares an XML sitemap URL | Supported |
| Crawl-delay | Requests slower crawling | Ignored by Google |
| Noindex | Attempts to deindex from robots.txt | Not supported, do not use |
| Pattern | Meaning | Example |
|---|---|---|
| Disallow: / | Blocks the entire site | Never ship this to production |
| Disallow: /admin/ | Blocks that directory | Trailing slash limits scope |
| Disallow: /*? | Blocks URLs containing a query string | Use with care, can block valid pages |
| Allow inside Disallow | Exception for one path | Longer matching rule wins |
| $ at the end | Anchors the match to the URL end | Disallow: /*.pdf$ |
Notes
Blocking a URL in robots.txt does not remove it from the index. If other pages link to it, Google can still list the URL with no description, because it is forbidden from crawling the page to see the noindex tag you added. To remove a page properly, allow the crawl and serve a noindex meta robots tag or an x-robots-tag header.
Never block CSS or JavaScript that the page needs to render. Google renders pages before evaluating them, and a blocked stylesheet or script can make a perfectly good page look broken or empty to the crawler. This was a common pattern in older configurations and still shows up in inherited sites.
The single most expensive mistake in this file is a staging Disallow: / promoted to production. Add a deployment check that fetches the live robots.txt after every release and fails the build if it disallows the root, because the traffic loss from this can take weeks to recover.
Google has changed its support for individual directives over time, including formally dropping unsupported rules. Check the current Google robots.txt specification and test with the robots.txt report in Search Console before publishing changes.
Frequently asked questions
Should I use robots.txt or noindex?
Use robots.txt to manage crawl budget on low value URL patterns, and noindex to keep a page out of search results. They solve different problems and using robots.txt for deindexing actively prevents the noindex from being seen.
Does robots.txt provide security?
No, the opposite. It is a public file and listing a sensitive path there advertises it. Protect private areas with authentication.
Where must the file live?
At the root of the host, at /robots.txt, and it applies per protocol, host and port. A robots.txt in a subdirectory does nothing.
Official sources
Platforms change specifications without notice. Check the primary documentation before a launch that depends on an exact value.
Read next
You might also like
Ranked by how closely each page overlaps with this one, using a similarity model over the whole library.
More SEO Reference
Meta Tag Length Guide
Title tag and meta description length limits, pixel widths, Open Graph tags, and what Google actually displays in results.
Schema Types for Marketers
The structured data types that matter for marketing sites, which ones earn rich results, and required properties for each.
HTTP Status Codes for SEO
What each HTTP status code means for crawling and indexing, when to use 301 versus 302, and how Google treats soft 404s.
Google Search Operators
Google search operators that still work, what each one does, and how to combine them for technical SEO audits and competitive research.
Core Web Vitals Thresholds
Core Web Vitals metrics and their good, needs improvement and poor thresholds, plus how field data is assessed and what moves each metric.
Canonical Tag Rules
How rel canonical works, correct implementation patterns, common mistakes, and when to use a canonical instead of a redirect or noindex.