All about robots.txt

What is robots.txt ?

robots.txt is a small plain-text file that sits at the root of a website and gives instructions to search engine crawlers (and other well-behaved bots) about which parts of the site they’re allowed to crawl.

It lives at a fixed, predictable location:

https://www.example.com/robots.txt

How it works ?

When a crawler like Googlebot visits your site, one of the first things it does is check robots.txt to see if there are any crawling rules it should follow.

A basic one looks like this:

“`
User-agent: *
Disallow: /admin/
Allow: /

Sitemap: https://www.example.com/sitemap.xml
“`

• User-agent: * — applies to all crawlers
• Disallow: /admin/ — “please don’t crawl anything under /admin/”
• Allow: / — everything else is fair game
• Sitemap: — points crawlers to your XML sitemap

List of User-agent

There’s no fixed, closed list of “all possible” user-agents for robots.txt — any crawler can identify itself with any token, and site owners can write rules for wildcards or fictional names. But here’s a practical breakdown:

Special/wildcard tokens

| Token | Meaning |
|—|—|
| User-agent: * | Applies to all crawlers that don’t have a more specific rule |

Common real-world crawler user-agents

Search engines
– Googlebot (general)
– Googlebot-Image, Googlebot-News, Googlebot-Video
– AdsBot-Google, AdsBot-Google-Mobile
– Mediapartners-Google (AdSense)
– Bingbot
– Slurp (Yahoo)
– DuckDuckBot
– Baiduspider
– YandexBot
– Sogou
– Exabot
– facebookexternalhit
– Twitterbot
– LinkedInBot
– WhatsApp
– Applebot
– SeznamBot

SEO/marketing crawlers
– AhrefsBot
– SemrushBot
– MJ12bot (Majestic)
– DotBot (Moz)
– BLEXBot
– rogerbot (Moz)

AI/LLM crawlers (increasingly common to block/allow explicitly)
– GPTBot (OpenAI)
– ChatGPT-User
– OAI-SearchBot
– ClaudeBot
– Claude-Web
– anthropic-ai
– CCBot (Common Crawl)
– Google-Extended
– Bytespider (ByteDance)
– PerplexityBot
– Amazonbot
– Applebot-Extended

Archival
– ia_archiver (Internet Archive Wayback Machine)

Important notes

1. Any string is valid — a bot’s operator chooses its own user-agent name; robots.txt has no enforcement mechanism, it’s purely a voluntary convention.
2. Case-insensitive matching is typical, and matching is usually by substring/prefix.
3. More specific rules override * — if you define both User-agent: * and User-agent: Googlebot, Googlebot follows its own block, not the wildcard one.
4. Malicious/scraper bots often ignore robots.txt entirely or spoof common browser user-agents, so it’s not a security control — just a crawling-etiquette signal.

Want me to generate a sample robots.txt (e.g., blocking AI scrapers but allowing search engines)?

The key thing to understand about it

robots.txt is a crawling instruction, not an indexing instruction, and it’s voluntary — it relies on the crawler choosing to respect it.

• Reputable bots (Google, Bing, etc.) honor it.
• Malicious bots and scrapers often ignore it entirely — so it’s not a security or access-control mechanism.
• Blocking a URL in robots.txt stops crawling, but if other sites link to that URL, Google can still show it in search results (just without a proper description, since it never crawled the content). To truly keep something out of search results, you need a noindex tag instead — which actually requires the page to be crawlable so Google can see that tag.

Why it exists ?

It’s essentially a traffic-control tool for your site’s relationship with search engines — helping crawlers spend their time on pages that matter, keeping them out of admin areas, internal search results, or duplicate content, and pointing them to your sitemap.

It’s a small file, but a wrong Disallow rule (like accidentally blocking / or your CSS/JS folders) can quietly tank your entire site’s visibility — so it’s worth getting right and checking periodically in Google Search Console.

What should be in robots.txt ?

A robots.txt file uses a simple set of directives to tell crawlers what they can and can’t access:

Sample content of robots.txt

“`
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sort=
Allow: /

Sitemap: https://www.example.com/sitemap.xml
“`

Core directives:

• User-agent — which crawler the rule applies to (* = all crawlers, or specific ones like Googlebot, Bingbot)
• Disallow — paths you don’t want crawled (admin panels, internal search results, cart/checkout pages, staging folders, duplicate filtered/sorted URL variants)
• Allow — explicitly permits a path, usually used to carve out an exception inside a disallowed folder (e.g., disallow /wp-admin/ but allow /wp-admin/admin-ajax.php)
• Sitemap — absolute URL to your XML sitemap(s); helps crawlers find your sitemap without it needing to be submitted separately
• Crawl-delay — supported by some crawlers (not Google) to throttle crawl speed

What typically gets disallowed: admin/login areas, internal search result pages, filter/sort URL parameters that create duplicate content, staging/dev environments, duplicate print-friendly pages, and internal scripts/cart/checkout flows that have no SEO value.

Important: Disallowing a page in robots.txt stops it from being crawled — it does not guarantee it stays out of the index. If other sites link to that disallowed URL, Google can still index the URL (with no content shown) from those external signals. To truly keep a page out of the index, use a noindex meta tag or header instead — and don’t combine noindex with Disallow, since a disallowed page can’t be crawled to see the noindex tag in the first place.

---

Case sensitivity in robots.txt

Directive names (field names) are case-insensitive. So Disallow, disallow, DISALLOW, and even DiSaLLoW are all treated the same by compliant parsers (Google, Bing, etc.). The convention/most common style is capitalized: User-agent, Disallow, Allow, Sitemap, Crawl-delay.

However, values are case-sensitive:
– User-agent: Googlebot — the bot name matching is generally case-insensitive in practice, but write it as documented to be safe.
– Paths in Disallow/Allow ARE case-sensitive. /Private/ and /private/ are different paths. This trips people up a lot — if your URLs use mixed case, your rule must match exactly.

Disallow: /Admin/ ← only blocks /Admin/, NOT /admin/

Order of directives — does it matter?

Within a record, there’s no strict required order, but the de facto/recommended convention is:

User-agent: <bot>
Disallow: <path>
Allow: <path>
Crawl-delay: <seconds>

User-agent must come first in each block — it defines which bot the following rules apply to. Everything after it (Disallow/Allow/Crawl-delay) applies until the next User-agent line starts a new block.

Rule-matching order (this is the part that actually matters)

Within a single User-agent block, specificity wins, not order of appearance. Most modern crawlers (Google, Bing) use the longest-matching-path rule, not “first match” or “last match”:

User-agent: *
Disallow: /folder/
Allow: /folder/public/

Even though Disallow comes first, /folder/public/page.html is allowed, because /folder/public/ is a longer, more specific match than /folder/.

Block-matching order (which User-agent block applies to a bot)

1. The crawler looks for a block with its exact name (e.g., Googlebot).
2. If none exists, it falls back to User-agent: *.
3. A bot only follows one block — the most specific one that matches it — not a cascade of all matching blocks.

Sitemap and other global directives

Sitemap: isn’t tied to any User-agent block — it can go anywhere in the file (commonly at the top or bottom) and applies globally.

Practical template order

“`

Specific bots first

User-agent: Googlebot
Disallow: /private/
Allow: /private/press-kit/

User-agent: GPTBot
Disallow: /

Wildcard fallback last

User-agent: *
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml
“`

Summary:
– Directive names: case-insensitive, but capitalize by convention.
– Paths/values: case-sensitive — match exactly.
– No enforced order of lines, but specific bots before *, and within a block, longest path match wins regardless of Allow/Disallow order.

---

Where should robots.txt be placed ?

Root directory only: https://www.example.com/robots.txt

It must sit at the domain root — not in a subfolder. example.com/pages/robots.txt is ignored entirely. Each subdomain needs its own: shop.example.com/robots.txt is separate from www.example.com/robots.txt. Same for HTTP vs HTTPS and www vs non-www — they’re technically treated as different origins, so if you serve on multiple, make sure the accessible version has the file.

Why you should have robots.txt ?

1. Crawl budget efficiency — Large sites especially benefit from steering crawlers away from low-value pages (admin, filters, duplicates) so Googlebot spends its limited crawl budget on pages that actually matter for rankings.
2. Keep sensitive/irrelevant areas out of casual crawling — staging environments, internal tools, cart/checkout flows.
3. Prevent duplicate content issues — blocking parameterized/sorted URL variants stops search engines from wasting resources crawling near-identical pages.
4. Point crawlers to your sitemap — a lightweight way to make sure your sitemap gets discovered.
5. Avoid server strain — on very large sites, uncontrolled bot crawling can add real load; robots.txt helps manage that.

One common misconception worth flagging: an empty or missing robots.txt doesn’t hurt your SEO — it just means “crawl everything.” You don’t strictly need one if you have nothing to restrict, but virtually every real site benefits from at least a minimal one that points to the sitemap and blocks a few low-value paths.

---

Here’s a well-tested robots.txt setup for a standard WordPress site:

“`
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /readme.html
Disallow: /license.txt
Disallow: /trackback/
Disallow: /comments/feed/
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom=
Disallow: /wp-login.php
Disallow: /xmlrpc.php

Sitemap: https://www.example.com/sitemap_index.xml
“`

Why each block is there

| Directive | Reason |
|—|—|
| Disallow: /wp-admin/ | Blocks the entire admin dashboard — no SEO value, and shouldn’t be crawlable |
| Allow: /wp-admin/admin-ajax.php | Many themes/plugins rely on this file loading correctly even for front-end AJAX calls — blocking it can break site functionality that Google evaluates during rendering |
| Disallow: /wp-includes/ | Core WP system files — irrelevant to crawl |
| Disallow: /wp-content/plugins/ | Plugin source folders — no content value, sometimes exposes unnecessary file structure |
| Disallow: /wp-content/cache/ | Cache-generated duplicate files |
| Disallow: /readme.html, /license.txt | Default WP files that reveal your WP version — minor security/info-disclosure hygiene |
| Disallow: /trackback/, /comments/feed/ | WordPress’s built-in trackback and comment-feed URLs create thin, duplicate content |
| Disallow: /?s=, /search/ | Internal search result pages — classic thin/duplicate content that shouldn’t be indexed |
| Disallow: /*?replytocom= | Comment-reply URL parameters WordPress generates per comment — creates massive duplicate URL variants of the same page |
| Disallow: /wp-login.php | Login page — no reason for it to be crawled |
| Disallow: /xmlrpc.php | Legacy WP API endpoint, frequently targeted by bots/attacks and has zero SEO value |

Important caveats

• Don’t block /wp-content/themes/‘s CSS/JS subfolders if your theme serves them from there. Google needs to render your pages to evaluate mobile-friendliness and layout — blocking stylesheets/scripts can hurt rankings, not help them. The rule above blocks the theme folder broadly for hygiene, but if your rendering breaks in Google Search Console’s URL Inspection tool, allowlist the specific CSS/JS paths your theme needs, e.g.:

Allow: /wp-content/themes/*/style.css
Allow: /wp-content/themes/*/*.js

• If you use Yoast SEO or RankMath, they auto-generate a virtual robots.txt and sitemap reference — check Yoast → Tools → File Editor or RankMath → General → Sitemap before manually creating a static file, so you don’t end up with two conflicting versions.

• Replace the Sitemap URL with your actual sitemap path — Yoast uses /sitemap_index.xml, RankMath uses /sitemap_index.xml too, but All in One SEO uses /sitemap.xml. Check yours directly.

• Don’t add noindex directives inside robots.txt — Google stopped supporting that in 2019. Use the SEO plugin’s per-page “noindex” setting instead for pages like tag archives or thin category pages.

• WooCommerce sites should additionally consider blocking /cart/, /checkout/, and /my-account/ since these are functional, user-specific pages with no search value:

Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

---

The Good, Ugly, Okay practise in SEO

White Hat SEO

White Hat SEO — techniques that follow Google’s guidelines and prioritize genuine user value, even if slower to pay off.

• Scenario: A software company writes an in-depth, well-researched guide on “how to choose a CRM,” backed by original research and expert quotes, then earns backlinks naturally because other sites find it genuinely useful and cite it.
• Scenario: An e-commerce site improves page load speed, adds proper alt text to images, and restructures navigation for better usability — all things that help both users and rankings.
• Scenario: A local bakery asks satisfied customers to leave honest Google reviews and builds genuine local citations (Yelp, local directories) to strengthen local search presence.

Black Hat SEO :

Black Hat SEO — techniques that directly violate Google’s guidelines to manipulate rankings, often for quick gains, with high risk of penalties or de-indexing.

• Scenario: A site stuffs invisible white text on a white background packed with keywords purely for crawlers, invisible to human visitors.
• Scenario: A private blog network (PBN) — a group of low-quality sites secretly owned by one person or agency — exists solely to link back to a “money site” and inflate its authority artificially.
• Scenario: Cloaking — showing search engine crawlers one version of a page (keyword-rich, optimized) while showing human visitors a completely different page.
• Scenario: Buying thousands of backlinks in bulk from link farms to fake authority signals.
• Real-world precedent: JCPenney was caught in 2011 running a massive paid-link scheme to dominate rankings for terms like “dresses” and “area rugs” — once exposed by the New York Times, Google manually penalized the site and its rankings collapsed almost overnight.

Grey Hat SEO

Grey Hat SEO — tactics that sit in an ambiguous zone: not explicitly banned, but pushing against the spirit of the guidelines, carrying some risk without being outright forbidden.

• Scenario: Buying an expired domain (one that previously had strong backlinks and authority) purely to redirect its “link juice” to a new site — not explicitly against the rules but frowned upon and increasingly detected by Google.
• Scenario: Aggressive guest posting purely for backlinks (rather than genuine audience value), where content is thin but technically not spam.
• Scenario: Using AI to mass-produce content at scale, lightly edited by a human, to fill topical gaps quickly — not clearly disallowed but risks tripping the Helpful Content System if quality is thin.
• Scenario: Reciprocal link exchanges between a small number of relevant, related sites (“I’ll link to you if you link to me”) — normal in moderation, but abusable at scale and something Google’s guidelines specifically warn can be seen as a link scheme if done excessively.

The practical line: if a tactic would embarrass you if Google’s webspam team personally reviewed your site, it’s black hat. If it’s defensible but you’d rather not have to defend it, it’s grey hat. If you’d be comfortable explaining exactly what you did and why, it’s white hat.

---