---
title: "Robots.txt"
description: "A plain text file at the root of a website (e.g., example.com/robots.txt) that instructs search engine crawlers which pages or sections they are and are not allowed to crawl."
url: "https://skribra.com/glossary/robots-txt"
type: "glossary-term"
category: "Technical SEO"
updated: "2026-06-08"
---

# Robots.txt

**Definition.** A plain text file at the root of a website (e.g., example.com/robots.txt) that instructs search engine crawlers which pages or sections they are and are not allowed to crawl.

**In short.** Robots.txt is a text file that tells crawlers what to access and what to skip. Misconfigured robots.txt files are one of the most common causes of pages not appearing in search results.

## Key points

- Robots.txt controls crawl access, not indexing — blocking a URL in robots.txt doesn't guarantee it won't be indexed if it has external links pointing to it
- The 'Disallow: /' directive blocks all crawlers from all pages — a common misconfiguration that can accidentally de-index an entire site
- Different crawlers can be targeted with different rules using the User-agent directive (e.g., Googlebot, Bingbot)
- Robots.txt can specify the location of your XML sitemap to help crawlers discover your content structure

## Robots.txt Syntax and Structure

A robots.txt file consists of [user-agent blocks and directives](https://skribra.com/tools/robots-txt-generator)[1]. User-agent specifies which bot the rules apply to (* means all bots). Disallow specifies paths the bot should not crawl. Allow overrides Disallow for specific sub-paths. Sitemap specifies the location of your [XML sitemap](https://skribra.com/glossary/sitemap). Example: `User-agent: * / Disallow: /admin/ / Sitemap: https://example.com/sitemap.xml`. Google Search Console's robots.txt tester validates your file and shows how Googlebot would interpret it — always test before deploying changes to avoid accidental [Crawlability](https://skribra.com/glossary/crawlability) blocks.

## What to Block in Robots.txt

Good candidates for blocking include internal search result pages (which create infinite URL variations that waste [crawl budget](https://skribra.com/glossary/crawlability)), admin/login pages, duplicate content generated by filters and sorting parameters, and confirmation pages[1]. Blocking these preserves crawl budget for your most important pages and prevents low-value URLs from diluting your [search index](https://skribra.com/glossary/indexing). Critically, avoid blocking CSS, JavaScript, or image files — Googlebot needs to render pages fully to assess their quality and your [E-E-A-T signals](https://skribra.com/glossary/e-e-a-t).

## The Robots.txt vs. Noindex Distinction

A critical distinction: robots.txt blocks crawling, while a noindex meta tag controls [Indexing](https://skribra.com/glossary/indexing). If you block a page in robots.txt, Googlebot can't read the noindex tag either — meaning the page may still appear in [search results](https://skribra.com/glossary/serp) if other sites link to it (Google will just know less about its content)[1]. For pages you want excluded from search results, use a noindex meta tag (and don't block it in robots.txt). Use robots.txt to manage [crawl budget](https://skribra.com/glossary/crawlability), not to hide content from search results.

## Related terms

- [Crawlability](https://skribra.com/glossary/crawlability): The ability of search engine bots to access, navigate, and read the pages on your website without encountering technical barriers.
- [Indexing](https://skribra.com/glossary/indexing): The process by which a search engine stores and organizes crawled web pages in its database so they can be retrieved and displayed in search results.
- [Canonical URL](https://skribra.com/glossary/canonical-url): An HTML tag that tells search engines which version of a page is the preferred, authoritative URL when multiple URLs serve the same or very similar content.
- [XML Sitemap](https://skribra.com/glossary/sitemap): A file (typically in XML format) that lists all the important URLs on a website, helping search engines discover and crawl content more efficiently.

## Sources

1. [Google Search Central — Robots Meta Tag Specifications](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)
2. [Google Search Central — Crawling and Indexing Overview](https://developers.google.com/search/docs/crawling-indexing)

## Machine-readable index

- HTML version of this page: https://skribra.com/glossary/robots-txt
- Site index for agents: https://skribra.com/llms.txt
- Sitemap: https://skribra.com/sitemap.xml
- Public REST API: https://skribra.com/api/v1
- OpenAPI specification: https://skribra.com/openapi.json