Robots.txt
Robots.txt is a file at the root of a domain that tells crawlers which paths they may request.
In detail
It controls crawling, not indexing. A blocked page can still appear in results without a useful description, and a page you want removed needs a noindex directive instead.
A single stray "Disallow: /" can hide an entire site.
Example
Blocking internal search-result paths so crawl budget goes to real content.
Why it matters
It is one line of text with site-wide consequences, and it is easy to deploy wrongly.
Common mistakes
- Shipping a staging robots.txt to production.
- Blocking CSS or JS needed to render the page.
Related questions
Related articles
- SEO Basics: 10 Things Every Business Website Should Get Right
A practical checklist covering the fundamentals, with how to check each one and what a failure looks like.
Related guides
- The Complete SEO GuideGuide
A cornerstone guide to organic search: technical foundation, on-page work, content, internal linking, authority and measurement — in the order that pays off.
Related terms
- XML Sitemap
An XML sitemap is a file listing the URLs you want search engines to know about, used as a discovery source.
- Indexability
Indexability is whether a page is eligible to be stored in a search engine's index and shown in results.
- Technical SEO
Technical SEO covers everything that determines whether search engines can reach, render, index and trust your pages.
Related questions
- What does Hamatra actually check?
Hamatra crawls the site the way a search engine would and reports on technical SEO, content, on-page metadata, site structure, search and AI readiness, indexability and internal linking, plus broken links found during the same crawl.