Robots.txt is probably the smallest file on a website, yet one of those capable of causing the most damage when something goes wrong. A single poorly written line can make an entire site disappear from search results within hours, usually with no visible alert warning the site owner before traffic collapses.
What this file actually does
Robots.txt tells crawlers which parts of a site they can or cannot browse. It must be located at the root of the domain, at the address example.com/robots.txt, otherwise it is simply not taken into account. A point often misunderstood deserves clarifying: robots.txt controls crawling, not indexing. A page blocked in this file can still appear in search results, usually without a description, if other sites link to it. To reliably remove a page from results, the noindex tag placed directly on the page remains the right tool, not robots.txt.
The most costly mistake: an accidental full block
The most damaging scenario remains a Disallow line followed by a simple forward slash, which blocks crawling of the entire site. This case happens frequently during deployment, when the robots.txt file used on the test environment, deliberately restrictive to prevent it from being indexed, ends up copied as is onto the live site without being changed. Once this mistake is made, correcting the file is not enough for everything to instantly return to normal: pages already removed from crawling can remain visible, but degraded, for several weeks while Google recrawls the entire site.
The trap of blocking CSS and JavaScript files
Another common problem, particularly on older sites, is accidentally blocking access to folders containing stylesheets and scripts. Google today analyzes a page much like a browser, taking its actual layout into account. If it cannot load the CSS and JavaScript, it perceives a poorly formed or broken page, which can negatively influence its assessment, even though the text content remains perfectly accessible.
The 2026 novelty: distinguishing search crawlers from AI crawlers
Since the rise of conversational assistants, a new category of crawlers now explores the web, with objectives different from traditional search engines. Some, such as GPTBot or CCBot, collect content to train language models. Others, such as the retrieval bots used by certain assistants' search features, aim to cite a specific source in response to a user's question, which often remains desirable for a site's visibility. These two categories are distinguished by their user agent name, and robots.txt lets you allow or block them independently of one another, rather than applying a single rule to every bot through the asterisk. One point worth noting: Google-Extended, which feeds training for the Gemini models, is a separate bot from Googlebot, the bot used for classic indexing in Google Search. Blocking Google-Extended therefore does not affect the site's ranking in regular search results.
Good practices for a reliable file
A safe structure is to start by broadly allowing crawling, then add precise, named blocking rules rather than vague wildcards, which limits the consequences of a syntax error. Character case matters in the paths listed: blocking a folder written in lowercase does not automatically block the same address written with a capital letter. It is also recommended to always include the line pointing to the sitemap's location, a simple step frequently forgotten, and to keep the file concise, knowing that major search engines generally ignore files exceeding several hundred kilobytes in size.
Checking after every change
After any modification to the file, particularly after deployment, a quick check of its actual content online avoids discovering an accidental block several days after it started affecting traffic. Search Console offers a dedicated tool for testing whether a specific URL is blocked by the current file, which remains the most reliable way to confirm a rule produces the intended effect.
Points to verify before publishing: confirm that the distinction between Googlebot and Google-Extended remains accurate at the time of publication, and check that the user agent names mentioned for AI crawlers (GPTBot, CCBot, Google-Extended) still match the currently official designations.