Robots.txt is one of the oldest files on the web, yet it keeps finding new relevance. With AI crawlers multiplying and more websites serving content in several languages, small mistakes in this plain text file can block the wrong pages or fail to block the right ones. Google recently shared fresh guidance that clears up several common misunderstandings. This blog outlines what that guidance says, why percent-encoding matters for international URLs, how robots.txt applies to AI crawlers, and the practical checks every site owner should run on their own file.
Key Takeaways
- Well-behaved crawlers, including Googlebot, use robots.txt to limit crawling.
- txt files should follow the RFC 9309 internet standard.
- Non-English characters and emojis in rule paths should be percent-encoded.
- AI crawler rules only work when they use the correct user-agent token.
- txt is a crawling preference, not a security tool or usage control.
What Google's Latest Guidance Covers
In a recent Search Central video, Google walked through how robots.txt really works and where site owners tend to go wrong. The guidance centers on three points: well-behaved crawlers, including Googlebot, use robots.txt to limit crawling; files should follow the official internet standard RFC 9309; and international characters in rule paths should be percent-encoded so every crawler reads them the same way.
Google also clarified details that regularly trip people up. Parsers read only a small set of fields, mainly user-agent, allow, disallow, and sitemap, and quietly ignore everything else. Rules that look perfectly clear to a human reader can still fail if they break the standard's syntax. The overall message is simple: write robots.txt for machines that follow strict rules, not for people skimming the file. Treated carelessly, robots.txt becomes an SEO landmine rather than a useful control.
Robots.txt Controls Crawling, Not Everything Else
One of the most persistent misunderstandings is that robots.txt hides pages from search results. It does not. A disallow rule asks compliant crawlers not to request a URL, but if other sites link to that URL, it can still appear in results, usually without a description. Keeping a page out of search results requires a noindex directive on a page that crawlers are actually allowed to access and read.
This distinction matters because the two signals can cancel each other out. If a page is blocked in robots.txt, crawlers never see its noindex tag. That gap comes down to the difference between crawling and indexing, and each goal needs its own tool. Google also reminded site owners that robots.txt is not a security mechanism, since it only asks crawlers to stay away and cannot stop anyone who ignores that request.
International URLs and the Case for Percent-Encoding
Websites serving content in languages other than English often have URL paths containing accented letters, non-Latin scripts, or even emojis. Google's advice is to write these paths in robots.txt using percent-encoding, which converts each character into a standard sequence of codes. For example, a path containing the letter "é" would use its encoded form "%C3%A9" instead of the raw character, so the rule is read consistently.
Google treats raw and encoded versions of the same path as equivalent, but other crawlers may not. Some could misread raw characters, causing rules to fail without any warning. Encoding removes that uncertainty and keeps behavior consistent across every compliant crawler. The guidance also notes that user-agent tokens cannot include emojis or non-ASCII characters, so crawler names should always appear exactly as their operators document them.
A Simple Encoding Example
- Raw path: /café/menu/ becomes /caf%C3%A9/menu/
- Raw path: /über-uns/ becomes /%C3%BCber-uns/
- Use the encoded version in every Allow and Disallow line.
Read More: URL Parameter SEO: Fix Duplicate Content, Crawling & Indexing Issues
Robots.txt sets crawling preferences, while other tools handle indexing, security, and usage.
What the Guidance Means for AI Crawlers
AI crawlers follow the same basic logic as search crawlers. A rule only applies when it names the crawler's recognized user-agent token, such as GPTBot, ClaudeBot, or Google-Extended. Misspelled names, invented tokens, or vague comments in the file do nothing at all. Since many AI companies now run several bots for different purposes, including model training, search, and user-requested fetching, site owners need to read each operator's documentation carefully. Getting these names exactly right is the difference between a rule that works and one that silently does nothing.
Compliance also varies from bot to bot. Major crawlers generally respect robots.txt, but some user-triggered fetchers may not follow it, and less reputable bots may ignore it entirely. For stricter control, robots.txt can be paired with server or firewall rules. Choosing between robots.txt and server-level controls for blocking AI crawlers depends on your content and your goals for visibility in AI tools.
Access Is Not the Same as Usage
Google's guidance draws a clear line between access and use. Under RFC 9309, robots.txt tells crawlers which URLs they may request. It says nothing about what happens to content after it has been fetched, including whether it is used to train AI models. For Google specifically, disallowing the Google-Extended token signals that content should not be used to train its generative AI models, while regular Search crawling continues as normal.
Work is underway on a standard way to express usage preferences inside robots.txt through a proposal at the Internet Engineering Task Force, but it has not become an official standard yet. Until then, treat crawling rules and AI usage decisions as separate policies. Complementary approaches are emerging as well, such as the llms.txt file, which some site owners use to guide how their content is presented to AI systems.
Special Considerations for Multilingual and Multi-Region Sites
International websites often organize content in different ways, using country-code domains, subdomains, or language folders. Each structure affects how robots.txt works, because the file only applies to the exact host where it lives. A rule on the main domain does not cover a separate subdomain or country-specific domain, so each of those needs its own robots.txt file with rules written for its own URL paths and language characters.
Consistency across these files matters. If one regional site blocks a resource that the others allow, crawlers may render pages differently depending on the region, leading to uneven visibility across markets. Keeping a shared template for robots.txt rules, then adjusting only what each region genuinely needs, reduces the risk of accidental differences. Reviewing every regional file together, rather than one at a time, also makes gaps much easier to spot.
Read More: Multilingual SEO: How to Rank Globally with Regional Strategy
A Practical Review Checklist for Your robots.txt File
A quick review catches most problems before they cause damage. Confirm the file sits at the root of the domain and returns a successful response. Check that each rule uses supported fields and correct syntax, that important pages and resources such as CSS and JavaScript are not accidentally blocked, and that sitemap locations are listed. For international sites, confirm every non-English path has been percent-encoded properly. What those files should contain is covered in the guide to XML sitemaps.
Then turn to crawler-specific rules. Verify that every AI user-agent token matches the operator's current documentation, decide which bots you want for visibility and which you want to limit, and review the result in the robots.txt report in Google Search Console. Revisit the file whenever you launch new sections, add languages, or change your stance on AI access, because one forgotten rule can quietly undo months of good work. A short note of each change and its date makes future troubleshooting much easier.
Keeping Your Crawl Rules Clear and Current
Google's latest guidance is a reminder that robots.txt works best when it is precise, standards-based, and used for the right job. It limits crawling by compliant bots, but it does not prevent indexing, secure private content, or control how AI systems use data. Percent-encoding international paths, naming AI crawlers correctly, and reviewing the file regularly keep it working as intended across every language and every crawler.
At The Ocean Marketing, we help businesses keep their technical foundations clean so search engines and AI tools can reach the right content. Our SEO team reviews robots.txt files, crawl settings, and indexing signals as part of a complete technical strategy, and a free SEO audit can uncover crawl issues you may not know exist. Contact us to make sure your site gives crawlers the right instructions.
Frequently Asked Questions
Does robots.txt stop pages from appearing in Google?
No. It limits crawling, but a blocked URL can still appear in search results if other pages link to it. A noindex directive on a crawlable page is the right way to keep it out of results.
Should I percent-encode URLs in robots.txt?
Yes, for paths that contain non-English characters or emojis. Google reads raw and encoded paths the same way, but encoding keeps rules consistent for other crawlers too.
Can robots.txt stop AI companies from training on my content?
Robots.txt controls which URLs compliant crawlers may access, not how content is used afterward. Blocking specific training crawlers by their user-agent tokens limits their access, but usage preferences are handled separately.
