robots.txt Generator
Build a robots.txt file for search engines and AI crawlers, test any path against it, then copy or download the file. Everything runs in your browser.
The robots.txt Generator is a free tool that builds a robots.txt file telling search engines and AI crawlers which parts of your site they may fetch. Choose a preset, adjust each bot, add blocked paths and your sitemap, then copy or download the result. A built-in test shows whether a path is allowed. It runs in your browser.
Last updated:
What is a robots.txt file?
A robots.txt file is a plain text file at the root of your website that asks crawlers which paths they may fetch. Search engines and AI crawlers read it before crawling, and the rules are defined in the Robots Exclusion Protocol (RFC 9309).
| Line | Meaning |
|---|---|
| User-agent: GPTBot | The rules below apply to this crawler. * means every other crawler. |
| Disallow: /admin/ | Do not fetch addresses that start with /admin/. |
| Allow: /admin/help/ | An exception inside a blocked path. |
| Disallow: /*.pdf$ | The * matches any text and $ marks the end of the address. |
| Sitemap: https://www.example.ae/sitemap.xml | Where to find your sitemap. |
When several rules match, the most specific one, meaning the longest, wins, and Allow wins a tie. The test box in this tool follows the same logic.
How do I create and upload a robots.txt file?
Pick a starting preset, adjust the bots and paths, then upload the file to the root of the domain it protects, so it opens at https://www.example.ae/robots.txt.
- Choose a preset: recommended, allow every bot, or block all AI bots.
- Add paths that no bot should fetch, one per line, such as /admin/.
- Add your sitemap address so crawlers can find it.
- Open the bot table and tick or untick individual crawlers.
- Copy the text or download robots.txt and upload it to your site's root folder.
- Open the live address in a browser to confirm it loads as plain text.
Each host needs its own file, so a subdomain such as shop.example.ae needs a separate robots.txt. Nothing you enter is sent to our server, and the tool never fetches your site.
Which AI crawlers should I allow or block?
Treat AI crawlers as three different jobs: training a model, building a search index, and fetching a page because a user asked an assistant. You can allow some and block others.
| Crawler | Owner | Purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page on a user's request |
| ClaudeBot | Anthropic | Model training |
| Claude-SearchBot | Anthropic | Claude search results |
| PerplexityBot | Perplexity | Perplexity search index |
| Google-Extended | Control token for Gemini training; does not affect Google Search | |
| Applebot-Extended | Apple | Control token to opt out of Apple AI training |
| Bingbot | Microsoft | Bing and Copilot answers |
| CCBot | Common Crawl | Open dataset used by many AI models |
| Bytespider | ByteDance | Training crawler |
The recommended preset allows search and user-request bots and blocks training crawlers. Blocking search crawlers can reduce the chance of your pages being used in those assistants' answers, so decide deliberately. The list is compiled from providers' public bot documentation and changes over time, so recheck it periodically.
What mistakes should I avoid?
The most damaging mistakes are blocking the whole site by accident and assuming robots.txt hides pages. Check these before you upload.
- A leftover Disallow: / from a staging site. It tells every crawler to stay out of the live site.
- Blocking CSS and JavaScript folders. Search engines may then fail to render your pages properly.
- Listing secret paths. The file is public, so a path written in it is visible to everyone.
- Using it to remove a page from Google. Blocking stops crawling, not indexing. To keep a page out of results, use a noindex tag and leave the page crawlable.
- Forgetting group rules. A crawler named in its own group ignores the * group, which is why this tool repeats your general rules for allowed bots.
Is robots.txt a way to stop bots from accessing my site?
No. robots.txt is a request, not access control. RFC 9309 states that its rules are not a form of access authorization. Well-behaved crawlers follow it; others may ignore it, and anyone can still open a URL in a browser.
For anything confidential, use a login, server rules or a firewall. To check that real crawlers can reach your pages and are not blocked by a firewall, try the AI bot access test. To describe your site to AI assistants, see the llms.txt generator.
Frequently asked questions
Where do I put the robots.txt file?
Upload it to the root of your domain so it opens at your-domain/robots.txt, for example https://www.example.ae/robots.txt. A file placed in a sub-folder is ignored. Every host, including each subdomain, needs its own file. After uploading, open the address in a browser to confirm it loads as plain text.
Does robots.txt remove a page from Google?
No. It asks crawlers not to fetch a page, but a blocked page can still appear in results if other sites link to it. To keep a page out of search results, add a noindex tag and leave the page crawlable so the tag can be read. Blocking the page and adding noindex together will not work.
Should I block AI crawlers on my business website?
It depends on your goals. Many businesses want to appear in AI search answers, so they allow search and user-request crawlers and block only training crawlers. If you are unsure, start with the recommended preset, review it with whoever owns your content, and revisit it as providers change their crawlers.
Do all bots obey robots.txt?
No. Reputable search engines and AI providers say their main crawlers follow it, but some user-triggered fetchers and bad actors may not. Because the file is public and voluntary, it cannot protect private pages. Use authentication or firewall rules when a page must not be reachable.
What is the Content-Signal line and should I add it?
Content-Signal is a newer, optional line that states preferences for search, AI answers and AI training. It is not part of the RFC 9309 standard and many crawlers ignore it, so treat it as a statement of intent rather than a block. It is switched off by default in this tool, and your Disallow rules do the real work.
Related pages
Sources
See a sample design for your own business
Browse fictional sample designs from hundreds of industries. Cannot see your type of business? Tell us what you do on WhatsApp and we will prepare a free sample design for you, with no obligation.
