robots.txt Generator
Build a clean robots.txt file for common crawl control tasks. Add Disallow and Allow rules, include a sitemap, keep optional lines separate, and export a copy-ready result in seconds.
Start with a preset
Quick setupGenerate your robots.txt file
Enter rules below, then copy or open the generated output.
One path per line. Use / to block the whole site.
Useful when a blocked folder still needs specific files accessible.
Optional and not used by Google. Keep it separate from sitemap logic.
Paste extra groups or directives exactly as needed.
Generated output
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
How to generate a robots.txt file with this tool
Set the bot group
Leave User-agent as * for a broad rule set, or target a specific bot when needed.
Add paths carefully
List each Disallow or Allow rule on its own line. Keep folders and files precise.
Include your sitemap
Add the sitemap URL so crawlers can find your important URLs faster.
Copy and review
Generate the file, review the output, then copy it into your site root after testing.
What robots.txt does well
robots.txt is useful for crawl guidance, keeping bots away from low-value sections, and sharing sitemap locations. It is best for path-level rules and lightweight crawl management.
What robots.txt does not replace
It does not replace authentication, proper server protection, or page-level index directives. If you need a page to stay out of search, combine the right technical approach instead of relying on one file alone.
robots.txt Generator FAQ
What is a robots.txt file?
A robots.txt file lives at the site root and gives crawlers path-based crawl instructions such as Disallow, Allow, and sitemap location.
What does this robots.txt Generator do?
It turns your User-agent, Disallow, Allow, sitemap, and optional custom lines into a clean output you can review, copy, and deploy.
Should every site have a robots.txt file?
Not every site needs a complex one, but many sites benefit from at least a simple file with a sitemap line and carefully chosen crawl rules.
Does robots.txt remove pages from Google?
No. It is mainly for crawl control. Blocking crawl is not the same as guaranteed removal from search results.
What does Disallow: / mean?
It tells the targeted crawler group not to crawl the whole site. This is common on staging sites and dangerous on live sites if left in place.
When should I use Allow rules?
Use Allow when a broader blocked folder still needs certain files or endpoints crawled, such as admin-ajax on some WordPress setups.
Should I add my sitemap to robots.txt?
Yes, in many cases that is a good idea because it helps search engines find the sitemap location easily.
Can I use this tool for WordPress robots.txt rules?
Yes. The preset includes a common WordPress-style starting point with blocked admin paths and an allowed admin-ajax endpoint.
What is the Host line field for?
It is optional and not part of Google’s main usage. Keep it only when you intentionally need it for other crawler ecosystems.
Where do I place the generated file?
Place robots.txt in the site root so it is available at yourdomain.com/robots.txt.
After robots.txt, review your canonical and noindex setup
Crawl control is only one part of technical SEO. Move next into page-level signals and indexing decisions.