DevHive

Robots.txt Generator

Generate robots.txt for your website

Rule 1
Rule 2
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /*.json$

User-agent: Googlebot
Allow: /
Allow: /blog/

Sitemap: https://example.com/sitemap.xml

How to Use Robots.txt Generator

  1. 1

    Choose which search engine bots you want to allow or block.

  2. 2

    Add any specific URL paths you want to disallow.

  3. 3

    Set your sitemap URL if you have one.

  4. 4

    Copy the generated robots.txt content and upload it to your website root.

About Robots.txt Generator

Generate a robots.txt file for your website. Control which pages search engines can crawl. Free.

Best Use Cases

  • •Creating a robots.txt file for a new website before submitting to Google
  • •Blocking crawlers from indexing staging or admin pages
  • •Adding a sitemap URL reference to your robots.txt file
  • •Allowing Googlebot but blocking other bots from specific directories
  • •Preventing search engines from crawling duplicate content folders

Examples

Basic website

Allow all bots, disallow /admin and /private. Add your sitemap URL. Copy the output to your site root.

Block staging site

Set User-agent to * and Disallow to / to block all crawling on a staging domain.

Selective bot access

Allow Googlebot full access, block other bots from /api and /internal directories.

Common Mistakes to Avoid

  • !Accidentally blocking your entire site with Disallow: / on production
  • !Assuming robots.txt prevents pages from being indexed (it only blocks crawling)
  • !Placing the file in a subdirectory instead of the website root

Limitations

  • –Cannot enforce crawl rules; bots may ignore robots.txt
  • –Does not test whether your existing robots.txt has syntax errors
  • –Cannot upload the file to your server for you

Frequently Asked Questions

What is robots.txt?

robots.txt is a file in your website's root that tells search engine crawlers which pages not to index or crawl.