SiteVoom
FREE TECHNICAL SEO TOOL

Draft your robots.txt
and understand every rule.

Create a crawler policy for your website. This file can affect crawling, but it does not protect secrets or reliably remove pages already in search results.

All free tools ↗Meta Tag Generator ↗Launch checklist ↗

Choose a crawler policy

Should compliant crawlers be allowed to visit the website?

Your robots.txt draft

Review the policy before publishing. Upload to the exact origin root, not a subfolder. Search engines interpret crawler instructions differently.

Critical limitations

  • Do not use this file to hide confidential information.

Important: Disallowing a URL controls crawler access; it does not guarantee deindexing. Use genuine authorization for private content, and verify your site's indexability with Search Console and live tests.