k-o.pro

Robots.txt Validator & Generator

Validate your existing robots.txt for crawl errors or generate a clean one from scratch — no syntax guesswork required.

What the validator checks

The tool fetches your robots.txt and parses every directive against the Robots Exclusion Standard. It flags:

  • Wildcard conflicts that block JS, CSS, or image files Googlebot needs to render pages
  • Missing or incorrect sitemap declaration
  • Directives that accidentally block all crawlers (Disallow: / for User-agent: *)
  • Crawl-delay values that may slow indexation

Easy to break everything

A single Disallow: / line under User-agent: * blocks every crawler on your site — including Googlebot. Takes one character to fix, but you have to catch it first.

How to use it

  1. Enter your domain (e.g. example.com) — the tool fetches the file at /robots.txt
  2. Review flagged issues sorted by severity
  3. Switch to Generator mode to build a replacement from rules you define
  4. Copy the output and upload to your root directory

When to run it

Before and after any site migration. When organic traffic drops without an obvious explanation and you want to rule out a crawl access issue. When onboarding a new client whose robots.txt hasn't been reviewed in years.

Frequently asked questions

Is this robots.txt tool free?

Yes, free and no signup. It validates an existing robots.txt or generates a new one.

Does robots.txt stop indexing?

No. It stops crawling. To keep a page out of the index, use a noindex tag and do not block the page in robots.txt, or the crawler cannot see the noindex.

Where does robots.txt go?

At the root of your domain, the same level as your homepage, served at /robots.txt.

Related tools: llms.txt generator and validator, AI crawler access checker, and sitemap generator and validator.