Find valuable content accidentally blocked by robots.txt.

Robots.txt Audit

Robots.txt controls which pages search engines are allowed to crawl. A misconfigured robots.txt can accidentally block important content — entire sections of your site may be invisible to Google without you knowing. Common mistakes include overly broad Disallow rules, blocking CSS/JS files needed for rendering, and missing Sitemap directives. EchoBat analyzes your robots.txt and cross-references it against crawled content to find blocked pages that should be accessible.

How It Works

EchoBat fetches and parses robots.txt during the Discovery phase. During the crawl, blocked URLs are recorded and categorized. The Robots Audit lens cross-references blocked URLs against the site's link graph and content — pages with inbound links or significant content that are blocked are flagged as potentially valuable blocked content.

Proof Returned in the Report

Every Robots.txt Audit finding is tied to crawl evidence: affected URLs, the source signal, severity, score impact, and the next action exposed in the portal, CLI JSON, and MCP tools.

Sample Evidence Fields

  • Missing Robots.txt: Detects if no robots.txt file exists, which means search engines have no crawl guidance.
  • Blocked Valuable Content: Identifies pages blocked by robots.txt that have inbound links, content, or traffic value.
  • Missing Sitemap Directive: Checks if robots.txt includes a Sitemap: directive pointing to your XML sitemap.

Why It Matters

  • Find accidentally blocked content before it disappears from search results
  • Validate robots.txt directives against actual site content
  • Ensure sitemap is referenced in robots.txt for discovery
  • Catch overly broad rules that block too much content

Related pages