🤖

Robots.txt Generator

100% Client-Side

Generate search-engine-compliant robots.txt files with custom user-agents, allow/disallow paths, crawl delays, and XML sitemaps.

User-agent

Disallow Paths (one per line)

Allow Paths (one per line)

Sitemap URL

Crawl-delay (seconds, optional)

Generated robots.txt

How to Use Robots.txt Generator

Specify Crawlers

Select standard User-agent (* for all, Googlebot, Bingbot).

Set Rules

Enter disallow paths (e.g., /admin/, /private/) and allow paths.

Export File

Copy the directives or download directly as robots.txt for your web root.

✓ Standard Syntax

Generates RFC 9309 compliant User-agent, Allow, Disallow, and Sitemap rules.

✓ Crawl Delay Support

Optionally specifies delay seconds for high-traffic or API-heavy servers.

✓ One-Click Presets

Quick templates for Allow All, Disallow Private Folders, or Block All.

🔒 Complete Client-Side Isolation

This tool runs entirely in your local browser runtime. No logs, analytics tracking, or remote server transmissions occur.

Generate complete, search-engine-friendly HTML meta tags, Open Graph tags, and Twitter card tags with live character counters.

Preview how your web page and social metadata look when shared on Facebook, Twitter/X, and Google search results.

Decompose any web URL into its standard protocol, hostname, port, pathname, hash, and structured query parameter key-value pairs.