Visual Robots.txt Generator & AI Bot Controller
Protect your content from AI training scrapers, keep search engines indexing your site, reverse-parse existing files, and test URL access in a live sandbox.
Choose a baseline policy or customize rules
Master AI Crawler Switch
Block or allow all 12 major generative AI and training scrapers simultaneously.
Crawls content to train OpenAI generative models like GPT-4 and future ChatGPT models.
Used when a ChatGPT user explicitly asks ChatGPT to read a webpage in real-time.
Scrapes web archives into open datasets widely used to train open-source and commercial LLMs.
Used by Google to train Gemini, Imagen, and Vertex AI models. Does NOT affect Google Search ranking.
High-frequency crawler used to train TikTok and ByteDance LLM systems and Doubao AI.
Crawls pages to provide conversational answers and citations in Perplexity AI search.
Used by Apple to train Apple Intelligence and foundation models across iOS/macOS.
Extracts structured knowledge graphs, articles, and entities for AI intelligence applications.
Default Rule for Search Engines (`User-agent: *`)
Allow legitimate web search engines to crawl and index your public content.
The main Google web crawler. Discovers and indexes pages for Google Search results.
Indexes images for Google Images search.
Indexes pages for Microsoft Bing, DuckDuckGo syndication, and Yahoo Search.
Crawls the web to maintain DuckDuckGo search indexes and instant answers.
Main web crawler for Yandex, the leading search engine in Eastern Europe.
Primary search crawler for Baidu in China and Chinese-language web searches.
Indexes pages for Siri suggestions, Spotlight search, and Safari search lookups.
Powers the Ahrefs backlink database and SEO site audit tools.
Audits websites and crawls backlink data for Semrush marketing tools.
Moz link intelligence crawler used to calculate Domain Authority (DA).
Directories or file types search engines should not crawl (e.g. admin panels, private files).
For example, allow `/wp-admin/admin-ajax.php` while blocking `/wp-admin/`.
Direct crawlers to your sitemap so they can discover all pages accurately.
Throttles crawler request frequency. (Bing & Yandex support this; Google ignores it).
Specifies preferred domain mirror (primarily used by Yandex).
Live Syntax & Security Audit
"Test a URL" Sandbox
Verify whether a bot is allowed or blocked on any path
Access BLOCKED by rule "DISALLOW: /" on line 12.
Everything You Need to Know About Robots.txt & AI Crawlers
Practical strategies to balance SEO visibility with creator copyright and content protection.
What is a Robots.txt File?
A robots.txt file gives instructions to automated web crawlers about which URLs on your website they are allowed or disallowed to crawl. Published as RFC 9309 in 2022, the Robots Exclusion Protocol is respected by all major ethical search engines including Google, Bing, and DuckDuckGo.
Why Block Generative AI Bots?
Traditional search crawlers crawl your site to send you human visitors through organic search results. In contrast, AI training crawlers (such as GPTBot, ClaudeBot, and CCBot) scrape your text, photography, and guides to train proprietary AI models without providing backlinks, referral traffic, or licensing compensation.
Googlebot vs. Google-Extended
Many site owners fear blocking AI might harm their Google Search presence. Fortunately, Google created Google-Extended as a distinct token. You can block Google-Extended to prevent your content from being used to train Gemini and Vertex AI without having any negative impact on your Google Search rankings.
Where to Deploy Your File
Your file must be placed at the top-level root domain: https://yourdomain.com/robots.txt.
• Cloudflare Pages / Astro: Place in public/robots.txt.
• WordPress: Upload to public_html/ via FTP or use Yoast/RankMath.
• Nginx / Apache: Save in your web root directory (e.g. /var/www/html/).
Robots.txt & AI Crawler FAQs
What is a robots.txt file and what does it do?
A robots.txt file is a plain text file placed at the root of your domain (e.g., https://yourdomain.com/robots.txt) that instructs search engine crawlers and automated bots which pages or sections of your site they are allowed or forbidden to crawl, following RFC 9309 standards.
Why should I block AI crawlers like GPTBot, ClaudeBot, and CCBot?
Generative AI bots crawl websites to scrape articles, tutorials, and creative works to train large language models (LLMs) without providing search referral traffic or attribution. Blocking them protects your proprietary content while keeping standard search engines (Google and Bing) completely open.
Does blocking Google-Extended hurt my Google Search rankings?
No. Google officially documents that Google-Extended is strictly used for training Gemini and Vertex AI models. Disallowing Google-Extended does NOT impact your rankings in Google Web Search or Google Images.
How does the "Test a URL" Sandbox work?
The sandbox implements the official RFC 9309 path-matching algorithm. It evaluates your entered path against user-agent rules, wildcard patterns (*), and end-anchors ($), adhering strictly to the longest-match priority to tell you definitively whether a bot is ALLOWED or BLOCKED.
Where should I upload the generated robots.txt file?
Upload the file to the root public folder of your web host so it is accessible at https://yourdomain.com/robots.txt. On WordPress, you can upload to your root directory or use an SEO plugin; on Cloudflare Pages, place it in the public/ folder.
Is any of my site data or robots.txt content uploaded to a server?
No. All file generation, parsing, linting, and URL path matching execute 100% locally in your browser memory. Zero data is transmitted to our servers.
Related Tools & Utilities
Explore complementary tools to streamline your document, design, and publishing workflows.
Image Resizer to 25 KB
Compress JPG, PNG, and WebP images to a 25 KB maximum or a custom file-size limit, entirely in your browser.
PDF & Document Conversion Hub
Convert, edit, compress, split, merge, watermark, and sign PDF files entirely in your browser with 100% privacy.
Compress PDF File Size
Reduce PDF file size without losing visual quality — perfect for email attachments.
Unicode to InPage Converter
Convert Unicode Urdu into legacy InPage-compatible text, or recover readable Unicode from copied InPage text.
Passport & Visa Photo Maker
Create 100% compliant US 2×2″, Schengen 35×45mm, UK, Canada & international passport photos. Tile 6 photos on a printable 4×6″ sheet for 35¢ pharmacy prints.
Create PDF Form Template
Design fillable PDF forms with interactive text fields, checkboxes, and signature fields.