Free Robots txt Generator

Set crawl rules for any search engine bot — allow, block, or restrict specific paths and add your sitemap URL in one click.

Global Settings
Path Rules (Allow/Disallow)
No rules added yet.
Specific Bot Rules
Generated File robots.txt
User-agent: * Disallow:

Related Tools

Source Code Viewer
Try tool →
Favicon Generator
Try tool →
Domain Age Checker
Try tool →
Google SERP Snippet Preview
Try tool →
Keyword Density Checker
Try tool →
CSS / Js Minifier
Try tool →
Website Page Speed Test
Try tool →
Color Picker
Try tool →
Bulk URL Opener
Try tool →

What Is a Robots.txt File?

A robots.txt file sits at the root of your website (yourdomain.com/robots.txt) and tells search engine crawlers which pages they can and cannot access. It’s one of the first files Googlebot and Bingbot check when they visit your site.

A missing or misconfigured robots.txt won’t automatically hurt your rankings — but a wrong one can accidentally block your entire site from Google. This tool generates correctly structured robots.txt syntax so you don’t have to write it manually.

What You Can Control

Allow or block all bots — The default User-agent: * rule applies to every crawler. Set it to allow all pages, disallow all, or create a custom combination.

Target specific bots — Set separate rules for Googlebot, Google Image Bot, Bingbot, or any custom crawler. Useful for blocking AI training bots while keeping search engines.

Block specific paths — Prevent crawlers from accessing admin areas, login pages, duplicate content, or staging directories. Example: Disallow: /wp-admin/

Add your sitemap URL — Include your XML sitemap location so search engines can find and process it faster. Example: Sitemap: https://yourdomain.com/sitemap.xml

Set crawl delay — Ask bots to wait between requests, which reduces server load on smaller hosting plans.

How to Add robots.txt to Your Website

Once you’ve generated the file, upload it to your website’s root directory — the same folder where your homepage lives. For WordPress sites, you can paste the content directly into Rank Math or Yoast SEO under the robots.txt editor, or upload the file via FTP/cPanel.

After uploading, verify it’s accessible by visiting yourdomain.com/robots.txt in a browser. You can also test individual URLs using Google Search Console’s robots.txt tester to confirm your rules are working as expected before any crawl happens.

Common Pages to Block from Crawling

Most websites benefit from blocking a few standard paths. Admin and login pages have no SEO value and waste crawl budget. Search result pages (like /?s=query on WordPress) create near-duplicate content that Google doesn’t need to index. Staging or development directories should always be blocked to prevent test content from appearing in search results. If your site has a print-friendly version of pages or URL parameters that generate duplicate content, adding those to your disallow rules keeps your crawl budget focused on pages that actually matter for rankings.

Frequently Asked Questions (FAQ)

Disallowing a URL in robots.txt prevents Googlebot from crawling it, but doesn’t guarantee it won’t appear in search results. If other sites link to a blocked page, Google may still index it from those links without seeing the content. To fully prevent indexing, use a noindex meta tag on the page itself along with the robots.txt rule.

Search engines will crawl your entire site by default. That’s fine for most websites. The main risk is crawl budget waste — Google spending time on low-value pages like tag archives, duplicate URLs, or admin pages instead of your important content. A robots.txt file helps direct crawlers to what matters.

Yes, though compliance isn’t guaranteed. Reputable AI crawlers like GPTBot (OpenAI) and CCBot (Common Crawl) do respect robots.txt. Add a custom bot rule in this tool and set it to Disallow to block them. However, less reputable scrapers may ignore robots.txt entirely.

WordPress already blocks /wp-admin/ from indexing by default, but explicitly adding Disallow: /wp-admin/ in your robots.txt is still good practice. Also consider disabling /?s= (search results pages) and /wp-login.php to avoid wasting crawl budget on pages with no SEO value.

It must be in the root directory — yourdomain.com/robots.txt. A file placed in a subdirectory like yourdomain.com/blog/robots.txt won’t work. WordPress users can edit it directly in Rank Math (SEO → General → Edit robots.txt) or Yoast SEO (Tools → File Editor) without FTP access.