How to create a robots.txt file
- Start from a preset such as Allow everything, Block AI crawlers or WordPress, or edit the rule group.
- Name the crawlers in User-agent (or pick them from the list), then add Disallow and Allow rules with the paths, and your sitemap URL.
- Press Download robots.txt and upload it to the top folder of your site, so it opens at yoursite.com/robots.txt.
How crawlers read your rules
A crawler looks for the group whose user-agent matches its own name most specifically and follows only that group, ignoring the rest. So if you have a group for Googlebot and a group for *, Googlebot obeys only its own group, not both. Within a group, Google uses the rule with the longest matching path, and when an Allow and a Disallow match equally, Allow wins. That’s how the WordPress preset blocks /wp-admin/ but still allows /wp-admin/admin-ajax.php.
Common mistakes
Blocking CSS and JavaScript files can stop Google from seeing your pages as visitors do, so leave them crawlable. A Disallow: / left over from a staging site blocks your whole site; check the file after every launch. The file must be at the root of each host: blog.example.com needs its own robots.txt. And never list secret folders here, because anyone can read the file and see what you’re hiding.