What does robots.txt do?
A robots.txt file tells crawlers which pages or files they can or can’t request from your site. It’s mainly used to avoid overloading your site with requests and to keep crawlers out of areas that are no use to searchers.
What does a robots.txt file look like?
This site’s file lives at https://thewebtoolbox.com/robots.txt. Here it is:
# thewebtoolbox.com — updated 1 November 2026
User-agent: *
Disallow: /admin/
Disallow: /forgot/
Disallow: /visitors/
Disallow: /core/
Disallow: /register/add-vote
Disallow: /contact/Contact-Form
Disallow: /utilities/language-get-code
Allow: /core/css/
Allow: /core/js/
Allow: /core/fonts/
Allow: /core/images/
Sitemap: https://thewebtoolbox.com/sitemap.xmlIn plain English: “every crawler (*) may visit everything except the admin and private folders; the stylesheets, scripts, fonts and images they need to see the pages properly are allowed; and the sitemap is here.”
How do I create one?
Use any plain-text editor and save the file as PLAIN TEXT — no bold or styling of any kind.
Name it
robots.txt, all lower case.Start each group with
User-agent:(*means every crawler), then oneDisallow:orAllow:rule per line. Paths are case-sensitive.Don’t allow and disallow the same folder. If you do, the more specific (longer) rule wins.
Never block the CSS, JavaScript and images your pages need — Google renders pages like a browser and needs to see them properly.
Add the full address of your sitemap on its own line. It’s good practice and helps every search engine find it.
Upload it to the root folder of your site, so it’s at
https://your-domain.com/robots.txt. Anywhere else, it’s ignored.
Remember that each page can also carry its own instruction in the header — index, follow or noindex — as shown in the structure of a web page.
Test it
Open your robots.txt address in a browser to check it’s there. Then, in Google Search Console, go to Settings → robots.txt to see the version Google last fetched and any problems it found. You can check whether a particular page is blocked with the URL Inspection tool.
Should AI crawlers use your content?
Several companies now crawl the web to train AI models, and most of them respect robots.txt. Whether you let them is your choice — blocking them doesn’t affect your normal Google or Bing search listings. To opt out, add groups like these:
# Opt out of AI model training (does not affect normal search results)
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /New crawlers appear regularly, so check the companies’ own documentation if this matters to you.
You’ve finished the eight steps
Your site is registered, live, secure, branded and ready for search engines. Next, work through the on-page SEO checklist to give every page its best chance of being found.