Sitemap and robots.txt: what they are and how to set them up

A robot holding a map walks toward the gate of a website drawn as a small town of pages, next to a sign with a check mark and a cross

Google reaches your pages in two ways: by following links and by reading lists. The sitemap is that list: it tells Google which pages exist. robots.txt is the gate: it tells Google which parts of the site it may enter.

They're two small files, and your platform usually creates them for you. But when either one is wrong, it can keep important pages out of Google without anyone noticing.

What a sitemap is

A tree diagram with the home page at the top and its sections and pages below, with a magnifying glass over it

An XML sitemap is a file listing the pages you want Google to find. It usually lives at yourdomain.com/sitemap.xml or, on WordPress with an SEO plugin, at yourdomain.com/sitemap_index.xml.

Each page appears with its address and sometimes its last-modified date:

<url>
  <loc>https://yourdomain.com/services/</loc>
  <lastmod>2026-11-10</lastmod>
</url>

What matters:

  • Only pages you want on Google: returning 200, not redirected, not noindexed, with a canonical pointing to themselves.
  • Final addresses: https, with or without www to match your site, the same ones your menu links to.
  • A real modified date. Google uses it when it's reliable. If every page claims it changed today, Google stops trusting it.
  • Priority and change frequency (priority and changefreq) do nothing: Google ignores them.
  • Limits: 50,000 URLs or 50 MB per file. Large sites use an index file that groups several sitemaps.

A sitemap doesn't force Google to index anything. It's a hint that helps Google discover pages, especially new ones and ones with few internal links.

What robots.txt is

A doorman robot lets a small robot through an open door and points another one away from a locked door

robots.txt is a text file at the root of your site (yourdomain.com/robots.txt) with instructions for crawlers. A typical one looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap_index.xml
  • User-agent says which crawler the rule applies to (* means all).
  • Disallow blocks a folder or address.
  • Allow makes an exception inside a blocked area.
  • Sitemap says where the sitemap is, so any search engine can find it.

The most misunderstood part: robots.txt blocks crawling, not indexing. If a blocked page has links pointing to it, Google can still show it, just without a description. To keep a page out of Google, use noindex and don't block it in robots.txt: if you block it, Google never sees the noindex.

How to check them in five minutes

  1. Open yourdomain.com/robots.txt in your browser. Look for a lone Disallow: / line: it blocks the whole site.
  2. Open the sitemap and check the addresses are right (https, www or not, no staging pages).
  3. In Google Search Console, go to Sitemaps, paste the sitemap address and submit it. Check the status says "Success" and how many pages it found.
  4. Under Settings → robots.txt report, check that Google read it without errors.
  5. Do the same in Bing Webmaster Tools: Bing also feeds some AI assistants.

robots.txt and AI crawlers

robots.txt is also where you decide whether AI crawlers can read your site (for example OpenAI's GPTBot or Google's Google-Extended). Blocking them keeps them from using your content, but it can also mean your business shows up less in those assistants' answers. For most businesses that want customers, letting them in makes sense. We explain it in SEO and AI search.

Common mistakes

  • A leftover Disallow: / after launching a new site. It's the costliest mistake: the whole site drops out of Google.
  • Blocking CSS or JavaScript. Google needs those files to see the page the way a visitor does.
  • Blocking a noindexed page in robots.txt. Google can't read the noindex.
  • A sitemap full of redirected, broken or noindexed pages. It makes Google trust the whole sitemap less.
  • A stale sitemap that doesn't update when you publish new pages.
  • Two different sitemaps, say one from the theme and one from a plugin, with different addresses.

If Google isn't showing some of your pages, also see why Google isn't indexing my pages.

Frequently asked questions

Does a small site need a sitemap?

It isn't required: if every page is linked from the menu, Google finds them anyway. But it costs nothing and helps new pages get discovered faster.

Do I have to resubmit the sitemap every time I publish?

No. Submit it once in Search Console. Google rereads it on its own.

Can I hide private pages with robots.txt?

No. robots.txt is public and anyone can read it. Use a password for private pages, and noindex for pages that shouldn't appear on Google.

How do I know if Google has trouble with my robots.txt?

Search Console flags it in the robots.txt report and in Page indexing, under the reason "Blocked by robots.txt".

Want us to check how Google crawls your site?

In our SEO audit we review your sitemap, robots.txt, indexing and redirects, and hand you a prioritized plan. See our prices or get in touch.

Get your SEO plan

What do you need?

We reply on WhatsApp or by email. We never share your details.

Rather talk first? Book a free video call or price your project.

Keep reading