What Is the Robots Meta Tag? noindex, nofollow and X-Robots-Tag Explained

The robots meta tag is a line of HTML in a page’s head that tells search engines whether they may index the page and how it can appear in search results. The X-Robots-Tag does the same job from the server, as an HTTP response header, so it also works for files that have no HTML, such as PDFs and images. Together they’re the main way to keep a page out of Google’s results or limit what Google shows from it.

This guide covers the rules Google supports, how the tag differs from robots.txt, where each one belongs on a local business site, and the mistakes that most often knock good pages out of search. It’s part of our technical SEO series. For the rest of the picture, see our SEO hub.

What the Robots Meta Tag Looks Like

The tag sits inside the <head> section of a page. A page that should stay out of search results would carry this:

<meta name="robots" content="noindex">

The name attribute says which crawlers the rule is for. robots means all crawlers. Google also reads two specific names: googlebot for regular web results and googlebot-news for news results. The content attribute holds one or more rules, separated by commas. Google’s robots meta tag documentation says both attributes are case-insensitive, so NOINDEX and noindex mean the same thing.

If a page has no robots meta tag at all, Google treats it as all, the default: no restrictions on indexing or serving. You only need the tag when you want to restrict something.

The Robots Rules Google Supports

Google’s documentation lists these rules. The ones a small business site uses most are at the top.

Rule	What it tells Google
noindex	Do not show this page, media or resource in search results.
nofollow	Do not follow the links on this page.
none	The same as noindex and nofollow together.
nosnippet	Do not show a text snippet or video preview for this page in search results.
max-snippet:[number]	Use at most this many characters as a text snippet. 0 means no snippet and -1 means no limit.
max-image-preview:[setting]	Set the largest image preview size: none, standard or large.
max-video-preview:[number]	Use at most this many seconds as a video preview.
noimageindex	Do not index the images on this page.
notranslate	Do not offer a translation of this page in search results.
indexifembedded	Allow this page’s content to be indexed when it’s embedded in another page. Used together with noindex.
unavailable_after:[date]	Do not show this page in search results after this date and time.
all	No restrictions. This is the default.

noindex

This is the rule that matters most. It keeps a page out of Google’s search results. Use it for pages that need to exist for visitors but don’t belong in search: thank-you pages after a form, internal search results, staging copies, or a login page.

nofollow

Page-level nofollow tells Google not to follow any links on the page. It’s rarely what a business site needs. If you want to qualify a single link, such as a paid or user-submitted one, use a rel attribute on that link instead. Our guide to link building explains rel="nofollow", rel="sponsored" and rel="ugc".

Snippet controls: nosnippet, max-snippet and data-nosnippet

These don’t remove a page from results. They limit the preview Google shows. nosnippet removes the text snippet entirely, and max-snippet caps its length. To hide only part of a page from snippets, Google supports a data-nosnippet attribute on span, div and section elements. Most local businesses want a helpful snippet, so use these only when you have a specific reason, like keeping a block of legal text out of the preview.

What about noarchive?

Older guides, including an earlier version of this article, recommended noarchive. Google’s documentation now says the rule is no longer used by Google Search, because the cached-page feature it controlled no longer exists. You can leave it out.

Combining rules and handling conflicts

You can list several rules in one tag, such as content="noindex, nofollow", or use more than one tag. When rules conflict, Google says the more restrictive one applies. So a page with one tag saying max-snippet:50 and another saying nosnippet gets no snippet.

The X-Robots-Tag HTTP Header

A PDF, image or video file has no <head>, so it can’t carry a meta tag. For those files, Google supports the same rules as an HTTP response header:

X-Robots-Tag: noindex

The header is set on the server, not in the page. On many WordPress hosts that means a rule in the server configuration or a plugin that can add headers. A common use for a local business is an old brochure, a price sheet or an internal form saved as a PDF that you don’t want showing up when people search your name. The header works for HTML pages too, and any rule that works in the meta tag works in the header.

Robots Meta Tag vs Robots.txt

These two get mixed up constantly, and the mix-up causes real problems. They control different things.

	robots.txt	Robots meta tag or X-Robots-Tag
Controls	Which URLs a crawler may request	Whether a page may be indexed and how it appears
Lives in	One file at the root of the site	Each page’s HTML head, or the server’s response header
Keeps a page out of results?	No. A blocked URL can still be indexed	Yes, with noindex
Main use	Managing crawler traffic	Keeping specific pages out of search and controlling snippets

Google’s introduction to robots.txt says it is not “a mechanism for keeping a web page out of Google.” Google can still index a disallowed URL if it’s linked from other places on the web. To keep a page out of results, Google points to noindex or password protection.

Don’t block a noindex page in robots.txt

This is the most important point in the whole article. Google’s guide to blocking indexing says that for noindex to be effective, the page must not be blocked by robots.txt. If robots.txt stops Google from crawling the page, Google never sees the noindex rule, and the URL can still appear in results if other pages link to it. To remove a page from search, let Google crawl it and serve noindex.

When to Use noindex on a Local Business Site

Most pages on a small business site should be indexed. Service pages, location pages, the home page, the about and contact pages and helpful articles are what you want people to find. Pages that usually don’t need to be in search include:

• Form thank-you and confirmation pages
• Internal site search results
• Staging, test or duplicate copies of pages
• Cart, checkout and account pages on sites that sell online
• Landing pages built only for paid ads, when they duplicate your main service pages
• Tag or author archives that only repeat posts listed elsewhere, if your SEO plugin creates them

One caution: noindex is not the tool for duplicate pages on your own site that you’d rather consolidate. Google’s guidance on consolidating duplicate URLs says it doesn’t recommend noindex for choosing a canonical within a site and points to rel="canonical" instead. Our guide to canonical tags walks through it. If the wrong page of yours is ranking for a search, see what to do when the wrong page ranks.

How to Set Robots Rules in WordPress

You rarely need to edit code. Most WordPress SEO plugins, including SEOPress, Yoast and Rank Math, add a robots setting to each page and post, usually a checkbox or dropdown for noindex and nofollow. They also have site-wide defaults for archives, tags and attachment pages, and those defaults are worth reviewing once.

There’s one WordPress setting worth checking right now. Under Settings, then Reading, there’s a box labeled “Discourage search engines from indexing this site.” It’s meant for sites under construction. If it’s left on after launch, it asks search engines to stay away from the entire site.

Common Robots Meta Tag Mistakes

• A noindex left on after a redesign. A staging site is often set to noindex, then copied live with the rule still in place.
• Blocking a page in robots.txt and expecting it to disappear. Google can’t read the noindex rule on a page it isn’t allowed to crawl.
• Noindexing pages you want to rank. A plugin default can noindex categories, service-area pages or a custom post type you use for locations.
• Using page-level nofollow to steer links. It stops Google following every link on the page, including links to your own service pages.
• Conflicting tags. A theme and a plugin both output robots tags, and the more restrictive one wins.
• Relying on noarchive. Google Search no longer uses it.

How to Check a Page’s Robots Rules

To see the tag, open the page, view the source and search for robots. To see the header, open your browser’s developer tools, go to the Network tab, reload, and look at the response headers for the page or file.

To see what Google saw, use the URL Inspection tool in Google Search Console. Its “Indexing allowed?” field shows whether the page disallowed indexing and why. Google notes that if robots.txt blocks the page, this field always says yes, because Google can’t see the noindex rule. If you’ve just removed a noindex tag, the same tool lets you request indexing. The Page Indexing report shows the pages where Google found a noindex rule, which is a quick way to catch a stray tag across the whole site.

Frequently Asked Questions

What is the difference between noindex and nofollow?

Noindex tells search engines not to show the page in search results. Nofollow, used in the robots meta tag, tells them not to follow the links on the page. They do different jobs and can be used alone or together.

Does robots.txt stop a page from being indexed?

No. Robots.txt controls crawling, not indexing. Google says it can still index a blocked URL if other pages link to it. Use a noindex rule, and make sure the page isn’t blocked in robots.txt so Google can read it.

What is the X-Robots-Tag?

It’s an HTTP response header that carries the same rules as the robots meta tag. Because the server sends it, it works for PDFs, images, videos and other files that can’t hold an HTML meta tag.

How long does it take for a noindex page to drop out of Google?

The page drops out after Google recrawls it and sees the rule, and the timing varies. You can use the URL Inspection tool in Search Console to check what Google last saw and request a recrawl.

Should I noindex thin or old blog posts?

Usually it’s better to improve them, merge them into a stronger page, or redirect them. Noindex keeps a page live for visitors but out of search, which suits utility pages more than articles you wrote to be found.

Get Help With Your Technical SEO

Local Blitz has worked on SEO for local businesses since 2009, from offices in San Diego, Encinitas and Indianapolis. We can review your robots tags, robots.txt, canonicals and indexing, and fix the stray rules that quietly keep pages out of search. Plans are month-to-month, there are no cookie-cutter packages, and we report on the calls, leads and rankings that follow. Book a free strategy call and we’ll tell you what we find.