k-o.pro

X-Robots-Tag: what it is and how to use it correctly

X-Robots-Tag controls indexing at the HTTP header level. Here's what it does, how it differs from robots.txt and meta robots, and real config examples.

Oleksii Khoroshun
··6 min read

You add noindex to a meta tag and Google keeps the page indexed anyway. The page is a PDF, or an image, or an API response, and none of those can hold a <meta> tag in the first place. That's the gap X-Robots-Tag closes.

X-Robots-Tag is an HTTP response header that carries the same indexing directives as the robots meta tag, but it works on any file type, not just HTML. Here's what it does, where it breaks, and how to set it up on a modern stack.

What X-Robots-Tag is

X-Robots-Tag is an HTTP header that tells search engines how to index and serve a URL. It carries the same directives as <meta name="robots">, but it lives in the server response instead of the page's HTML, so it applies to PDFs, images, JSON responses, and any other file a browser or crawler can request.

A minimal response looks like this:

typescript
1HTTP/1.1 200 OK
2X-Robots-Tag: noindex

You can combine directives in a comma-separated list, or send multiple X-Robots-Tag headers in the same response, one per crawler if you need different rules for Googlebot versus Bingbot.

X-Robots-Tag vs robots.txt vs meta robots tag

These three get mixed up constantly because they all say "don't show this page," but they control different things.

  • robots.txt controls crawling. It tells bots which URLs they're allowed to request. It says nothing about indexing, and Google formally dropped support for the unofficial noindex directive inside robots.txt in July 2019.
  • Meta robots tag controls indexing and serving for a single HTML page, set inside the page's <head>.
  • X-Robots-Tag controls indexing and serving for any URL, HTML or not, set in the HTTP response header. It also supports crawler-specific rules and site-wide patterns via server config, which the meta tag can't do.

Rule of thumb: block crawling in robots.txt when you don't want the bot spending budget on a URL at all. Use X-Robots-Tag or the meta tag when the bot should fetch the page, but you don't want it in search results.

Directives explained

Any directive valid in a robots meta tag also works in X-Robots-Tag. The common ones:

  • index / noindex: allow or block the URL from search results.
  • follow / nofollow: allow or block crawlers from following links on the page.
  • none: shorthand for noindex, nofollow.
  • noarchive: don't show a cached copy in search results.
  • nosnippet: don't show a text snippet or video preview.
  • max-snippet:[number]: cap the snippet length in characters. max-snippet:-1 means no limit.
  • max-image-preview:[none|standard|large]: control image preview size.
  • max-video-preview:[number]: cap video preview length in seconds.
  • noimageindex: don't index images on the page.
  • notranslate: don't offer a translated version in results.
  • unavailable_after:[date]: stop showing the URL in results after a specific date, useful for time-limited content like event pages or press releases.
  • indexifembedded: lets a noindex page still get indexed when it's embedded in another page via iframe. Google introduced this one on January 21, 2022, and it's the newest directive on this list, which is probably why most guides still skip it.

Directives apply to all crawlers by default. To target one specifically, prefix it with the bot's name: X-Robots-Tag: googlebot: noindex.

The gotcha: robots.txt can block X-Robots-Tag from being read

This trips up more people than any directive on the list. A crawler has to fetch a URL before it can read that URL's headers. If robots.txt disallows crawling for a path, the crawler never requests it, so it never sees the X-Robots-Tag on it, and the noindex directive does nothing.

The fix is to allow crawling in robots.txt and let X-Robots-Tag handle indexing. Blocking both at once doesn't stack the protection. It just means the page might still get indexed anyway if something else links to it, since Google can index a URL from external signals alone without ever crawling it.

Check what your robots.txt currently blocks before you add any X-Robots-Tag rule, or the header you just added won't do anything.

How to set it up

The syntax depends on your server or hosting platform. Apache and Nginx examples are everywhere, so here they are along with the setups most guides skip: Next.js and Vercel.

Apache (.htaccess), blocking all PDFs site-wide:

typescript
1<Files ~ ".pdf$">
2  Header set X-Robots-Tag "noindex, nofollow"
3</Files>

Nginx, same rule:

typescript
1location ~* .pdf$ {
2  add_header X-Robots-Tag "noindex, nofollow";
3}

Next.js, via next.config.js, for a static rule tied to a path pattern:

javascript
1module.exports = {
2  async headers() {
3    return [
4      {
5        source: '/preview/:path*',
6        headers: [
7          { key: 'X-Robots-Tag', value: 'noindex' },
8        ],
9      },
10    ]
11  },
12}

If the rule needs to be dynamic, for example only on draft content resolved at request time, use middleware instead:

typescript
1import { NextResponse } from 'next/server'
2import type { NextRequest } from 'next/server'
3 
4export function middleware(request: NextRequest) {
5  const response = NextResponse.next()
6  if (request.nextUrl.pathname.startsWith('/preview')) {
7    response.headers.set('X-Robots-Tag', 'noindex')
8  }
9  return response
10}
11 
12export const config = {
13  matcher: '/preview/:path*',
14}

Vercel, via vercel.json, when you're not on Next.js or want the rule at the platform level:

javascript
1{
2  "headers": [
3    {
4      "source": "/documents/(.*).pdf",
5      "headers": [
6        { "key": "X-Robots-Tag", "value": "noindex" }
7      ]
8    }
9  ]
10}

Cloudflare Workers, for edge-level control in front of any origin:

javascript
1export default {
2  async fetch(request) {
3    const response = await fetch(request)
4    const newResponse = new Response(response.body, response)
5    newResponse.headers.set('X-Robots-Tag', 'noindex')
6    return newResponse
7  }
8}

Whichever platform you're on, scope the rule to the narrowest path pattern that covers your case. A rule on /:path* is one typo away from deindexing the whole site.

How to check if it's live

You won't see X-Robots-Tag by viewing page source, since it's not in the HTML. Check the response headers instead.

With curl:

typescript
1curl -I https://example.com/report.pdf

Look for the header in the output:

typescript
1HTTP/2 200
2content-type: application/pdf
3x-robots-tag: noindex

For a faster check without a terminal, I built X-Robots-Tag detection into my HTTP status tool. Paste a URL and it shows the status code, redirect chain, and response headers including X-Robots-Tag in one pass. I added it after losing time to exactly this bug: a staging subdomain that got indexed because the header was set on the wrong path pattern. My Chrome extension, Keep Optimized, does the same header check from the toolbar while you're on the page, which is faster when you're auditing as you browse rather than checking one URL at a time.

Common mistakes

  • A wildcard path pattern that's too broad. source: '/:path*' in Next.js or location ~* .*$ in Nginx catches more than intended, including pages you meant to keep indexed.
  • Setting noindex and then wondering why the URL is still in search results. If robots.txt blocks the path, see the gotcha above. Google has to crawl the page to see the header.
  • Forgetting the header after a migration or redesign. Staging environments often carry a sitewide noindex header. If that config gets copied to production without removing it, the whole site drops out of search. Always check response headers on the live domain right after a deploy.
  • Conflicting directives across layers. If the meta tag says index and the X-Robots-Tag header says noindex, the more restrictive rule wins. Don't rely on that behavior. Keep one source of truth per URL.

Frequently asked questions

Can I see X-Robots-Tag in my browser like a regular meta tag?

No. It's an HTTP response header, not part of the page's HTML, so it won't show up in view-source. Use browser DevTools (Network tab, click the request, check Response Headers) or run curl -I against the URL.

Does X-Robots-Tag work on pages rendered with JavaScript?

Yes, and this is actually where it has an advantage over the meta tag. The header is set by the server before any JavaScript runs, so it doesn't depend on client-side rendering completing correctly. A meta tag injected by JavaScript can be missed if a crawler doesn't fully render the page.

Does Bing support X-Robots-Tag the same way as Google?

Mostly. Bing supports noindex, nofollow, max-snippet, max-image-preview, and max-video-preview. It uses nocache as its own synonym for noarchive. Directive support outside the major engines varies, so test on the specific crawler you care about.

Is the "X-" in X-Robots-Tag outdated?

The IETF deprecated the general practice of prefixing custom HTTP headers with "X-" back in RFC 6648, published in June 2012. That guidance was about new header names going forward. X-Robots-Tag predates it and is too established to rename, the same way X-Frame-Options stuck around. The prefix is a naming artifact at this point, not a sign the header itself is deprecated.

What to check next

X-Robots-Tag earns its place the moment you need to control indexing on something that isn't an HTML page, or you need one rule to apply across a whole path pattern instead of editing pages one by one. Start by checking what's already set on your non-HTML files. PDFs, exported reports, and staging paths are the usual places a header got set once and forgotten. Pull the response headers on a handful of URLs first. If robots.txt is blocking any of them, fix that before you touch the header, or the directive you add won't be read at all.

Oleksii Khoroshun

About the author

Oleksii Khoroshun

SEO specialist at SE Ranking with 8+ years of technical and on-page work. Led migrations, built ranking strategies for sites from 10K to 100K+ pages, and shipped Chrome extensions for workflows no existing tool handled well. Top Rated on Upwork (100% Job Success Score).

LinkedInAbout me

More from the blog

View all

Discovered currently not indexed: what it means and how to fix it

Discovered currently not indexed means Google found your URL but has not crawled it yet. Here are the real causes and the steps to get pages indexed fast.

AEO vs SEO: What's the Difference and How to Measure Both

AEO and SEO target different surfaces but share the same signals. Here is what separates them, where GEO fits in, and how to measure all three in one stack.

How to track AI chatbot traffic in GA4: 3 methods (2026)

GA4 now has a native AI Assistant channel, but it misses most AI sessions. 3 methods to track AI chatbot traffic from ChatGPT, Perplexity, Gemini and more.