Free X-Robots-Tag Checker — Verify Indexing Directives

What Is X-Robots-Tag?

The X-Robots-Tag is an HTTP response header that allows webmasters and developers to control how search engines crawl, index, and display content from their websites. Unlike the more commonly known meta robots tag, which is embedded directly in the HTML of a page, the X-Robots-Tag is sent as part of the HTTP response headers. This makes it particularly useful for controlling indexing behavior on non-HTML resources such as PDF files, images, video files, and other document types that do not support HTML meta tags.

The X-Robots-Tag was introduced by Google and is now supported by all major search engines, including Bing and Yandex. It accepts the same directives as the meta robots tag, such as noindex, nofollow, noarchive, and nosnippet. When a search engine crawler encounters an X-Robots-Tag header in a server response, it processes the directives in the same way it would process a meta robots tag found in the HTML head section.

One of the key strengths of the X-Robots-Tag is its flexibility. Because it is set at the server level, it can be applied to any response returned by the web server, regardless of file type. This means you can prevent indexing of PDF documents, image galleries, downloadable files, API responses, or any other resource type. You can also apply different directives to different user agents, allowing you to give Google and Bing different instructions for the same content.

Search engines combine directives from multiple sources when processing a page. If a page contains both a meta robots tag and an X-Robots-Tag header, the search engine will follow the most restrictive combination of the two. For example, if the meta robots tag says "index, follow" but the X-Robots-Tag header says "noindex," the page will not be indexed. Understanding this behavior is critical for avoiding accidental deindexing of important pages.

X-Robots-Tag vs Meta Robots: Key Differences

While the X-Robots-Tag and the meta robots tag serve the same fundamental purpose -- instructing search engines on how to handle content -- they differ in important ways that affect when and where you should use each one. Choosing the right mechanism depends on the type of content, your server configuration capabilities, and the scope of the directive.

The meta robots tag is placed inside the HTML head section of a web page using the syntax <meta name="robots" content="noindex, nofollow">. Because it lives inside the HTML document, it can only be used on resources that contain HTML. You cannot add a meta robots tag to a PDF file, an image, a CSS stylesheet, or a JavaScript file. This is the primary limitation that the X-Robots-Tag was designed to address.

The X-Robots-Tag is set via the HTTP response header, which means it can be applied to any type of resource. Whether the response is an HTML page, a PDF document, an image file, or a JSON API response, the X-Robots-Tag header can instruct the search engine on how to handle that resource. This makes it the only viable option for controlling indexing of non-HTML content.

Here is a comparison of the two approaches:

An important consideration is that modifying HTTP headers requires access to server configuration, which may not be available on all hosting platforms. Shared hosting environments often limit access to server headers, making the meta robots tag the more practical option for HTML pages. On dedicated servers, virtual private servers, or platforms like Nginx and Apache where you have full configuration control, the X-Robots-Tag offers more powerful and flexible options.

In practice, most SEO professionals use both approaches together. The meta robots tag is used for page-level HTML control, while the X-Robots-Tag is used for non-HTML resources and for applying broad rules across entire directories or file types through server configuration. The key is to ensure the two methods are not in conflict, since search engines follow the most restrictive directive found in either source.

X-Robots-Tag Directives Explained

The X-Robots-Tag supports a wide range of directives that give you granular control over how search engines interact with your content. Each directive instructs the crawler to handle a specific aspect of indexing, caching, or displaying your content in search results. Multiple directives can be combined in a single header, separated by commas.

How to Set X-Robots-Tag Headers

Setting the X-Robots-Tag header requires access to your web server configuration or application code. The exact method depends on your server software, hosting environment, and the level of control you need. Below are configuration examples for the most common server environments.

Apache (.htaccess or httpd.conf)

Apache uses the Header directive from mod_headers to set X-Robots-Tag headers. You can apply the header globally, to specific file types, or to specific directories:

# Apply noindex to all PDF files
<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

# Apply noindex to specific directories
<Directory "/var/www/html/private">
    Header set X-Robots-Tag "noindex"
</Directory>

# Apply noindex to all image files
<FilesMatch "\.(png|jpg|jpeg|gif|webp|svg)$">
    Header set X-Robots-Tag "noimageindex"
</FilesMatch>

# Target a specific user agent (Google only)
<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "googlebot: noindex"
</FilesMatch>

# Combine multiple directives
Header set X-Robots-Tag "noindex, noarchive, max-snippet:0"

# Apply to multiple user agents with separate headers
Header append X-Robots-Tag "googlebot: noindex"
Header append X-Robots-Tag "bingbot: noindex, nofollow"

Nginx

In Nginx, use the add_header directive within server or location blocks to set X-Robots-Tag headers:

# Apply noindex to all PDF files
location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

# Apply noindex to an entire directory
location /private/ {
    add_header X-Robots-Tag "noindex" always;
}

# Apply noindex to staging or preview pages
location /preview/ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

# Apply to all image files
location ~* \.(png|jpg|jpeg|gif|webp|svg)$ {
    add_header X-Robots-Tag "noimageindex" always;
}

# Target specific user agent
location ~* \.pdf$ {
    add_header X-Robots-Tag "googlebot: noindex" always;
}

# Set on the entire site (use with caution)
server {
    # ...
    add_header X-Robots-Tag "max-image-preview:large" always;
}

The always keyword ensures the header is sent regardless of the response status code, which is important for 4xx and 5xx error pages. Without it, Nginx only adds custom headers to successful (2xx) and redirect (3xx) responses.

Application-Level (PHP, Node.js, Python)

You can also set the X-Robots-Tag header directly from your application code, which provides the most dynamic control:

# PHP
header("X-Robots-Tag: noindex, nofollow", true);

# Node.js (Express)
app.use('/private', (req, res, next) => {
    res.set('X-Robots-Tag', 'noindex, nofollow');
    next();
});

# Python (Django middleware)
class XRobotsTagMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
    def __call__(self, request):
        response = self.get_response(request)
        if request.path.startswith('/private/'):
            response['X-Robots-Tag'] = 'noindex, nofollow'
        return response

# Next.js (in next.config.js headers)
async headers() {
    return [
        {
            source: '/private/:path*',
            headers: [
                { key: 'X-Robots-Tag', value: 'noindex, nofollow' },
            ],
        },
    ];
}

Application-level configuration is ideal when the decision to noindex a page depends on dynamic factors such as the content type, user role, or business logic that cannot easily be expressed in static server configuration rules.

How to Check X-Robots-Tag Headers

Verifying that your X-Robots-Tag headers are set correctly is essential for ensuring your indexing strategy works as intended. There are several methods to inspect HTTP headers returned by your server.

Using Browser Developer Tools

Open your browser's developer tools (F12 in Chrome, Firefox, or Edge), navigate to the Network tab, and reload the page. Click on the primary document request and inspect the Response Headers section. Look for the X-Robots-Tag header. If present, its value will show the directives being applied. This method works for any resource type, including PDF files and images, as long as the browser loads them directly.

Using Command Line Tools

The curl command-line tool is one of the quickest ways to inspect HTTP response headers. Use the -I flag to fetch only the headers, or -D - to see headers alongside the body:

# Fetch only headers
curl -I https://example.com/document.pdf

# Look specifically for X-Robots-Tag
curl -sI https://example.com/ | grep -i "x-robots-tag"

# Check with a specific user agent
curl -A "Googlebot/2.1" -sI https://example.com/ | grep -i "x-robots-tag"

Using Our Free X-Robots-Tag Checker

Our X-Robots-Tag Checker tool automatically crawls your website and inspects the HTTP headers of every page and resource. It identifies pages with noindex directives, detects conflicts between X-Robots-Tag headers and meta robots tags, checks for accidental noindex on important pages, and flags staging or development configuration that may have leaked into production. The tool provides a comprehensive report with prioritized issues and actionable fix recommendations.

Using Google Search Console

Google Search Console's URL Inspection tool shows how Google sees your pages, including any X-Robots-Tag directives that were detected during crawling. If Google has found an X-Robots-Tag with noindex on a page, the URL Inspection tool will report it as "Excluded by noindex." The Coverage report also aggregates pages that are excluded from indexing, making it easy to spot patterns where X-Robots-Tag may be applied unintentionally.

Common X-Robots-Tag Mistakes

Misconfigured X-Robots-Tag headers can cause significant damage to your search visibility. Because the header operates at the server level and is often invisible to content editors, mistakes can persist for weeks or months before being discovered. Here are the most common errors and how to avoid them.

Accidental noindex on Important Pages

The most damaging mistake is accidentally applying a noindex directive to pages that should be indexed. This typically happens when a broad server configuration rule unintentionally matches important pages, or when a staging or development configuration is deployed to production without being removed. For example, a rule intended to noindex a specific directory may use a regex pattern that also matches other URLs. Always test your server configuration rules against your full URL inventory before deploying.

To prevent this, maintain an explicit list of URL patterns that should have noindex applied, and implement monitoring that alerts you if the number of indexed pages in Google Search Console drops unexpectedly. A sudden decrease in indexed pages often indicates an accidental noindex deployment.

Staging Configuration Leaking to Production

Many development teams apply a global X-Robots-Tag: noindex header to staging and development environments to prevent search engines from indexing test content. When the application or server configuration is deployed to production without removing this header, the entire production site becomes deindexed. This is one of the most common and catastrophic SEO mistakes that teams make.

The solution is to use environment-specific configuration that sets the X-Robots-Tag header conditionally based on the environment. In your deployment pipeline, verify that noindex headers are not present in production builds. Implement automated checks that flag any X-Robots-Tag: noindex header on production URLs as a critical alert.

Conflicting Directives Between X-Robots-Tag and Meta Robots

When both X-Robots-Tag and meta robots directives exist for the same page, search engines follow the most restrictive combination. If your meta robots tag says "index, follow" but your X-Robots-Tag header says "noindex," the page will not be indexed. These conflicts are difficult to detect because they involve two different systems -- the HTML template and the server configuration -- that are often managed by different teams.

Audit your site regularly by comparing the meta robots directives in HTML against the X-Robots-Tag headers returned by the server. Our checker tool automates this comparison and highlights any conflicts. When conflicts are found, decide which system should be authoritative for each URL pattern and remove the conflicting directive from the other.

Sitemap Conflicts with noindex

Including URLs in your XML sitemap that have an X-Robots-Tag: noindex header sends conflicting signals to search engines. The sitemap tells crawlers "this page is important, please index it," while the header says "do not index this page." Google has stated that when these signals conflict, the noindex directive takes precedence. However, this inconsistency wastes crawl budget, generates warnings in Google Search Console, and indicates poor site configuration.

Ensure that any URL with a noindex directive (whether via X-Robots-Tag or meta robots) is excluded from your XML sitemap. Set up automated sitemap generation that checks for noindex directives before including a URL. If you use a CMS plugin to generate your sitemap, verify that it respects both meta robots tags and X-Robots-Tag headers.

Forgetting to Include the Header on All Response Codes

In Nginx, custom headers added with add_header are only sent on 2xx and 3xx responses by default. If you need the X-Robots-Tag to apply to error pages (such as a custom 404 page that you do not want indexed), you must include the always keyword. In Apache, the Header directive applies to all responses by default, but be aware of the interaction between set, append, and merge behaviors to avoid overwriting or duplicating headers.

When to Use X-Robots-Tag vs Meta Robots

Choosing between X-Robots-Tag and meta robots depends on your specific use case. In many situations, the answer is clear-cut based on the type of resource and the control you need. Here are guidelines for making the right choice.

Use X-Robots-Tag When:

Use Meta Robots When:

For maximum control and coverage, many sites use both mechanisms in coordination. The meta robots tag handles per-page decisions within the HTML, while the X-Robots-Tag handles non-HTML resources and broad directory-level rules at the server configuration level. The key is to document your indexing strategy clearly so that both systems remain consistent and no conflicts arise.

X-Robots-Tag Best Practices

Following best practices for X-Robots-Tag implementation ensures that your indexing strategy is effective, maintainable, and free of unintended consequences. These guidelines are based on Google's official documentation and industry experience.