Robots Meta Tag
The robots meta tag is an HTML element in the `<head>` that provides page-level instructions to search engine crawlers about indexing and link-following behavior.
While robots.txt controls crawling at the URL level, the robots meta tag controls indexing and link behavior at the page level. Common directives: `index`/`noindex` (whether to index the page), `follow`/`nofollow` (whether to follow links), `nosnippet` (no text snippet in results), `noarchive` (no cached version), and `max-snippet` (limit snippet length).
The meta tag only works if the crawler can access the page — if robots.txt blocks the URL, the crawler never sees the meta tag. This is a common mistake: blocking a page in robots.txt expecting it to be deindexed, while it remains indexed because the noindex tag is never seen.
In Next.js App Router, set robots directives via the `metadata.robots` property. Indxel validates that robots meta tags are consistent with your robots.txt rules and flags conflicts.
Example
// Next.js App Router — robots meta tag
export const metadata = {
robots: {
index: true,
follow: true,
"max-snippet": -1,
"max-image-preview": "large",
"max-video-preview": -1,
},
};
// Noindex a specific page
export const metadata = {
robots: { index: false, follow: true },
};Related terms
Noindex
Noindex is a robots meta tag directive that instructs search engines to exclude a page from their search index, preventing it from appearing in search results.
Nofollow
Nofollow is a link attribute (`rel="nofollow"`) that tells search engines not to pass PageRank (link equity) through a specific link, treating it as a hint rather than a directive.
Robots.txt
Robots.txt is a plain text file at the root of a website that instructs search engine crawlers which URLs they are allowed or disallowed from accessing.
X-Robots-Tag
X-Robots-Tag is an HTTP response header that provides the same indexing directives as the robots meta tag but can be applied to any file type, including PDFs, images, videos, and API responses.
Stop shipping broken SEO
Indxel validates your metadata, guards your CI/CD pipeline, and monitors indexation — so you never miss an SEO issue again.