IndexNow API: Instant Indexation for Developers

You migrate a Next.js application from the Pages router to the App router. You change the URL structure for 4,000 product pages to implement a new nested routing scheme. You deploy, update the sitemap, and wait. Three weeks later, your server logs show Bingbot has crawled exactly 42 of the new URLs. Organic traffic from Bing, Yandex, and Naver stalls, dropping 18% overall.

Waiting for search engine crawlers to discover sitemap changes is passive and unpredictable. The IndexNow API makes indexation active. You push a URL payload, and the search engine schedules it for immediate crawling.

Here is exactly how to implement the IndexNow protocol, host the verification key in a Next.js app, construct the POST payload, and automate the entire submission process in your CI/CD pipeline.

What is the IndexNow protocol and who supports it?

IndexNow is an open-source ping protocol created by Microsoft and Yandex that allows websites to notify search engines immediately when URLs are created, updated, or deleted. It is supported by Bing, Yandex, Naver, and Seznam, but not Google.

The protocol operates on a single HTTP POST request. Instead of pinging each search engine individually, you send your payload to a central endpoint (like api.indexnow.org). That endpoint automatically distributes the URLs to all participating search engines.

You can submit up to 10,000 URLs per request. This replaces the legacy approach of pinging an XML sitemap URL and hoping the crawler parses the <lastmod> tags. With IndexNow, if a URL is in the payload, the engine knows it requires immediate attention.

How does IndexNow compare to the Google Indexing API?

IndexNow is an open protocol accepting up to 10,000 URLs per request for general web content, while the Google Indexing API is a proprietary, quota-restricted endpoint limited to 200 URLs per day specifically for Job Postings and Broadcast Events.

Google strictly gatekeeps its Indexing API. If you attempt to use the Google Indexing API for standard e-commerce product pages or blog posts, your requests will return a 200 OK but Googlebot will silently drop the URLs. IndexNow has no content-type restrictions.

IndexNow wins on developer experience. You generate a text file, host it, and send a standard JSON payload. Google requires provisioning a Google Cloud Project, generating a Service Account JSON key, exchanging it for a short-lived OAuth token, and handling granular quota limits.

How do you generate and host an IndexNow API key?

You must generate a minimum 8-character hex key, save it as a .txt file matching the key name, and host it at the root of your domain to verify ownership.

Search engines use this file to prevent malicious actors from submitting URLs on your behalf. Generate a secure 32-character hex key using OpenSSL in your terminal:

openssl rand -hex 16
# Output: 8f7b2c9a1e3d4f5b6a7c8d9e0f1a2b3c

If you are building a static site, create a file named 8f7b2c9a1e3d4f5b6a7c8d9e0f1a2b3c.txt in your public directory. The contents of the file must be exactly the key itself.

If you are using Next.js App Router, serving a static file works, but managing environment variables across staging and production environments is safer. You can serve the key dynamically using a Route Handler.

Create the file app/[key]/route.ts:

import { NextResponse } from 'next/server';
 
export async function GET(
  request: Request,
  { params }: { params: { key: string } }
) {
  const indexNowKey = process.env.INDEXNOW_KEY;
 
  // Ensure the requested filename matches the key exactly
  if (params.key !== `${indexNowKey}.txt`) {
    return new NextResponse('Not Found', { status: 404 });
  }
 
  // The file content must be the key itself
  return new NextResponse(indexNowKey, {
    headers: {
      'Content-Type': 'text/plain',
    },
  });
}

How to construct the IndexNow POST request?

Send a JSON payload to https://api.indexnow.org/indexnow containing your host, key, key location, and an array of up to 10,000 fully qualified URLs.

The keyLocation parameter is optional but highly recommended. If you omit it, search engines will attempt to find the key at https://yourdomain.com/YOUR_KEY.txt. If your Next.js middleware or routing structure serves the key from a different path, you must specify keyLocation.

Here is the TypeScript implementation using the native fetch API:

interface IndexNowPayload {
  host: string;
  key: string;
  keyLocation: string;
  urlList: string[];
}
 
async function submitToIndexNow(urls: string[]) {
  const host = 'www.example.com';
  const key = process.env.INDEXNOW_KEY!;
  
  const payload: IndexNowPayload = {
    host,
    key,
    keyLocation: `https://${host}/${key}.txt`,
    urlList: urls,
  };
 
  const response = await fetch('https://api.indexnow.org/indexnow', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json; charset=utf-8',
    },
    body: JSON.stringify(payload),
  });
 
  return response;
}

Every URL in the urlList array must be a fully qualified string matching the host exactly. Submitting https://example.com/page when the host is www.example.com results in a 400 Bad Request.

How do you handle IndexNow API rate limits and errors?

The IndexNow API returns HTTP 200 for success, 202 for queued requests, 400 for invalid formats, 403 for key mismatch, and 429 when you exceed the undocumented rate limit.

While the protocol allows 10,000 URLs per request, sending 50 requests of 10,000 URLs in a single second triggers a 429 Too Many Requests. You must implement exponential backoff and array chunking for bulk submissions.

Here is a robust submission function that chunks URLs into batches of 10,000 and retries on 429 or 5xx errors:

async function submitWithRetry(urls: string[], maxRetries = 3) {
  // Chunk URLs into batches of 10,000
  const chunkSize = 10000;
  const batches = [];
  for (let i = 0; i < urls.length; i += chunkSize) {
    batches.push(urls.slice(i, i + chunkSize));
  }
 
  for (const batch of batches) {
    let attempt = 0;
    let success = false;
 
    while (attempt < maxRetries && !success) {
      try {
        const res = await submitToIndexNow(batch);
 
        if (res.status === 200 || res.status === 202) {
          console.log(`Successfully submitted batch of ${batch.length} URLs`);
          success = true;
          continue;
        }
 
        if (res.status === 403) {
          throw new Error('IndexNow Key verification failed.');
        }
 
        if (res.status === 429) {
          const delay = Math.pow(2, attempt) * 1000;
          console.warn(`Rate limited. Retrying in ${delay}ms...`);
          await new Promise(r => setTimeout(r, delay));
          attempt++;
        } else {
          throw new Error(`Unexpected status: ${res.status}`);
        }
      } catch (error) {
        if (attempt === maxRetries - 1) throw error;
        attempt++;
      }
    }
  }
}

How to extract changed URLs from a Next.js build?

To submit only changed URLs to IndexNow, parse the sitemap.xml generated in your .next build directory and compare it against the live sitemap using a Node.js script.

Submitting your entire database of 500,000 URLs on every deployment is inefficient and increases the risk of rate-limiting. The optimal strategy is to diff the URLs.

First, install xml2js to parse the sitemaps:

npm install xml2js

Create a script scripts/indexnow-sync.ts that runs after your framework builds. This script fetches the current live sitemap, reads the newly generated local sitemap, and extracts the difference:

import fs from 'fs';
import path from 'path';
import { parseStringPromise } from 'xml2js';
 
async function getLiveUrls(domain: string): Promise<Set<string>> {
  try {
    const res = await fetch(`https://${domain}/sitemap.xml`);
    if (!res.ok) return new Set();
    const xml = await res.text();
    const parsed = await parseStringPromise(xml);
    const urls = parsed.urlset.url.map((u: any) => u.loc[0]);
    return new Set(urls);
  } catch {
    return new Set();
  }
}
 
async function getLocalUrls(): Promise<string[]> {
  // Path depends on your framework setup. For Next.js static export:
  const sitemapPath = path.join(process.cwd(), 'out', 'sitemap.xml');
  if (!fs.existsSync(sitemapPath)) return [];
  
  const xml = fs.readFileSync(sitemapPath, 'utf-8');
  const parsed = await parseStringPromise(xml);
  return parsed.urlset.url.map((u: any) => u.loc[0]);
}
 
async function run() {
  const domain = 'www.example.com';
  const liveUrls = await getLiveUrls(domain);
  const localUrls = await getLocalUrls();
 
  const newOrUpdatedUrls = localUrls.filter(url => !liveUrls.has(url));
 
  if (newOrUpdatedUrls.length === 0) {
    console.log('No new URLs detected. Skipping IndexNow submission.');
    return;
  }
 
  console.log(`Found ${newOrUpdatedUrls.length} new URLs. Submitting...`);
  await submitWithRetry(newOrUpdatedUrls);
}
 
run();

How to automate IndexNow submissions on Vercel deployments?

You can automate IndexNow submissions by executing your sync script inside a GitHub Actions workflow that triggers immediately after a successful Vercel deployment.

Running the submission as a postbuild script inside Vercel directly is risky. If the submission fails or takes too long due to rate limits, it fails the entire deployment. Decoupling the submission from the critical build path ensures your site ships regardless of the IndexNow API's uptime.

Create .github/workflows/indexnow.yml:

name: IndexNow Sync
on:
  deployment_status:
 
jobs:
  submit-urls:
    if: github.event.deployment_status.state == 'success' && github.event.deployment.environment == 'Production'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
 
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
 
      - name: Install dependencies
        run: npm ci
 
      - name: Run IndexNow Sync
        env:
          INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}
        run: npx tsx scripts/indexnow-sync.ts

This workflow waits for Vercel to report a successful production deployment, checks out the code, and runs the diffing script. A typical Next.js app with 5,000 pages takes 3 seconds to parse the XML, diff the sets, and dispatch the POST request.

How to submit URLs to IndexNow using Indxel?

Indxel abstracts the IndexNow and Google Indexing APIs into a single CLI command that automatically detects new URLs in your build and submits them to all search engines.

Writing custom XML parsers, managing diff states, and handling 429 exponential backoffs adds maintenance overhead to your infrastructure. Indxel handles the protocol requirements natively. It verifies the presence of your key, validates the URLs for formatting errors, and dispatches them to Bing, Yandex, Naver, and Google simultaneously.

Instead of writing the 150 lines of parsing and retry logic above, you run one command in your CI pipeline:

npx indxel sync --providers=indexnow,google --diff

The CLI outputs standard compiler-style logs. It parses the sitemap, skips unmodified pages, and reports the exact HTTP status codes returned by the engines:

$ npx indxel sync --providers=indexnow,google --diff
 
[indxel] Loaded sitemap from .next/server/app/sitemap.xml
[indxel] Found 4,000 URLs (42 new, 3,958 unchanged)
[indxel] Validating URLs... 42/42 valid.
[indxel] Submitting to IndexNow (Bing, Yandex, Naver)...
[indxel] ✓ IndexNow: 200 OK (42 URLs)
[indxel] Submitting to Google Indexing API...
[indxel] ✓ Google: 200 OK (42 URLs)
 
Done in 1.4s. 42 URLs submitted successfully.

If you misconfigure your key file, Indxel catches it locally before dispatching the payload, failing the command with a clear file path error rather than a generic 403.

Frequently Asked Questions

Does Google support the IndexNow API?

No, Google does not support the IndexNow API. Google relies strictly on XML sitemaps and its proprietary Google Indexing API (which is restricted to specific schemas like JobPosting). To achieve instant indexation across all engines, you must implement IndexNow for Bing/Yandex and the Google Indexing API for Google.

How many URLs can I submit to IndexNow per day?

You can submit up to 10,000 URLs per POST request, and there is no strict daily limit. However, search engines monitor the quality of the submitted URLs. If you repeatedly submit spam, 404 pages, or unchanged content, the search engines will permanently ignore your API key.

Do I need to submit to Bing and Yandex separately?

No, submitting to api.indexnow.org distributes the URLs to all participating engines automatically. You do not need to send separate POST requests to bing.com/indexnow or yandex.com/indexnow. A single request to the central API is shared across the network.

Can I use IndexNow to delete URLs from search results?

Yes, you submit the URL exactly as you would a new page. When the search engine crawler visits the submitted URL and receives a 404 Not Found or 410 Gone HTTP status code, it immediately purges the URL from its index.

What happens if I change my IndexNow key?

If you rotate your key, you must update the .txt file on your server before sending the next POST request. Search engines do not cache the old key. The verification happens dynamically when the API receives your payload. If the file contains the new key, the submission succeeds.

To validate your IndexNow configuration and submit your first batch of URLs right from your terminal, run the Indxel sync command:

npx indxel sync --providers=indexnow --dry-run