Automated Sitemap Submissions: Bing & Google Guide
Software engineer and entrepreneur based in San Francisco.
Purpose of this Guide
Automating sitemap submissions to search engines like Bing (via IndexNow) and Google Search Console ensures they discover your content updates faster. This guide details how to set up a Node.js/TypeScript script for this, including private key handling and development environment considerations.
Understanding the APIs
1. Google Search Console API
The Google Search Console API allows programmatic interaction with Google Search Console. For sitemap submission, it enables an application to inform Google about new or updated sitemaps without manual intervention in the Search Console interface, which can help Google schedule crawls more efficiently.
2. Bing IndexNow API
IndexNow is an initiative by Microsoft Bing, Yandex, and other search engines allowing websites to notify search engines whenever their website content is created, updated, or deleted. Submitting a sitemap URL via IndexNow alerts participating search engines to update their index with the latest changes quickly. Its setup is generally straightforward.
Setting Up Sitemap Submissions
1. Set Up Google Service Account & API Access for Sitemap Submission
To submit sitemaps to Google programmatically, a Service Account with access to the Google Search Console API is required. The process involves several steps in Google Cloud Platform (GCP).
- Enable API: In your GCP project, enable the "Google Search Console API" and optionally the "Search Indexing API." The 'Google Search Console API' is for submitting the sitemap file, while the 'Search Indexing API' can be used for submitting individual URLs for faster updates, if supported by your script.
- Create Service Account: In GCP, navigate to "IAM & Admin" > "Service Accounts." Create a new service account with a descriptive name. Granting a project-level role at this stage is typically not necessary for this specific purpose.
- Download JSON Key: After creating the service account, generate a JSON key and download it. This file contains the credentials for the script, including the private key and client email. Store this file securely.
- Grant Search Console Permissions: In Google Search Console, navigate to your verified property. Go to "Settings" > "Users and permissions." Add a new user with the service account's email address (e.g.,
your-service-account-email@your-project-id.iam.gserviceaccount.com
). Grant this service account "Owner" or "Full" permission to allow sitemap submissions for that property.
Important Note on siteUrl
Important: The siteUrl
used in API calls (as shown in script examples) must exactly match a property verified in Google Search Console (e.g., https://your-production-domain.com
). Submissions for properties not associated with the service account or against localhost
will be rejected by the API.
2. Prepare IndexNow Verification (Bing)
IndexNow uses a simple verification step. This involves placing a text file in the site's public
root directory (or equivalent static file directory), where the filename acts as the key.
- Generate/Choose Your Key: Decide on a unique key string. This will be the value of your
INDEXNOW_KEY
environment variable and the name of the verification file (without the.txt
extension). - Create the file: In your project's
public
directory, create a text file namedYOUR_KEY.txt
(e.g., if your key isabcdef12345
, createpublic/abcdef12345.txt
). - File content: The file must contain only the key itself (e.g.,
abcdef12345
), with no extra characters, spaces, or the.txt
extension.
Public Accessibility for IndexNow
Accessibility: This verification file must be publicly accessible on your live domain (e.g., https://your-production-domain.com/YOUR_KEY.txt
). IndexNow checks for this file to verify submissions. Submissions may fail (e.g., with a 403 Forbidden error) if the file is not accessible or if submissions are attempted against localhost
.
Implementing the Submission Script
To automate the submission process, a script, such as one written in Node.js/TypeScript, can be used.
googleapis
Client Library (Example)
1. Install For interacting with the Google Search Console API, a common library is googleapis
.
npm install googleapis
# or: bun install googleapis / yarn add googleapis / pnpm install googleapis
2a. Core API Interaction Examples
Below are simplified excerpts demonstrating core interactions with Google Search Console and Bing's IndexNow API. These functions would typically be part of a larger script incorporating comprehensive error handling, environment checks, and authentication setup.
Submitting to Google Search Console:
import { google, type Auth } from 'googleapis';
async function submitSitemapToGoogle(authClient: Auth.JWT, siteUrl: string, sitemapUrl: string): Promise<boolean> {
try {
// Ensure 'webmasters' and 'indexing' scopes are included when creating the authClient
const webmasters = google.webmasters({ version: 'v3', auth: authClient });
await webmasters.sitemaps.submit({
siteUrl: siteUrl, // e.g., "https://your-production-domain.com"
feedpath: sitemapUrl, // e.g., "https://your-production-domain.com/sitemap.xml"
});
console.log(`[SitemapSubmitGoogle] ✅ Successfully submitted sitemap (${sitemapUrl}) to Google for site ${siteUrl}`);
return true;
} catch (error) {
// In a production script, 'error' would likely be an instance of GaxiosError or similar
// console.error("[SitemapSubmitGoogle] ❌ Error submitting to Google:", error.message, error.response?.data);
console.error("[SitemapSubmitGoogle] ❌ Error submitting to Google:", error.message);
return false;
}
}
Submitting to Bing (IndexNow):
async function submitSitemapToBing(siteUrl: string, sitemapUrl: string): Promise<boolean> {
const indexNowKey = process.env.INDEXNOW_KEY; // From an environment variable
if (!indexNowKey) {
console.error('[SitemapSubmitBing] ❌ INDEXNOW_KEY not set. Skipping Bing/IndexNow submission.');
return false;
}
try {
const host = new URL(siteUrl).host;
const indexnowUrl = `https://www.bing.com/indexnow?url=${encodeURIComponent(sitemapUrl)}&key=${indexNowKey}&host=${host}`;
const response = await fetch(indexnowUrl, { method: 'GET' });
if (response.ok) {
console.log(`[SitemapSubmitBing] ✅ Successfully submitted sitemap (${sitemapUrl}) via IndexNow to Bing`);
return true;
}
console.error(`[SitemapSubmitBing] ❌ IndexNow submission failed: ${response.status} ${response.statusText}`);
return false;
} catch (error) {
console.error('[SitemapSubmitBing] ❌ Error submitting via IndexNow to Bing:', error.message);
return false;
}
}
Note: The authClient
for Google requires proper setup with credentials and scopes (e.g., https://www.googleapis.com/auth/webmasters
). A complete submission script should handle this, along with more comprehensive environment checks.
2. Set Up Submission Script Logic
Background Info
A well-designed submission script might export primary functions such as:
submitSitemapFilesToSearchEngines()
: Submits a generated/sitemap.xml
file to both Google Search Console and Bing (IndexNow), respecting production environment checks.submitIndividualUrlsToGoogle()
: Iterates sitemap entries (e.g., from a Next.js sitemap), selects recently updated pages (e.g., last 14 days), checks Google's last crawl date for each URL, and resubmits only URLs needing an update.
Once a submission script (e.g., path/to/your/submit-sitemap.ts
) is prepared with the necessary functions, it can be invoked. The following illustrates a simple runner script:
import 'dotenv/config'; // For loading environment variables
import {
submitSitemapFilesToSearchEngines,
submitIndividualUrlsToGoogle
} from './path/to/your/submit-sitemap'; // Adjust path as needed
(async () => {
await submitSitemapFilesToSearchEngines();
await submitIndividualUrlsToGoogle(); // Optional, if implementing individual URL submission
})();
3. Example: Submitting the Sitemap File Only
# Example command using Node.js; adapt for your runtime (e.g., Bun, Deno)
node path/to/your/submit-sitemap.ts --sitemaps-only
4. Example: Submitting Recently Updated URLs Only
node path/to/your/submit-sitemap.ts --individual-only
5. Example: Submitting Both (Default Behavior)
node path/to/your/submit-sitemap.ts
Note on script logic: A robust script should include logic to skip submissions if
NODE_ENV
is 'development' AND theSITE_URL
points to localhost. This prevents accidental submissions during local development. It should also log a warning ifNODE_ENV
is 'production' but theSITE_URL
doesn't match the intended production site URL.
Configuration & Environment Details
1. Configure Environment Variables
The script will rely on environment variables for API keys, service account details, and site URLs. These should be defined in your project's .env
file (and templated in a .env.example
or similar).
# For Bing's IndexNow
# The key chosen/generated for the IndexNow verification file.
INDEXNOW_KEY="your-generated-indexnow-key" # e.g., abcdef12345
# For Google Search Console API (values from the downloaded Service Account JSON key)
# Ensure these variable names match those used in the script.
GOOGLE_PROJECT_ID="your-gcp-project-id"
GOOGLE_SEARCH_INDEXING_SA_EMAIL="your-service-account-email@your-project-id.iam.gserviceaccount.com"
# CRITICAL: GOOGLE_SEARCH_INDEXING_SA_PRIVATE_KEY must be the complete string
# from the "private_key" field in the JSON key file, including -----BEGIN PRIVATE KEY-----
# and -----END PRIVATE KEY-----. All
characters from the JSON string must be
# preserved literally within the double quotes.
# The entire multi-line key must be enclosed in a SINGLE pair of double quotes.
GOOGLE_SEARCH_INDEXING_SA_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\nMIIEvg...your...entire...key...goes...here...with...all...the...\n...sequences...preserved...exactly...as...in...the...JSON...file...including...the...final...\n...-----END PRIVATE KEY-----\n"
# Site's main URL (e.g., your Next.js public URL), used by the script to construct sitemap URLs.
# Ensure this matches a verified property in Google Search Console for Google submissions.
SITE_URL="https://your-production-domain.com" # e.g., https://williamcallahan.com (NEXT_PUBLIC_SITE_URL is a common convention for Next.js)
# Node environment, used by the script to conditionally skip submissions.
NODE_ENV="development" # Set to "production" in deployment environments
Key Points for GOOGLE_SEARCH_INDEXING_SA_PRIVATE_KEY
:
- Copy the entire string value from the
private_key
field in the downloaded Service Account JSON key. - This includes
-----BEGIN PRIVATE KEY-----
, all base64 encoded lines, and-----END PRIVATE KEY-----
. - Crucially, all
.env
file. If the.env
parser or shell has issues with this, alternative methods like base64 encoding the key for storage and decoding in the script, or using a secrets manager, may be necessary. Direct string with literal - The entire value should be enclosed in a single pair of double quotes in the
.env
file. - The script logic should process this private key string, replacing
\n
with actual newline characters, which is essential for Google Auth libraries.
2. Automate Script Execution
To ensure sitemaps are submitted regularly or after content updates, integrate the submission script into your workflow:
- Post-build script: Run as part of the deployment process after the site and sitemap are built.
- Scheduled task/Cron job: Set up a recurring job (e.g., daily) to execute the script.
- Webhook: Trigger the script via a webhook from a CMS when content is published or updated.
Example package.json
script entry:
{
"scripts": {
"submit-sitemaps": "node path/to/your/submit-sitemap.ts"
}
}
This can then be run, for example, with npm run submit-sitemaps
.
This setup provides a robust and automated way to manage sitemap submissions to Google and Bing, emphasizing critical aspects like private key formatting and differentiating between development and production environments.