Automated indexing for new pages

What the automated indexing workflow does

Every few hours the workflow reads your sitemap, asks Google's URL Inspection API which URLs are actually indexed, and sends the eligible ones that aren't to the Google Indexing API. Everything that isn't eligible lands on a list for you to act on. Submissions are logged, so the workflow never burns quota re-sending the same page.

About 2-5 minutes per run. Most runs do nothing because everything is already indexed – the workflow only burns time when there are genuinely new pages to push.

Why new pages wait days or weeks for Google

How long does Google take to index new pages? Google's own documentation says crawling "can take anywhere from a few days to a few weeks", and a site with little crawl demand sits at the slow end. For a site publishing 5-10 pages a week, that's a steady backlog of pages earning nothing while they wait.

Google gives you two ways to ask for a crawl. The Request Indexing button in Search Console's URL Inspection tool works for any page, one URL at a time. The Google Indexing API is easier to automate, but Google's documentation says it "can only be used" for pages with JobPosting or livestream BroadcastEvent markup. Either way, the problem is doing it consistently:

1. You publish a page. You're meant to copy-paste the URL into GSC's URL Inspection tool, click "Request Indexing", wait for the "URL submitted" confirmation. Five minutes per page.
2. You forget. The page sits unindexed for a month before someone notices traffic isn't coming.
3. You remember sometimes. The pages you remembered to submit get indexed in a day. The ones you forgot don't. Your indexation rate becomes random.

The workflow turns "check every new URL" into something that happens whether anyone remembers or not. It reads your sitemap (already authoritative, since your CMS generates it), checks what's indexed, submits what's eligible, flags the rest, and logs everything for audit.

On sites publishing 100+ pages a quarter, the flagged list is often worth more than the submissions. A page that stays unindexed after Google has crawled it is usually telling you about quality or duplication, and no API call fixes that.

A sitemap URL and a service account

Two config values, set once:

• The sitemap URL (usually https://yourdomain.com/sitemap.xml)
• The GSC property the site is verified under (e.g. https://yourdomain.com/)

Plus a service account with GSC access to your property and the Indexing API enabled in Google Cloud. Anything beyond Google's default testing quota needs Google's approval, requested from the same console. The setup doc walks you through this – it's 10 minutes of clicking in Google Cloud Console.

That's it. The workflow runs on its own from there.

A submission log you can audit

Every run produces:

• A list of unindexed URLs, split into eligible pages that were submitted and everything else, which needs a manual request or a fix
• A Google Sheet row per submission – URL, date, response from the API, status (success / quota-exceeded / error)
• Nothing in your inbox unless something errors. The workflow is meant to be invisible when it's working.

You can open the sheet any time to see indexing trends: how many pages got pushed last month, which ones failed, whether your sitemap is generating expected URL volume.

When pages arrive in bulk, say from a listicle pipeline or a programmatic build, read the sheet weekly: a jump in unindexed URLs shows up there first.

Runtime and the URL Inspection limits

Workflow time: 2-5 minutes per scheduled run. Most of it is waiting on GSC's URL Inspection API, which Google caps at 600 queries a minute and 2,000 a day per site.

Your time: zero, after setup. The workflow runs on the schedule you pick (every 2 hours, every day, whatever fits the publishing pace). No notification spam by default.

End-to-end per month for a typical site (~50 new URLs): about 30-60 minutes of workflow runtime spread across the month. Your time: zero.

Sites that get the full push

Run the full workflow if your site matches most of this:

• You publish at least 4-5 new pages a week. Below that, manual submission via GSC is fine.
• Your sitemap is up to date – generated by your CMS or by a tool you trust. Garbage sitemap = garbage submissions.
• Your new pages are eligible, or a flag-only run is enough. Google limits the Indexing API to JobPosting and livestream pages and applies its spam policies to anything submitted, so everything else goes on the manual list.
• You'd rather get pages indexed in days, not weeks. For some sites this is a competitive edge, especially in news-adjacent and rapid-publishing niches.

Job boards and live-streaming sites get the full push. Everyone else gets the inspection log, which still shows at a glance what Google hasn't picked up.

When the Google Indexing API is the wrong tool

Four cases where it won't help:

• Your site publishes a handful of pages a quarter. Manual submission for 5-10 pages is faster than building this.
• You don't have GSC verified. Whole thing requires it.
• You want ordinary blog posts or landing pages pushed through the Google Indexing API. Google's documentation limits the API to job posting and livestream pages, and I won't set it up to submit anything else. For those pages the workflow stops at inspection.
• You're trying to fix a site-wide indexation problem that comes from low page quality, duplicate content, or thin pages. Submission won't help; the page will get crawled and rejected. Fix the underlying issues first.

Teams most often hope submissions will fix the last case. If Google crawls a page and still leaves it out, the page is the problem.

From sitemap to submission in n8n

In n8n, a run goes from sitemap to submission like this:

1. Schedule trigger (every 2 hours by default)
2. Fetch sitemap XML over HTTP
3. Parse XML to JSON. If the sitemap is a sitemap index, recurse into each child sitemap.
4. Filter to URLs published in the last N days (configurable – default 90)
5. Check submission history in Google Sheets – skip anything already submitted in the last 14 days
6. For each remaining URL: call GSC's URL Inspection API. Did Google index it?
7. Filter to URLs Google says are NOT indexed
8. For each eligible page (JobPosting or livestream markup): call the Indexing API with a URL_UPDATED request, and log the rest for a manual request
9. Add a delay between submissions to stay under the rate limit
10. Log each submission to the history sheet

There's also a fallback path: if a URL was submitted but Google's response was "Quota exceeded", the workflow queues it for retry next run.

The reason this is a workflow and not a one-off script is the history-aware logic. A naive version submits every URL every run, burns its daily quota in 10 minutes, and gets you nothing. The history check + 14-day debounce means submissions actually get spread out so the quota lasts the day.

What's handed over with the workflow

The handover covers the whole setup:

• The full n8n workflow file
• The Google Sheet template (submission history + a dashboard tab)
• A setup doc walking through service account creation in Google Cloud + GSC permission grants
• A runbook for the common errors (quota exceeded, permission denied, sitemap-not-reachable)
• A Loom showing one full run from sitemap to submission
• Optional add-on: email alert if the daily quota gets exhausted (means your sitemap is generating URLs faster than the API can absorb – useful early signal)

Once it's running, the history sheet doubles as an indexing record you can show a client or a stakeholder.

Four details naive versions skip

Most automatic website indexing tools are an API call on a timer, and the mechanics here are simple too: n8n and two Google APIs. What earns its keep is the logic around them:

• The 14-day debounce so you don't re-submit the same URL hundreds of times and waste quota. Sounds obvious. Naive versions skip it.
• The sitemap-index recursion so the workflow works for multi-sitemap sites without rewriting.
• The "submitted but quota exceeded" retry queue so a bad day doesn't lose URLs.
• The age filter so you don't waste quota on pages already indexed for 2 years.

Each of these came from running the workflow at scale and watching it fail in a specific way. The version you get already has each fix baked in.

What the two Google APIs cost

Per run: free, but capped. The Google Indexing API quota starts at a default of 200 requests meant for onboarding and testing, and Google must approve anything beyond it. URL Inspection allows 2,000 queries a day per site. n8n hosting if you self-host: about $5/month on a small VPS.

There's no separate build fee. Indexing checks are one of the systems covered by execution capacity, the flat $2,000 a month for AI agents and automations that sits on top of a fractional Head of Content engagement. Setup takes 1-3 days, mostly Google Cloud service-account plumbing and sheet wiring.

Running on a schedule within a week

Bring your sitemap URL and GSC access to a call and we'll create the service account together. It can be running on a schedule by the end of the same week.