Hunting Soft 404s Using the URL Inspection API and Custom Regex Filters

Soft 404s are the silent killers of crawl budget. A page that returns a 200 status code while serving a “no results found” or “content unavailable” message tricks Googlebot into indexing a dead end, wasting your crawl allocation and diluting index quality. Most SEOs know the problem, but the standard fix—begging a developer to audit every path or implement a proper 410—is a non-starter in lean teams. The savvy alternative? Weaponize the URL Inspection API and your own regex patterns to detect these impostors at scale, all without touching a single line of backend code.

Google Search Console’s URL Inspection API is a free, RESTful endpoint that exposes per-URL data like indexing status, crawl date, and the actual rendered content (via the `inspectionResult.crawledAs` field). To use it, you need a Google Cloud Project with the Search Console API enabled and a service account authorized for your property. That’s a one-time setup, and the entire process lives in a Python script or even a Google Sheets Apps Script you can run on a cron. No developer required—just a marketer who can copy‑paste a few OAuth snippets.

The real power comes when you combine the API with regex filters to surface the telltale patterns of soft 404s. Start by exporting your site’s URL list from a crawl tool (Screaming Frog, Sitebulb) or directly from GSC’s Performance report. For each URL, call the API’s `urlInspection.index` method. The response includes the `indexStatusResult.verdict` (PASS, PARTIAL, FAIL) and, crucially, the `inspectionResult.sitemap` and `inspectionResult.crawledAs`. But the gold is in the `loadingStrategy` and `resourceLoadStatus`? Not quite. The rendered HTML snapshot isn’t returned via the API directly—you’d need to pull it via a headless browser—but you can infer soft 404 behavior from metadata anomalies.

A better approach: use the API to detect URLs that Google has marked as “crawled but not indexed” (the `indexStatusResult.coverageState` of `Crawled - currently not indexed`). Cross‑reference that list with your own regex pattern for known empty‑state pages. For example, e‑commerce sites often have category paths like `/category/?q=empty` or product pages that show “This product is no longer available” but return a 200. Write a regex that matches those path patterns—`/search/.(?:no-results|0-found)`, or `\/product\/.\?status=discontinued`—and filter your API results. Any URL that matches the pattern and has a “not indexed” verdict is a strong candidate for a soft 404.

But don’t stop at patterns. The API also reveals the last crawl date and the HTTP response code. If a URL returns a 200 but the crawl date is old (say, over 30 days) and Google hasn’t re‑crawled it, that’s a signal the content is considered low‑value. Combine with response status by checking the `inspectionResult.crawledAs.httpStatusCode`. When that’s 200 yet the rendering shows error text—well, you need the actual content. Here’s where a headless Chrome tool like Puppeteer or Playwright (both free, both scriptable) can be called from the same Python script to grab the rendered DOM. Check for strings like “404”, “not found”, “empty”, “no results”, and flag the URL. That’s still a DIY job, not a developer ticket.

The true crackerjack move: use regex to match the absence of expected content. For a blog, every post should contain an article body; if the DOM text length is under 100 characters, it’s likely a soft 404. For a product page, check for the string “add to cart” presence. Regex across the full HTML? Not ideal—use a simple character count and keyword presence. But for path‑level filtering, regex reigns. Write a pattern like `\/category\/[^\/]+\/$` for all category pages, then sample a few from the API’s “not indexed” set to manually verify if they’re actually empty. Automate the sampling by exporting to a CSV and using `grep` or even Excel’s `FILTER` function.

The bottom line: you don’t need a developer to fix soft 404s. You need a regex mindset, a free API key, and a few lines of Python to call the URL Inspection endpoint. Once you’ve identified the offending URLs, implement either a proper 410 or a meta noindex with a blank response—both can be done via `.htaccess` redirect rules or a CMS snippet you can write yourself. Yes, you’ll need server access, but that’s often already in the marketer’s hands via hosting panels or Git push rights. No code review, no dev sprint.

Start small: pull your top 500 URLs with the most crawl errors from GSC’s “Pages” report, run them through a Python script that hits the API and checks for your regex pattern of empty‑state paths. Patch the first twenty with 410s. Then measure your crawl budget recovery in GSC’s Crawl Stats report. The shift from “crawled, not indexed” to “indexed” or at least “not found” will confirm the fix. And you own the entire process—no dependency, no excuses.