Automating Keyword Cannibalization Detection with TF-IDF and Cosine Similarity

The solo marketer’s nightmare is the recurring audit that takes five hours every month and yields the same vague conclusion: “some pages might be competing.“ Keyword cannibalization is a chronic, low-grade inflammation of your rankings. It does not scream like a broken sitemap or a 404 flood, but it silently divides your authority across analogous URLs. The manual process of exporting rank data, cross-referencing queries, and eye-balling titles is not only tedious but also error-prone. You need a deterministic, codified approach that treats this repetitive task as a data engineering problem. Build a script that ingests a crawl export, vectorizes the content, and outputs an overlap matrix ranked by semantic similarity and shared lexical terms. This is not theoretical; it is a lightweight pipeline you can run on a laptop or a scheduled GitHub Action.

Start with the data source. Your crawler, whether Screaming Frog, Sitebulb, or a custom Scrapy spider, can export URLs, titles, meta descriptions, and raw body text. The body text should be stripped of navigation boilerplate and HTML tags. Store each page’s clean content as a single document in a SQLite table or a Parquet directory. The filename is the URL slug. From there, compute TF-IDF vectors. The TfidfVectorizer from scikit-learn is perfect because it naturally handles the frequency-inverse document frequency weighting that separates a genuine topical focus from filler words. Set ngram_range to (1,2) to capture multiword entities like “voice search optimization” without exploding the feature space. Use min_df=2 to discard terms that appear on only one page, which would be noise for cannibalization detection. The result is a sparse matrix where each row is a URL and each column is a token or bigram.

The hard part is comparing those vectors in a way that scales. For a site with a few hundred pages, a dense pairwise cosine similarity matrix is trivially computed with cosine_similarity from scikit-learn. Five hundred pages yields 125,000 pairwise comparisons, a rounding error in RAM. But for a site with ten thousand pages, that is 100 million pairs, which becomes unwieldy. The scalable solution is to use approximate nearest neighbor libraries like Annoy or NMSLIB. You index the TF-IDF vectors and query each URL for its top twenty neighbors above a similarity threshold. This drops the computational complexity from O(n^2) to O(n log n). For a solo marketer, this is the difference between a script that completes in seconds versus one that times out at the end of your lunch break.

But raw cosine similarity is not enough. Two pages about “leather jackets” might share a high cosine score simply because they both discuss product materials, prices, and shipping policies. You need to intersect the top weighted terms for each pair. After vectorization, extract the top 10 tokens per URL based on their TF-IDF values. If two URLs share more than three of those top tokens and also have a cosine similarity above 0.8, flag them as potential cannibals. Why three? Because a single overlapping word like “free” or “guide” is too generic. Three indicates a meaningful topical overlap. The threshold is tunable. Run the script, examine a few output rows, and adjust the tradeoff between recall and precision. This is where your judgment as a marketer comes in, but the automation does the heavy lifting of proposing candidates.

Now operationalize it. Turn this from a one-off script into a scheduled job. Write a Python module that reads the latest crawl data, performs the analysis, and writes a CSV with columns for source URL, target URL, cosine similarity, and the list of overlapping keywords. The same script can also check each flagged pair for existing rel=canonical tags or meta robots directives by parsing the HTML of the source pages. This adds context: if you are already telling Google which URL to prefer, the cannibalization risk is lower. The output then becomes a decision-ready artifact. You no longer need to open a rank tracker and guess. You have a text file that lists exactly which pages need attention.

The real power emerges when you schedule this in a CI/CD context. A GitHub Action that runs on a weekly cron can pull the crawl data from your cloud storage, execute the analysis in a container, and post the results as a PR comment or a Slack notification. To take it further, add a regression test to your repository. The test asserts that no current production URL pair exceeds your similarity and overlap thresholds. If your content team accidentally publishes a near-duplicate page, the test fails and blocks the deployment. This turns cannibalization detection from a periodic audit into a preventive check. You are not just reacting to lost rankings; you are building a safety net that catches the problem before it hits search results.

The deeper principle here is that any repetitive SEO task is a candidate for abstraction. Manual audits do not scale because they rely on subjective pattern matching. A script, on the other hand, applies the exact same logic every time. It is deterministic, idempotent, and version-controllable. You can change the thresholds, tweak the tokenization, or add new signals like user intent classification from a transformer model. The initial time investment to write this script is about three hours. Given that you would spend at least that much on a single manual audit, the return on investment is immediate. And as your site grows, the script scales with zero additional effort from you. That is the definition of scalable processes for solo marketers. Embrace the nerdiness. Write the code, schedule it, and let the terminal do the grunt work while you focus on the strategic decisions that require a human brain.