Automating Keyword Clustering with TF-IDF and Cosine Similarity in Python

If you are a solo marketer scraping your way through hundreds of long-tail keywords every month, you already know the bottleneck is not discovery—it is categorization. The manual labor of sorting “SEO audit tools for small business” from “SEO audit checklist for startups” is the kind of drudgery that kills momentum. You need a vectorized, math-backed approach that turns a terminal into a clustering engine. Python provides the entire stack for free, and if you are comfortable with pandas, you can build a keyword clustering pipeline that rivals enterprise tools costing hundreds per month.

The core idea is simple: treat each keyword as a document, measure its semantic similarity to every other keyword using term frequency–inverse document frequency (TF-IDF) vectors, and then group them via cosine distance. No API calls, no monthly subscription, just your own CPU and a little elbow grease. Start by collecting your keyword list into a plain CSV with a single column. Load it into a pandas DataFrame, then use `TfidfVectorizer` from scikit-learn to convert the raw strings into a sparse matrix of numerical features. The vectorizer strips common stop words, applies n-grams (bigrams are usually sufficient for SEO phrases), and normalizes term frequencies. The output is a matrix where each row represents a keyword and each column represents a term or n-gram weighted by its rarity across the entire corpus.

From that matrix, compute a cosine similarity matrix using `cosine_similarity` from scikit-learn’s pairwise module. This produces a symmetric matrix where entry `[i][j]` is a float between 0 and 1 representing how semantically close keyword i is to keyword j. A value of 1.0 means the vectors are identical; 0.0 means orthogonal. Now you need to decide where to draw the line. Experiment with a threshold between 0.5 and 0.8, but remember that SEO intent often benefits from looser clusters—a 0.4 threshold might capture “best CRM for startups” alongside “top small business CRM” while filtering out “CRM implementation guide.” For production, iterate the threshold against a validation set of hand-labeled keywords.

With the similarity matrix and a threshold in hand, run a greedy clustering algorithm. One clean hack is to treat the matrix as an adjacency list and use networkx to build a graph, then extract connected components. Scikit-learn’s `DBSCAN` also works well because it does not require specifying the number of clusters beforehand—it takes `eps` (the distance threshold, which is 1 minus your cosine similarity) and `min_samples` (set to 1 to catch singleton keywords). DBSCAN will tag each keyword with a cluster ID, and outliers get -1. That is perfect for flagging orphan keywords that deserve their own content pillar or need rephrasing.

After clustering, write the results back to a CSV with columns: keyword and cluster_id. You can then sort by cluster ID and inspect the groups. A typical output might show “on-page SEO checklist for eCommerce,” “on-page SEO audit steps,” and “on-page SEO factors Google” all falling into cluster 17, while “technical SEO checklist for migration” lands in cluster 42. This lets you plan pillar pages, topic clusters, and internal linking strategies without manually reading a thousand rows.

The real power emerges when you chain this pipeline. Write a Python script that accepts a path to your keyword CSV, runs the clustering, and dumps a neatly grouped spreadsheet. Run it weekly after pulling new keywords from Google Search Console or Ahrefs exports. You can even embed it into a cron job or a lightweight GitHub Action that commits the updated cluster map to a repo. Solo marketers often forget that automation is not just about doing tasks faster—it is about reducing cognitive load. By offloading the grouping logic to a blend of linear algebra and graph theory, you free your brain to interpret the clusters, spot content gaps, and prioritize creation.

A few tweaks elevate this from a toy script to a production-grade tool. First, preprocess the keywords with lemmatization using `spaCy` or the lighter `nltk` to conflate “optimizing” and “optimization.” Second, feed the TF-IDF vectors into an agglomerative clustering algorithm instead of DBSCAN if you prefer dendrograms and manual cluster cuts. Third, add a label extraction step: after clustering, find the most representative keyword in each cluster by selecting the one with the smallest average distance to all other members. That becomes the cluster’s anchor term for your content brief. None of these steps require a credit card—only hours of focused coding, which a solo marketer with technical chops already loves to invest.

In a world where SaaS tools charge $99/month for “AI keyword clustering,” building your own in Python is not just frugal; it is a strategic advantage. You own the pipeline, you control the threshold, and you can extend it to include search volume, click-through rate, or even topic model overlays from Latent Dirichlet Allocation. The only limit is your imagination and your ability to write clean, modular code. Start with a single CSV, a Jupyter notebook, and the three imports: pandas, sklearn, and networkx. The clusters will surface patterns you never saw with your naked eyes, and the time saved will let you focus on the one thing software cannot do—crafting content that converts.