Reverse Engineering Competitor Anchor Text Profiles with Python and BeautifulSoup

The myth that anchor text is dead persists among the lazy. For anyone who has actually parsed a backlink profile at scale, anchor text distribution remains one of the most revealing signals of a competitor’s link building strategy, content targeting, and even their penalty risk. Manual inspection using free, open-source tools is not only possible but often superior to black-box SaaS reports because you control the parsing logic, the weighting, and the deduplication. You can uncover patterns that tools like Ahrefs or Moz abstract away, such as partial match ratios, branded vs. unbranded semantic drift, and contextual co-occurrence across different linking domains. This is the kind of reverse engineering that separates the script kiddies from the people who actually understand link equity flow.

Start by harvesting the competitor’s backlinks. Majestic’s free tier still gives you a CSV export of up to 1,000 referring pages, which is enough for a small to mid-size competitor. Alternatively, if your target site has fewer than 500 referring domains, you can manually scrape the “link:“ operator in Google (with some creative query parameter manipulation) or use the Open PageRank API’s free endpoint. Whatever the source, you want a flat table of referring URLs, target URLs, and the anchor text as it appears on the referring page. This is the raw ore.

Load that CSV into a Pandas DataFrame. Now the real reverse engineering begins. Do not just count exact anchors. That’s what beginners do. Instead, calculate TF-IDF scores for each anchor relative to the entire backlink corpus. A high TF-IDF anchor that appears on only a few high-authority domains signals a deliberate editorial link. A low TF-IDF anchor that appears hundreds of times across low-quality directories signals a spammy automated campaign. Use the `TfidfVectorizer` from scikit-learn with custom tokenization that strips punctuation but preserves URL fragments and hyphenated terms. This gives you a numeric fingerprint of a competitor’s anchor strategy that no free tool directly reports.

Next, segment anchors by the ratio of branded terms to generic terms. Define a set of branded tokens (the competitor’s domain name, company name, trademarks) and compute a brandedness ratio per referring domain. Plot the distribution. If you see a long tail of domains with near-zero brandedness, that is manual outreach gone wrong or a private blog network in disguise. If the brandedness is uniformly high, the competitor likely has a strong direct linking culture, possibly through guestographics or resource pages. Cross-reference this with the domain’s trust flow from Moz’s free API or the Open PageRank value. A domain with very few unique brands and high trust flow is almost certainly paying for link placements. That is a vulnerability you can exploit by offering more natural, contextually rich anchor diversity.

Then perform co-occurrence analysis. For each referring domain, extract all anchors pointing to different pages on the competitor’s site. Use BeautifulSoup to parse the HTML of the referring page and locate all links that share the same host as the competitor. Build a bipartite graph of referring domain to anchor phrase, then compute Jaccard similarity between different target pages. Pages that share many of the same anchor phrases across multiple domains indicate a topic cluster that the competitor is aggressively interlinking. You can reconstruct their content silo hierarchy purely from external anchor text. This is manual deep-linking reverse engineering without ever looking at their internal navigation.

Finally, map each anchor to its likely keyword intent. Use a pre-trained BERT model from Hugging Face (free) to classify anchors into informational, navigational, commercial, or transactional. Do this offline with a small batch. The distribution of intent across the backlink profile tells you whether the competitor is building links for brand awareness or direct conversion. If most anchors are transactional (e.g., “buy X”, “cheap Y”) but the linked pages are blog posts, their anchor profile is misaligned and likely underperforming. You can capitalize by building links with informational anchors to informational pages, matching user intent more cleanly.

All of this requires nothing more than a Python environment, a CSV, and about forty minutes of code. The output is a custom report ranking the competitor’s link sources by strategic value, not by raw quantity. You will know exactly which anchors to target in your own outreach and which to avoid at all costs. That is manual competitor analysis at its most surgical, built entirely on free data and your own parsing intelligence. The tools are just scaffolding; the insight comes from your willingness to look beyond the top-level numbers.