How an AI SEO agency runs keyword research

An explainer on how an AI SEO agency runs keyword research end to end—turn inputs into expanded seed sets, classify intent from SERPs, score difficulty vs. opportunity, cluster into topics, and convert priorities into page maps and an operational workflow.

Sev LeoLinkedInTwitter

Sev Leo is an SEO expert and IT graduate from Lapland University, specializing in technical SEO, search systems, and performance-driven web architecture.

Research Inputs

Keyword research is only as good as the inputs you feed it. An AI SEO agency treats inputs like constraints, because constraints decide what keywords even count as “good.”

Business primitives

You start with business facts, because “traffic” is not a goal. A keyword is only valuable when it matches your offer and your economics, like “emergency plumber Austin” versus “how to fix a leaky faucet.”

Turn primitives into constraints:

• Offer → problems you solve, in buyer language
• ICP → roles, industries, deal size, pain severity
• Margins → max CAC, payback window, LTV tolerance
• Geography → countries, cities, service radius, languages
• Seasonality → peak months, budget cycles, weather spikes

The tighter these are, the fewer keywords you regret ranking for.

Site evidence

Your site already contains demand signals and hard limits. You mine first-party data to see what Google thinks you deserve.

• Search Console queries and pages
• Analytics landing pages and paths
• Server logs and crawl patterns
• Internal search terms and refinements
• Index coverage and exclusions

If coverage is weak, fix indexation before you expand targets.

SERP environment

Keywords live inside SERPs, not spreadsheets. You profile the current results to estimate effort, timeline, and the real intent behind the query.

Capture the SERP environment:

• True competitors, not just category peers
• SERP features stealing clicks, like maps or AI overviews
• Content formats winning, like tools or listicles
• Intent volatility across days and devices

If the SERP is unstable, your “perfect” keyword may be a moving target.

Risk guardrails

Guardrails prevent rankings that create legal risk or wasted work. You define what the model must avoid, before it suggests anything.

1. Flag YMYL topics and require expert review.
2. Block brand-unsafe themes and prohibited claims.
3. Apply compliance rules, like HIPAA, FINRA, or local ads law.
4. Prevent cannibalization with one intent per URL.
5. Set “no-target” lists for low-value or high-support queries.

The fastest keyword win is the one you never chase.

Seed Expansion Engine

Agencies win keyword research by building a huge candidate set, then making it clean enough to score. You want models for breadth, but you need deterministic sources for reality checks.

Seed generation

You need strong seeds or the model wanders into trivia. Seeds anchor intent around what you sell and what your pages already cover.

1. Extract product nouns and modifiers from your catalog and pricing pages.
2. List top pain points from sales calls, tickets, and reviews.
3. Translate jobs-to-be-done into verbs plus objects, like “track invoices”.
4. Pull H1s, titles, and internal search queries from existing pages.
5. Tag each seed with audience, stage, and primary intent.

Bad seeds create impressive lists that never convert.

Multi-source expansion

One source always lies to you in its own way. Mixing sources gives coverage, intent variety, and a built-in bias check.

• Ask the LLM for intent variations and modifiers
• Pull autocomplete and “people also ask”
• Collect related searches and SERP suggestions
• Import competitor ranking keywords by URL
• Map category taxonomy nodes to queries

If two or more sources agree, you probably found a real demand pocket.

Normalization layer

Raw lists are noisy, and noise breaks scoring. You normalize so “best ai crm”, “AI CRMs”, and “best CRM with AI” don’t fight each other.

Canonicalize with a ruleset plus light entity resolution:

• Standardize spelling and casing, like “ecommerce” vs “e-commerce”.
• Normalize plurals and tense when intent stays identical.
• Resolve entities and aliases, like “GA4” vs “Google Analytics 4”.
• Apply locale rules, like “accountant” vs “chartered accountant”.
• Remove stopwords only when they don’t change intent.

Normalization is where your scoring stops being random.

Deduplication rules

Expansion creates near-duplicates that look different but behave the same. You cluster them so you score one idea, not fifty spellings.

1. Embed each query and run cosine similarity clustering.
2. Add string similarity checks for short queries and typos.
3. Split clusters when SERPs differ by intent or page type.
4. Keep the representative with highest volume, links, or rankings.
5. Store the rest as synonyms for matching and reporting.

Your final list should be smaller than your ego, and stronger than your competitors.

Intent Classification

Intent is inferred from what Google rewards and how people phrase the query. When the SERP is full of “Top 10” lists, your product page won’t stick.

Intent drives both content shape and conversion fit. Rank with the wrong page type, and you’ll convert the wrong visitors.

Intent types

You classify intent by the job the searcher is trying to finish. The same keyword can carry two jobs, like “notion alternatives” which can be research or purchase-adjacent.

Informational queries want understanding, like “what is schema markup.” Commercial queries want options, like “best CRM for agencies.” Transactional queries want action, like “buy ahrefs” or “book a demo.” Navigational queries want a place, like “HubSpot login.”

Mixed intent is where money leaks. Treat it as a segmentation problem, not a category mistake.

SERP-as-label

The SERP already tells you what intent Google believes. You use those features as weak labels, then validate with traffic and conversion.

• Shopping ads, product grids, “Buy” buttons → transactional
• Listicles, “Top stories,” “People also ask” → informational
• Comparison pages, “Best,” review snippets → commercial
• Sitelinks, brand homepages, knowledge panels → navigational

If your target page type disagrees with the SERP, you’re fighting the grader.

Modifier signals

Modifiers are mechanical intent cues because they compress context into one word. “Best” often demands comparisons, while “pricing” demands numbers and terms.

“Near me” signals local pack eligibility and urgency. “Vs” signals evaluation, not education. “Pricing,” “cost,” and “quote” signal conversion readiness, even without a brand mentioned.

Modifiers don’t just change content. They change what you should measure—see this SEO guide for intent signals for how to translate those cues into page decisions.

Modifiers don’t just change content. They change what you should measure.

Ambiguity handling

Some queries stay unclear even after SERP review. You treat those as multi-intent candidates until evidence collapses the options.

1. Assign two intents with confidence scores, not one label.
2. Map each intent to a page archetype you can actually ship.
3. Check SERP stability across days and locations.
4. Run a split test with two page types or modules.
5. Keep the winner, and reclassify the query.

Ambiguity isn’t a blocker. It’s a prompt to test the page, not argue about the label.

Difficulty and Opportunity Scoring

Rankings correlate with three forces you can measure: authority, relevance, and the SERP’s shape. Authority decides if you’re allowed to compete, relevance decides if you deserve to compete, and SERP composition decides how many clicks exist to win.

If the page-one set is packed with brands, intent-matched formats, and sticky features, difficulty rises even when volume looks great. Score those forces consistently and your “hard” keywords stop being surprises.

Demand estimates

Search volume is a ceiling, not a forecast, so you estimate clicks instead of chasing ranges. You combine volume bands, position-based click curves, and SERP feature suppression to model what organic traffic is actually available.

Start with the midpoint of the volume range, then apply a click curve for your expected rank band. Reduce again for features that steal attention, like ads, local packs, and AI answers.

The number you want is “clicks you can capture,” not “searches people made.”

Competition signals

You score what’s already winning, because Google already told you what it trusts. The goal is to estimate your win-rate, not to argue with the SERP.

• Benchmark linking domains and anchor patterns
• Check topical authority across the whole domain
• Match the dominant content format and intent
• Verify freshness expectations and update cadence
• Note brand bias and SERP stability

When three signals stack against you, you’re buying a long timeline.

Gap detection

You look for competitor rankings you should own, then check if your site has a real home for them.

1. Export competitor keywords, filtered to relevant intents.
2. Cluster by offer, job-to-be-done, and funnel stage.
3. Map clusters to existing URLs and note weak matches.
4. Flag “no destination” clusters as net-new pages.
5. Flag “thin destination” clusters as rebuild candidates.

A keyword gap is usually an information architecture gap wearing a keyword costume.

Composite scoring

Single metrics lie, so you blend four numbers into one: demand, win-rate, value, and effort. Each input gets a tunable weight, because a startup and an enterprise do not have the same constraints.

A common internal shape is: Priority = (Demand × Win-rate × Value) ÷ Effort, then weights adjust each term. You can overweight win-rate when you need quick proof, or overweight value when sales capacity is tight.

Tune the weights to match your business, then let the math be boring and consistent.

Clustering into Topics

A raw keyword list is just potential energy. Clustering turns it into a page plan that avoids “five pages targeting the same thing.”

Your goal is simple. One intent, one page, with supporting content around it.

Embedding clusters

Embeddings let you group keywords by meaning, not matching words. That keeps “best CRM for startups” close to “startup CRM software,” even with zero overlap.

You run each query through a vector model, then cluster by cosine similarity.
You validate clusters by checking SERP overlap, because meaning without intent is how thin pages happen.

If two queries share a SERP, treat them like one job.

SERP overlap test

Use SERP overlap when the embeddings look “maybe.” It’s the fastest way to confirm if Google thinks two queries are the same.

1. Pull the top-10 organic URLs for query A.
2. Pull the top-10 organic URLs for query B.
3. Count shared URLs, ignoring sitelinks and tracking parameters.
4. Decide: merge if 3+ overlap, split if 0–1 overlap.
5. Sanity-check intent shifts like “template” vs “tool.”

If you see three or more shared URLs, you’re looking at one intent.

Canonical keyword choice

Each cluster needs one canonical query to steer the page. Without it, you’ll write a page that targets “everything” and ranks for nothing.

• Prefer the clearest intent signal.
• Prefer the highest CTR-shaped SERP.
• Prefer the query matching your product vocabulary.
• Prefer the query with linkable angle.
• Prefer the query with stable volume.

Pick the canonical keyword, then write the page title like you mean it.

Cannibalization prevention

Clusters only work if they map cleanly to URLs. Agencies assign a single “URL owner” per cluster, so two teams don’t publish competing pages.

They also set consolidation rules for legacy overlap.
Merge duplicates, 301 the weaker page, and keep one updated URL as the source of truth.

Coordination beats cleanup every time.

Prioritization Matrix

You need a way to choose between keyword clusters without defaulting to “highest volume wins.” A simple impact × feasibility matrix makes tradeoffs explicit and defensible.

You’re not picking keywords. You’re picking battles your site can win this quarter.

Page Mapping Blueprint

You need a one-to-one match between each keyword cluster and a page that can own it. Without that map, you get cannibalization, thin pages, and endless “quick fixes.”

1. Assign each cluster to one primary URL candidate and one page type.
2. Score the existing page against intent, depth, and internal-link fit.
3. Decide the action: update, merge, create, or leave alone.
4. Write a one-paragraph page brief: promise, sections, and primary entities.
5. Set a review trigger based on rank volatility and content decay signals.

If two clusters want the same promise, you don’t need two pages. You need one better page.

Operational Workflow

Keyword research only works when it keeps changing with your market. AI gives you speed and coverage, while humans protect meaning, risk, and business fit.

Treat it like a living system. Not a spreadsheet you finish once.

Roles and reviews

AI drafts the first pass because breadth matters, fast. Strategists review because the word “best” can mean research, not purchase.

AI proposes assumptions, intent labels, and preliminary scores for every cluster. Strategists approve or override three things: the intent call, the business promise, and the final prioritization. That’s the line between “rankable” and “worth ranking for.”

Feedback loops

Use performance data to update what you believe, monthly.

1. Pull rankings, clicks, conversions, and assisted conversions by URL.
2. Detect SERP shifts: new features, new competitors, rewritten titles.
3. Reweight scoring factors using outcomes, not opinions.
4. Refresh clusters and intent labels, then flag broken mappings.
5. Reprioritize the roadmap and queue new content tests.

If you don’t change the weights, you’re just collecting metrics.

Quality controls

AI moves fast, so your checks must be boring and strict.

• Flag hallucinated entities and unsupported claims.
• Detect brand conflicts, legal risks, and restricted terms.
• Merge or split duplicate clusters with overlapping SERPs.
• Catch intent vs page-type mismatches before briefs.

Fix the system first. Ideas are easy.

Reporting artifacts

You ship artifacts people can argue with, not vibes. A simple example: “We dropped ‘pricing’ keywords after low-converting demo traffic.”

Deliver four outputs: a keyword universe, a cluster map, a scoring rationale, and a changelog explaining priority moves. The changelog is the trust layer, because it shows your model learned something real—use an ultimate checklist for streamlining SEO content to keep those outputs consistent and audit-ready.

Turn the model into a repeatable system

AI-assisted keyword research works when every step produces an artifact you can audit: inputs and guardrails, an expanded and deduped universe, intent labels grounded in SERPs, and transparent opportunity scores. From there, clustering and cannibalization checks turn keywords into topic units, and the prioritization matrix and page map turn topic units into publishable decisions. Treat the workflow as a loop—ship pages, measure outcomes, feed wins and misses back into the next round—so your research gets sharper every sprint.