How to Run a Technical SEO Audit: Tools and Setup

A step-by-step guide to running a technical SEO audit that produces developer-ready proof — Domain property verification and a pre-audit access checklist, Screaming Frog setup for a Google-like crawl, URL Inspection and performance validation to confirm what’s actually rendered, robots.txt/sitemaps/Crawl Stats checks, and a findings-to-tickets table so nothing gets lost in translation.

Sev LeoLinkedInTwitter

Sev Leo is an SEO expert and IT graduate from Lapland University, specializing in technical SEO, search systems, and performance-driven web architecture.

Pre-audit access

Before you crawl anything, lock down scope and access so every finding is verifiable. In seo technical work, crawlability (whether a crawler can fetch a URL and its critical resources) and indexability (whether the page is eligible to be stored in a search engine’s index, often controlled by directives like noindex and by crawler access) are the two gates you’re testing—without access, you can’t prove which gate is failing.

If you’re budgeting headcount, decide up front who owns each access-dependent step: adding a DNS TXT record, exporting Search Console data, pulling raw server logs, and approving robots.txt/sitemap changes. A technical audit stalls when the crawl is ready but the last permission (or approver) isn’t.

Verify Domain property (Google Search Console)

1. Create a Domain property in Google Search Console (covers all protocols and subdomains).
2. Generate the verification token and add it as a DNS TXT record with your DNS provider.
3. Confirm ownership once DNS propagates, then grant access to the people who will export data and approve fixes.
4. If you also need a tighter view (e.g., one host/path), add a URL-prefix property too—but treat Domain as your source of truth.

Access checklist (so tests don’t lie)

• Server logs (raw request data) to confirm what bots and users actually hit.
• CMS/admin access (or a dev contact) to inspect templates, canonicals, and metadata at the source.
• CDN/WAF controls so your crawler and Google aren’t rate-limited or blocked.
• Staging/auth areas credentials and rules, so you don’t crawl a login wall by accident.
• DNS access (or the owner) for Search Console verification and emergency changes.
• A named approver for robots.txt / sitemap updates—because a robots.txt block is not a reliable deindexing method, and Google can’t see a noindex on a page it’s blocked from crawling.

Crawl that matches Google

A technical audit is only as good as the crawl you base it on. Your “source of truth” crawl should mirror how Google actually discovers and evaluates pages, or you’ll waste time ticketing issues that never affect indexing.

Start by matching mobile-first indexing—Google uses the mobile version of your content as the primary version for indexing and ranking, crawled with a smartphone agent. That means your crawl should use a smartphone user-agent and should reflect your real robots and authentication rules.

Next, decide whether you need JavaScript rendering—a crawl mode where the tool executes client-side JavaScript to evaluate the rendered DOM, not just the raw HTML response. If your navigation, internal links, canonicals, or key content are injected by JavaScript, an HTML-only crawl can miss (or misread) what Google ends up rendering.

Robots settings are the line that most often gets crossed. A robots.txt Disallow is not a reliable way to keep a URL out of search results; Google can still show the URL (often without a snippet). And if Googlebot can’t fetch a page, it can’t see a noindex directive on that page either—so a “blocked + noindex” setup does not behave the way people assume. For your primary dataset, crawl with robots respected unless your explicit goal is to diagnose what robots is hiding from your tooling (in which case: label that crawl as diagnostic and keep it separate).

Screaming Frog setup

1. Pick your crawl mode: start with HTML; switch to JavaScript rendering if your site relies on client-side rendering for links/content.
2. If using JavaScript rendering, know what it’s doing: Screaming Frog renders pages in a headless Chrome (Chromium-based) browser and extracts data from the rendered HTML.
3. Set user-agent to smartphone: choose a Googlebot Smartphone user-agent so your crawl matches mobile-first reality.
4. Decide robots behavior (and record it): keep robots.txt respected for the “Google-parity” crawl; only ignore robots.txt for a separate diagnostic run.
5. Handle authentication intentionally: either configure credentials so you crawl real content, or exclude login-walled areas so you don’t fill your dataset with redirect loops and blocked templates.
6. Limit scope before you hit Start: constrain to the intended host/subfolder and exclude known traps (logout URLs, infinite parameter spaces).

Note the practical constraint: the free version of Screaming Frog SEO Spider is limited to 500 URLs per crawl, so plan your scope accordingly and export/snapshot your settings so the dataset is reproducible—and consider bookmarking resources to simplify SEO workflows to streamline repeatable audits.

Prove what Google sees

Your crawler is where you find problems; Google-native tools are where you prove them. Before you write a ticket for “blocked,” “not rendered,” or “missing structured data,” confirm the issue shows up in the same surfaces Google uses to fetch, render, and interpret pages.

URL Inspection workflow

Use Search Console’s URL Inspection tool to validate a specific URL the way Googlebot experiences it, including a rendered view (with a screenshot) and the ability to inspect loaded resources and JavaScript output.

1. Open URL Inspection → “Test Live URL”. This checks the current fetch, not an older cached crawl.
2. Open the rendered view (“View tested page”). Save the screenshot as evidence when a layout, content block, or navigation element is missing.
3. Compare raw vs rendered HTML. If your crawler flags missing titles/canonicals/links, confirm whether they’re absent in rendered output or only in the initial HTML.
4. Review loaded resources + JS output. When CSS/JS files are blocked, erroring, or timing out, this is where it shows up clearly.
5. Attach artifacts to the ticket. Include the rendered HTML snippet (or the relevant diff), the screenshot, and the failing resource URL(s) so the fix is unambiguous.

For structured data issues, cross-check with the Rich Results Test, which supports JSON-LD, RDFa, and Microdata—so you can tell whether the problem is markup format, missing required properties, or Google simply not seeing the rendered markup.

Performance validation

CrUX (Chrome User Experience Report) is aggregated, real-user performance data used by multiple Google performance surfaces. Core Web Vitals are a defined set of UX performance metrics (including LCP, INP, and CLS) reported with thresholds.

If your crawl says “slow” but CrUX is clean (or the reverse), treat it as a data mismatch to resolve, not an argument—your backlog should reflect what Google can measure at scale.

Robots, sitemaps, crawl stats

1. Verify robots.txt is reachable and complete
   Request https://yourdomain.com/robots.txt directly and confirm it returns a valid robots.txt file (it can be populated or empty) or a 404.

• Google requests robots.txt frequently, and if it can’t get an acceptable response it will slow or stop crawling until it can.
• Keep the file under 500 KiB; anything after that limit is ignored, which can silently drop critical rules.

2. Confirm you’re using the right control for the job
   robots.txt is a host-level file that controls what paths crawlers are allowed to fetch; it does not reliably keep a URL out of search results by itself.
   If your goal is removal from the index, use noindex—an indexing directive (meta tag or HTTP header) that prevents indexing when crawlers can access the page and see the directive—often delivered via the X-Robots-Tag response header.

3. Validate XML sitemap constraints (and basic hygiene)
   Open the XML sitemap URL(s) and confirm the XML is readable and uses absolute URLs (not relative).

• Keep each sitemap within 50,000 URLs and 50MB uncompressed.

4. Read Crawl Stats as a “crawl control” diagnostic, not a vanity chart
   In the Search Console Crawl Stats report, look for patterns that line up with crawl controls:

• robots.txt fetch problems (these can precede a crawl slowdown)
• shifts in response codes (more 4xx/5xx)
• sudden drops in total crawling

5. Classify the failure: blocked vs overloaded
   Treat it as blocked when robots.txt fetching is failing or a new Disallow is in place. Treat it as overloaded when Crawl Stats shifts toward server errors/timeouts. That classification decides whether you fix rules (robots/sitemap) or infrastructure (capacity/WAF/CDN settings) first.

Turn findings into tickets

SEO enhancements are the concrete changes that remove crawl/index/render/performance blockers so Google can reliably fetch, understand, and evaluate your pages. ChatGPT can turn exports into draft ticket text (repro steps, suggested owners, acceptance checks), but it can’t replace the evidence you collect from your crawl and Google’s own views. Apply the 80/20 rule in SEO by prioritizing the few fixes that eliminate whole classes of issues (template-level, robots/indexing controls, recurring performance regressions) instead of polishing one-off URLs.

After fixes ship, Skribra’s Search Console-based monitoring and refresh workflow can flag post-change drops and route affected pages back into this same ticket format—especially when aligned to your broader SEO guide and best practices.

Audit what Google can verify

A technical SEO audit stops being a generic checklist the moment you treat it as a controlled test of two gates: crawlability and indexability—and you only get trustworthy answers when you have the access to prove which one failed. Set up your “source of truth” crawl to match Googlebot Smartphone, your real robots/auth rules, and—when the site needs it—JavaScript rendering, then separate any robots-ignoring run as diagnostic so you don’t contaminate the dataset. Before anything becomes a dev ticket, validate it in Google’s own views (URL Inspection’s live test, rendered HTML/resources, and screenshots; Rich Results Test; field vs lab performance) so you’re fixing what Google can actually fetch, render, and measure. Your first move is simple: secure Domain property + logs/CMS/CDN access, then run one reproducible Google-parity crawl and attach its artifacts to every ticket you ship.