Screaming Frog SEO Spider is the single most important tool in any technical SEO toolkit. I’ve used it on every client engagement since 2014. Here is everything you need to know to get started — and the features most beginners miss.
What Is Screaming Frog?
Screaming Frog SEO Spider is a desktop website crawler that analyses URLs in real time. It crawls your site the same way Google does — following links, reading HTML, and collecting data on every page it finds. The free version crawls up to 500 URLs. The paid licence (currently around GBP 239 per year) removes that limit and unlocks advanced features like JavaScript rendering, custom extraction, and crawl scheduling.
Installing and Setting Up
Download Screaming Frog from screamingfrog.co.uk and install it — it runs on Windows, Mac, and Linux. Before your first crawl, I recommend adjusting a few settings. Go to Configuration > Spider and tick “Check External Links” and “Crawl Outside of Start Folder” (only if you want to check subdomains). Under Configuration > Speed, set the max threads to 2–5 for your first crawl to prevent overloading your server.
Running Your First Crawl
Enter your homepage URL in the top bar and click Start. Screaming Frog will begin crawling your site page by page, following every internal link it finds. As it runs, you’ll see data populating in real time across multiple tabs: Internal, External, Response Codes, Page Titles, Meta Descriptions, Headings, Images, and more. Each tab gives you a different lens into your site’s health.
The Key Tabs Every Beginner Should Check
Response Codes — Filter by “Client Error (4xx)” to find broken pages. Any 5xx errors indicate server problems needing immediate attention. This tab is also essential for a broken link audit.
Page Titles — Filter by “Missing”, “Duplicate”, or “Over 60 Characters” to quickly identify title tag issues across your entire site. Title tags remain one of the strongest on-page ranking signals.
Meta Descriptions — Filter by “Missing” or “Duplicate”. While meta descriptions don’t directly affect rankings, they heavily influence click-through rates from search results.
H1 Headings — Filter by “Missing” or “Duplicate”. Every indexable page should have exactly one unique H1 that includes your target keyword.
Images — Filter by “Missing Alt Text” to find images without alt attributes, which are important for accessibility and image SEO.
Advanced Features Worth Learning Early
Custom Extraction — Under Configuration > Custom, you can set up XPath or CSS selectors to extract specific data from every page. I use this to check for structured data, verify tracking scripts, or confirm that certain elements exist across thousands of pages.
JavaScript Rendering — If your site uses JavaScript frameworks (React, Angular, Vue), enable JavaScript rendering under Configuration > Spider > Rendering. This tells Screaming Frog to render pages like a browser rather than reading raw HTML — critical for auditing modern web applications.
Crawl Comparison — Save your crawl data and compare it against future crawls to track changes over time. This is invaluable for monitoring the impact of site migrations, redesigns, or large-scale SEO changes.
How I Use Screaming Frog on Client Engagements
Screaming Frog is the first tool I open on every new engagement. A full crawl gives me a comprehensive baseline of site health within minutes. I combine crawl data with Google Search Console, Google Analytics, and backlink data to build a complete picture of where the site stands and where the biggest opportunities lie. Monthly re-crawls then track progress and catch new issues before they compound. If you want to understand how this fits into a full audit process, read my guide on how to conduct a technical SEO audit.