Screaming Frog is a web crawler software that is designed to scan websites and extract vital information for search engine optimisation purposes. It is commonly used to crawl websites and gather data to be indexed by search engines like Google.
Before delving into the world of Screaming Frog, it is important to understand the basics of the software and its main menu options.
Here is a quick overview of the main navigation menus and settings:
File
The File menu option allows you to save a crawl as a file so that you can access the crawl data at any time. You can also access the last six crawls that you’ve performed with Screaming Frog. Additionally, you can use this menu to configure default settings for the application.
Configuration
The Configuration menu is perhaps the most important menu option in Screaming Frog. Here, you will find all the crawl settings that you need to configure for your crawl. Click on “Spider” to customize the type of content you want to crawl, and mark the data you want to see. You can choose to include or exclude images, JavaScript, CSS, HTML, subdomains, and other specific content types from your crawl. To further customise your crawl, click on “Include & Exclude” to add or remove specific site addresses. If you want to integrate your crawl with Google Analytics or Google Search Console, click on “API.”
Bulk Export
If you want to quickly export your crawl data, the Bulk Export menu option allows you to select specific response codes to export, such as 404 errors, links, text, images, and more.
Reports
In the reports menu, you can download reports on data such as redirects, errors, canonical tags, and more.
Sitemaps
The sitemaps menu option allows you to build a sitemap for your website.
What else can be done with Screaming Frog?
- Finding duplicate titles – as you probably already know, one of the most common problems we encounter during site optimization is duplicate titles, which can ultimately harm rankings. Screaming Frog provides a perfect solution – to identify these duplicates, go to the Page Title tab, filter the table in descending order, and find on which pages the title appears more than once.
- Finding broken links – as part of site optimisation, we exchange links to strong sources on the network and even to internal pages on the site itself, but usually, we have no way to verify if they are not broken. Here, Screaming Frog offers us a simple and quick solution, all you have to do is click on the External tab and set the view of the columns according to status.
- HTTP status codes – if you look under the Response Codes tab, you can see the http status codes of different pages on the site. Search for general problems on the site – if you want to get more general data about the site, through the Reports menu you can get various useful data such as finding titles that are too long, pages that do not have descriptions, heavy images and multimedia, and more.
- Meta description analysis – similar to titles, with Screaming Frog you can check that there are no duplicates in the page descriptions, that they all include keywords and that they are the correct length.
- Finding insecure content – Screaming Frog allows us to check the security of the site and whether it is protected from malicious actors. If you go to Reports, and from there to Insecure Content, you can see a detailed report that displays all the content (whether images, CSS, JavaScript snippets, and more) that may be exposed to hacking.
- Preview of the site in Google (SERP) – one of the most wonderful things about Screaming Frog is that you can see how your site pages appear in Google search results. This way you can verify if the title and meta descriptions look good, if there is a need to sharpen or change them, if keywords need to be added, and more.
- Checking internal links – we all know that internal links are an integral part of site optimization and are one of the powerful tools at our disposal. With Screaming Frog, you can find all the internal links on the site and check various data such as – matching the anchor to the text on the page, or finding broken/incorrect links or those linking to a redirected page.
In conclusion, Screaming Frog is a powerful web crawling tool that can be used for a wide range of purposes, including SEO, website optimisation, and more. By understanding the main menu options and settings, you can begin to explore the full potential of Screaming Frog and improve your website’s online visibility.