Screaming Frog SEO Spider

Screaming Frog SEO Spider is a website crawler that allows you to crawl website URLs and extract critical information for analysis and auditing technical and onsite SEO. Download for free, or get a license for more sophisticated features.

The SEO Spider app is a robust and versatile site crawler that can crawl both tiny and extremely large websites fast while allowing you to analyze the results in real time. It collects vital onsite data to help SEOs make informed judgments. 

What can you do with the SEO Spider Software?

Find broken links. 
Crawl a website instantly to detect broken links (404s) and server issues. Export the errors and source URLs in bulk for fixing or sending to a developer. 

Audit Redirects 
Find temporary and permanent redirects, identify redirect chains and loops, or upload a list of URLs to audit during a website transfer. 

Analyze page titles and meta data. 
During a crawl, examine page titles and meta descriptions to find those that are too long, short, missing, or repeated throughout your site. 

Discover Duplicate Content. 
Using a md5 algorithmic check, identify exact duplicate URLs, partially replicated parts such as page names, descriptions, or headings, and low content pages. 

Extract data using XPath 
Collect data from a web page's HTML using CSS Path, XPath, or regex. This might include social meta tags, additional headings, prices, SKUs, and more! 

Review Robots and Directives. 
View URLs that have been restricted by robots.txt, meta robots, or X-Robots-Tag directives like 'noindex' or 'nofollow', as well as canonicals, rel="next" and rel="prev". 

Generate XML Sitemaps 
Screaming Frog allows you to quickly build XML Sitemaps and Image XML Sitemaps, with advanced URL configuration options such as last changed, priority, and change frequency. 

Integrate with Google Analytics 
Connect to the Google Analytics API to retrieve user data, such as sessions, bounce rate, conversions, goals, transactions, and revenue for landing pages, from the crawl. 

Crawl JavaScript websites. 
Render web pages with the integrated Chromium WRS to crawl dynamic, JavaScript-rich websites and frameworks like Angular, React, and Vue.js. 

Visualize Site Architecture. 
Internal linking and URL structure can be evaluated using interactive crawl and directory force-directed diagrams, as well as tree graph site visualizations. 

Features and Highlights

• Find Broken Links, Errors & Redirects
• Analyse Page Titles & Meta Data
• Review Meta Robots & Directives
• Audit hreflang Attributes
• Discover Duplicate Pages
• Generate XML Sitemaps
• Site Visualisations
• Crawl Limit
• Scheduling
• Crawl Configuration
• Save Crawls & Re-Upload
• Custom Source Code Search
• Custom Extraction
• Google Analytics Integration
• Search Console Integration
• Link Metrics Integration
• Rendering (JavaScript)
• Custom robots.txt
• AMP Crawling & Validation
• Structured Data & Validation
• Store & View Raw & Rendered HTML

Note: Limited to crawling a maximum of 500 URLs in demo version.