Sometimes as a business owner learning SEO, you need to crawl your website to understand where your largest areas of improvement lie, how the search engines are accessing your site, or to identify gaps in your content by crawling a competitor’s site. While crawling and auditing your log files is just one of many tactics used by businesses and professional SEOs to understand their sites, it’s an important one that no good SEO program can go without.
To that end, we’ve compiled the best website crawling and auditing tools that you should investigate using to improve your website’s organic traffic from SEO.
Table of Contents
Semrush
Semrush is a leading SEO and digital marketing tool that is our tool of choice here at Credo, which is specifically why we list them at the top here.
Semrush’s Site Audit tool is powerful with filters, scheduled crawls, and more that keep you on top of your site’s health and help you identify and fix problems on the go.
They also have a wide array of other SEO and digital marketing tools within their toolkit, including:
- Keyword research tools (including their magical Keyword Magic tool)
- Backlink Analytics
- Competitor research tools
- Link building tool
- Keyword and Backlink Gap tools
- And more
Check them out with a free 30 day trial
ScreamingFrog
Screaming Frog is most likely the most popular SEO website crawling and auditing tool available online.
Built by the team at Screaming Frog, an agency based in the UK, this SEO spider runs off your own machine and can be very powerful when used with custom filters. You can crawl as different user agents, exclude or include robots.txt, and so much more.
With the free tier you can crawl up to 500 URLs. On the paid tier (just £149 per year) you can crawl unlimited URLs, schedule crawls, crawl as different user agents, crawl Google AMP URLs, and so much more. Plus they’ve updated the interface and it’s now more beautiful and functional than ever.
In fact, SEER Interactive wrote a phenomenal guide to it.
Other great write-ups on Screaming Frog also exist such as How To Run Screaming Frog on AWS by iPullRank and Screaming Frog Indexing Tips and Tricks over on Authority Labs.
Visit: http://www.screamingfrog.co.uk/seo-spider/
Botify
Botify is a relatively new tool on the market.
Different from ScreamingFrog, it is a subscription based model that gets you access to their SaaS (Software As A Service) tool that crawls your site from the cloud (aka not on your personal machine, so you can do other things as your crawl happens) and gives you in-depth reports with clear actions to take next on your site.
Botify markets itself as “a leading enterprise SEO platform”, which means that it is built specifically for enterprise in-house SEO teams (and their agencies) to track changes over time and help drive reporting and actions to improve organic traffic.
Their Intelligence tool also helps you “prioritize your SEO tasks by greatest impact,” which means you’ll work more often on the right things that will move the needle on organic traffic.
Visit: https://www.botify.com/
DeepCrawl
DeepCrawl is our enterprise crawling tool of choice.
While it is more expensive than the above two mentioned SEO crawling tools, DeepCrawl goes deep into your site and provides you the recommendations you need to help you prioritize your technical SEO fixes for your site. Pricing is based on the number of pages you crawl each month and can be dialed up and down as needed.
I’ve used this on sites with 10 million+ pages and it worked a charm and took our site’s SEO to the next level. Highly recommended.
Visit: https://www.deepcrawl.com/
Splunk
Splunk is actually an IT and server admin tool, but it can be very powerful when applied to your log files for SEO.
Using Splunk, you can identify how the search engines are actually crawling your website and make optimizations from that data. It’s not a cheap tool when used for the full power, though there is a free version that can get you started. If you are working on any level of enterprise website, though, it could be worth your money.
A great post was written here on Moz about using it for Technical SEO.
Visit: http://www.splunk.com/
Moz
If you’re in the SEO world, you know Moz. They have the popular Moz blog and put on their annual Mozcon conference (one of the best conferences each year in the digital marketing world, in my opinion). Their Pro subscription allows you to set up not only rank tracking and other diagnostics for your site and some competitors, but it also gives you a weekly crawl of your site, delivered to your inbox, that shows you the issues they found on your site so you can prioritize the fixes.
Visit: https://moz.com/products/pro
These are the tools that we’ve used and recommend, but there are others such as Ryte (formerly OnPage.org), OnCrawl, and URL Profiler that could have been mentioned as well. We just haven’t used those as extensively.
The list is short. I would definitely add http://www.oncrawl.com which has great semantic features. And I would add open source solutions like http://nutch.apache.org/ that can be hacked to have a customized audit tool.
The list purposefully short because I want others to give their opinions on their favorite tools 🙂
John, you nailed it there’s a time and place for everyone of them.
However I find myself using deepcrawl for almost everything. Every SEO should have at least one of the great product listed. You really can’t lose with any of them.
I’ve used Screaming Frog a lot. Moz’s crawling tool is ok, not the most robust, but works. Never heard of Splunk or Botify, but I’ll have to check those out! I would have loved to use a tool like DeepCrawl for the enterprise sites I used to work on…wish I’d known more about it at the time!
-Rachel