Hear about new posts as they happen. Improve your marketing.

Sometimes as a business owner learning SEO, you need to crawl your website to understand where your largest areas of improvement lie, how the search engines are accessing your site, or to identify gaps in your content by crawling a competitor’s site. While crawling and auditing your log files is just one of many tactics used by businesses and professional SEOs to understand their sites, it’s an important one that no good SEO program can go without.

To that end, we’ve compiled five great website crawling and auditing tools that you should investigate using to improve your website’s organic traffic from SEO.



Screaming Frog is most likely the most popular SEO website crawling and auditing tool available online. Built by the team at Screaming Frog, an agency based in the UK, this SEO spider runs off your own machine and can be very powerful when used with custom filters. You can crawl as different user agents, exclude or include robots.txt, and so much more. In fact, SEER Interactive wrote a phenomenal guide to it. Other great writeups on it exist such as How To Run Screaming Frog on AWS by iPullRank and Screaming Frog Indexing Tips and Tricks over on Authority Labs.

Visit: http://www.screamingfrog.co.uk/seo-spider/



Botify is a relatively new tool on the market. Different from ScreamingFrog, it is a subscription based model that gets you access to their Saas (Software As A Service) tool that crawls your site from the cloud (aka not on your personal machine, so you can do other things as your crawl happens) and gives you indepth reports with clear actions to take next on your site.

Visit: https://www.botify.com/



DeepCrawl is our enterprise crawling tool of choice. While it is more expensive than the above two mentioned SEO crawling tools, DeepCrawl goes deep into your site and provides you the recommendations you need to help you prioritize your technical SEO fixes for your site. Pricing is based on the number of pages you crawl each month and can be dialed up and down as needed. I’ve used this on sites with 10 million+ pages and it worked a charm and took our site’s SEO to the next level. Highly recommended.

Visit: https://www.deepcrawl.com/



Splunk is actually an IT and server admin tool, but it can be very powerful when applied to your log files for SEO. Using Splunk, you can identify how the search engines are actually crawling your website and make optimizations from that data. It’s not a cheap tool when used for the full power, though there is a free version that can get you started. If you are working on any level of enterprise website, though, it could be worth your money. A great post was written here on Moz about using it for Technical SEO.

Visit: http://www.splunk.com/



If you’re in the SEO world, you know Moz. They have the popular Moz blog and put on their annual Mozcon conference (one of the best conferences each year in the digital marketing world, in my opinion). Their Pro subscription allows you to set up not only rank tracking and other diagnostics for your site and some competitors, but it also gives you a weekly crawl of your site, delivered to your inbox, that shows you the issues they found on your site so you can prioritize the fixes.

Visit: https://moz.com/products/pro

These are the tools that we’ve used and recommend, but there are others such as Ryte (formerly OnPage.org), OnCrawl, and URL Profiler that could have been mentioned. What’s your tool of choice for crawling and auditing your website for SEO purposes?


Looking for tool and book recommendations?

Sometimes the hardest part of growing your company is finding the right tools to use to execute on your strategies. Tools are a dime a dozen, but the right tool for the job is hard to find.

Check out our recommendations for lead generation and SEO tools as well as the books we recommend reading as you grow your business.

4 thoughts on “5 Website Crawling and Auditing Tools For Your Website”

      1. John, you nailed it there’s a time and place for everyone of them.

        However I find myself using deepcrawl for almost everything. Every SEO should have at least one of the great product listed. You really can’t lose with any of them.

  1. I’ve used Screaming Frog a lot. Moz’s crawling tool is ok, not the most robust, but works. Never heard of Splunk or Botify, but I’ll have to check those out! I would have loved to use a tool like DeepCrawl for the enterprise sites I used to work on…wish I’d known more about it at the time!


Comments are closed.

More Articles

ROI vs ROAS – What’s the difference?

Posted by on December 13, 2018 in Growth

When you’re speaking with an agency or consultant about them running your paid acquisition campaigns, such as Google Ads or Facebook Ads, you may hear…

View Post
The top business podcasts we’re listening to at Credo

Posted by on December 11, 2018 in CredoCast/Entrepreneurship

  Check out this episode on iTunes. As I’ve been building Credo for the last few years, I’ve realized that in order to have a…

View Post
When should CPG companies start investing in social ads?

Posted by on December 5, 2018 in CredoCast/Social Media

There’s a lot of advice out there about investing in social media advertising and how it’s the new wild west of advertising, with cheaper clicks…

View Post