SEMrush for Technical SEO

Like many of their competitors, SEMrush has a concept wherein you can track a website (yours or a competitors) and they will regularly crawl your website and email you with recommendations.

For a long time I did not use this feature, but that was to my detriment as they actually provide great takeaways and deep insights. While I firmly believe in using multiple tools (and I have access to not only SEMrush, but also Ahrefs, Screaming Frog, DeepCrawl, Moz, and more), SEMrush has surprised me how it has become a larger part of my workflow as I’ve used their various tools.

Note: Credo is a SEMrush affiliate, and because of that they’re offering you a FREE 14 day trial when you use our exclusive link.

I had always used SEMrush for keyword research and competitor research, but when I began using SEMrush’s “Projects” the utility of the tool skyrocketed.

Here’s how to use SEMrush for technical SEO.

Set up a project

First, you need to set up a Project for your website. Find it here:

On the next page, select Add New Project:

Enter your site’s domain and name your project:

Now you will be taken to your project’s dashboard where you can set up as many of their tools as you want. Let’s choose the Site Audit tool:

When you click Set Up, you will see this modal appear. Depending on the size of your site you can adjust the number of pages they crawl. I set it to 1,000 for my site because it has ~180 URLs in the search index but I also want them to pick up images. Set according to your site’s needs (taking into account future growth plans). If you need to understand how many pages your site has in the index, you can use a simple site:YOURDOMAIN.com search in Google:

Remember, you don’t need to crawl your whole site every week! While at the beginning of an SEO engagement it can be very helpful to have a full crawl, if your site is very large (tens of thousands or more of pages) then a full crawl might not make sense as at that scale, your technical SEO strategies must be more scalable than fixing specific issues like a few 404s or slow-loading pages.

If your site is under 10,000 pages (the SEMrush auditor goes up to 20,000 as you can see), then a full crawl weekly may be useful to identify trends over time. The choice is yours.

The next tab is the Crawler setup where you can tell SEMrush how to crawl your site, such as as SEMrushBot or GoogleBot. Both can be set as desktop or mobile. Do be careful here because crawling as GoogleBot can mess with your site crawl statistics if you are monitoring server logs. It is fine to leave this as SEMrushBot unless you have a very good reason to crawl as GoogleBot:

I choose to not have SEMrushBot respect robots.txt because I want to understand what URLs are blocked by robots.txt in case a change was made and something became blocked unintentionally. This does happen, and thus is something to monitor over time.

I also like to schedule my crawl to happen weekly. I do it on Sundays because Sundays are less trafficked days for my site and should something go wrong with the servers it won’t affect business (this has never happened, but I have brought down sites before with various crawlers by crawling too aggressively):

Now click Start Site Audit and they will get to crawling.

Weekly technical monitoring

Every week when the crawl completes, you will receive an email like this:

As they complete more crawls, you will receive emails with your score, crawled pages, and updates on how things have changed week on week.

I will say that my one gripe with the technical monitoring in SEMrush is that their scores tend to vary widely based on the scale of different “issues”, though these “issues” are not always super high priority.

For example, I received these email which made my heart palpitate, but upon inspection there were no smoking guns and nothing major had changed on my site. And what they are telling me is an issue is a slow site, of which I am already aware and working to fix:

At the end of the day I’d rather have more notifications than fewer, so overall these emails provide a lot of value and having crawl data weekly is incredible for monitoring issues.

Using the Site Audit information

Here is the dashboard of a site crawl (from the above crawl I started):

I tend to ignore the Total Score because it does not give me insight into overall how my site is doing or why the score is what it is. Total Score seems to be a directional metric weighting Errors, Warning, and Notices and assigning a score. You should also be careful of chasing a score within any tool, as even when you achieve a “perfect score” there are still many things to do and that is just a score for one part of your site.

Along the top of the dashboard you will see the navigation options:

You should navigate through each of the sections as they each have their own value to add, though I usually use the Issue section first to see the issues SEMrush has discovered as prioritized by count:

Now click into each and see if they are actually issues, meaning something you can fix or something that you should fix. For example, in this audit I am told that pagination is creating duplicate title tags, BUT I know that I have pagination set up correctly so this really is not an issue. The first set of duplicate title tags are and are something I can (and maybe should) fix, though it might not be highest priority for me:

Unfortunately you cannot hide individual issues within these views, though you can do that within the overall Errors and Issues pages.

One thing that I love about SEMrush technical reports like this is the ability to send them to Trello. I operate my business within Trello and so can easily create each issue as a Card within the Trello Board and List of my choice. From there, you can prioritize them!

Week over week

I talked earlier about comparing week over week crawls, so let me show you that within another Project I have set up on SEMrush.

Here is a Dashboard where you can see the overall trend of issues:

Let’s click on the Compare Crawls tab:

Here’s how it looks, where you can also compare against older crawls to really see the progress you are making month over month, or week over week, depending on how often you are fixing issues on your site:

There are three things to keep in mind here.

First, I love this view because it helps me understand well where issues are being fixed and where they are growing. This lets me see all of the issues, and their counts, within the context of others which lets me prioritize them based on impact and not just respond to the biggest increase of the week (in percentages). Give yourself the gift of focus!

Second, you’re not just chasing a number. While we should be looking to minimize as many issues as possible, prioritizing them based on impact is much more effective.

Third, and most importantly, this can view can be a great way to make a business case for the fact that an issue is growing on your site and becoming a major issue. If your 404s spike, then you can a) dig into it and b) make a case for it being a large issue that deserves to be fixed. Use this data to make things happen, especially if you are working inhouse and have to prioritize fixes according to business impact and work with product managers, designers, and engineers to get things done.

Now that you’ve finished this article, you understand better how to use SEMrush for technical auditing. Go forth and optimize!

Note: Credo is a SEMrush affiliate, and because of that they’re offering you a FREE 14 day trial when you use our exclusive link. If you sign up for a paid account after the free trial, we may be compensated monetarily.