Webpage rot is part of the reality of the internet. A study in 2003 found that 2% of the internet went away every week, and a 2016 study found that 66.68% of domains that ranked in the top 30 in Google were nowhere to be found in 2015.
Content attrition like this is natural, but it also has SEO ramifications.
Search engine optimizers (also known as SEOs) love talk about 404s and fixing them, so much so that they’ll be included in an audit when there are very few 404ing pages on a website. Many 404s should be fixed after all, but not all.
The question facing SEOs more and more over the last few years has been whether websites should use the 404 status code or the 410 status code when a page goes away.
After all, different status codes have different ramifications for SEO. SEOs are largely focused on giving search engines the correct signals, so the question of 404 vs 410 is a very valid one.
Keep reading to learn more about these two status codes and when to use each.
Table of Contents
What is a 404 status code?
According to W3.org, the 404 status code is defined as:
“The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent.”
In layman’s speak, a 404 simply means “this server cannot find this requested page, and that’s all we know.”
The problem with 404s is that there is no indication from the server of whether this was on purpose or not. Thus, search engines are conservative with how they treat these URLs. They often leave them in the search results for a while.
The problem with this approach is that searchers have a bad experience when they find 404s on the internet. And, 404s on your website will hurt your SEO if they have inbound links that were previously helping that page rank.
What is a 410 status code?
The 410 status code, on the other hand, is a much more specific status code than 404. According to W3, a 410 status code is defined this way:
“The requested resource is no longer available at the server and no forwarding address is known. This condition is expected to be considered permanent. Clients with link editing capabilities SHOULD delete references to the Request-URI after user approval.”
According to this definition, the 410 status code should be used when the URL is purposefully gone forever. And when the URL has not been redirected. URLs that contain backlinks or get organic traffic should be redirected when there is a new URL.
410s, because they are more permanent, have also been found by SEOs to be crawled fewer times. And less often than 404s. This is before they are removed from the search index. This means, hypothetically, a better experience for searchers. This is something search engines are focused on. After all, providing them with the right status code will also help users be more satisfied with their search results. Thus, they will search more often.
When should you use 404 vs 410 and vice versa?
The line of when to use a 404 vs a 410 is pretty clear based on the definitions of the two. Of course, Google has made sure to muddy the waters, albeit not necessarily intentionally, by saying that either is acceptable.
So, to continue Credo’s goal of giving you signal over noise, here is when to use each status code:
- Use a 404 status code as your default status code for when a page goes away and it is not purposeful. Then, have a monitoring system set up so that your web server alerts you when these appear. Then you can fix them either by restoring the page or redirecting the URL to the new location.
- Use a 410 status code when a page is purposefully removed, should no longer appear in the search engine’s search results, and does not have a replacement page to which it should be 301 redirected.
Of course, with everything in SEO and marketing, there are edge cases. There are also some technologies that are inflexible to be able to let you control 404s vs 410s. In that case, we do the best we can.
As a generality, use 404s when pages don’t exist and shouldn’t resolve. Use 410s if your server permits it. Otherwise use a 404 and just accept that it may take longer for search engines to drop the page from their index than if you were able to use a 410.
Which 404s Should You Prioritize?
Before we finish, you should know that not all 404s need to be fixed! 404s are a natural part of the web and expected to happen.
404s do need to be fixed when either of these occur:
- Website visitors are finding the page and therefore having a bad experience,
- External websites link to the 404ing page, which is bad for SEO performance.
In either case, the 404 should be fixed via a 301, or permanent, redirect assuming that the page is not meant to come back. If the page is meant to come back and is just gone temporarily, then a 302 temporary redirect should be used.
Assuming one of the two, or both, cases listed above is true, how do you prioritize?
Well, it depends on the reason you’re fixing 404s.
If you’re fixing 404s for SEO gains, or because you saw an SEO performance drop, then you should prioritize based on pages with the most external links. You can use a tool like Semrush to do this.
First, crawl your site using an SEO tool like Screaming Frog’s SEO Spider (free up to 500 URLs, paid after that). You’ll have a full copy of your site:
Then click the Client Error option on the right side:
Now you’ll have a full list of URLs.
First, make sure that these URLs are not linked internally from another page on your site. You can find these internal links by navigating to Client Error -> clicking the URL -> Inlinks and then reviewing all internal links. You should fix these internal URLs as well as redirecting the 404ing page. Click here for our guide to fixing internal 301s.
Next, export the full list of erroring URLs. First sort by Inlinks to prioritize those with internal links. Work down this list fixing each as you go and deciding:
- Should this URL exist? If so, then fix the page.
- The page goes away: ask if there is a page to which it should be redirected?
- If a page does not yet exist, then create the page where it should be redirected.
Once you have your full list of 404s and where the page should redirect to, it’s time to prioritize them by backlinks.
We used Semrush for this analysis, but you can use whatever tool you like to find linking root domains by URL. Ahrefs has a tool as well.
First, use Semrush’s Bulk Backlink Analysis tool to enter the URLs:
Once you click Compare, they will run the URLs through their comprehensive backlink database and give you backlink counts by URL:
Export this data and prioritize the pages that have the most external links for redirection. If a page doesn’t have external links, internal links, or traffic (as seen from Google Analytics or whatever analytics tool you are using), don’t worry about redirecting it.
Need Technical SEO Help?
Technical SEO is a growing area of SEO and takes an expert to untangle what’s happening on a site so that the site can rank.
If you’re experiencing technical SEO issues, get in touch with our team at Credo to talk about how we can introduce you to agencies from our network who can handle these tough technical issues for you.
