How To Deal with 404 Errors

"404 – page not found" – who hasn’t seen this when browsing the internet? There’s nothing more annoying than seeing this message when you’re searching for information. 404 errors can have negative effects on your website in many ways - they frustrate users and send negative signals to search engine bots. It’s therefore important to quickly identify and deal with 404 errors.

What causes a 404 error?

A 404 error message is caused when a URL is no longer accessible. This might be because the page no longer exists, or the URL has been changed.

• Deleted or removed pages: a website operator may remove a page because it is no longer up to date, for example a certain product is no longer available in an online shop.
• Website relaunch or domain transfer: the URLs belonging to the previous domain should be redirected to the new site with a 301-redirect. If not URLs are redirected, because, for example, you want to save resources and only redirect the most important URLs, 404 errors will occur. Your URLs may also be linked from external resources. If you knowingly change a URL, you should implement a redirect.
• URL restructuring: if, for example, you change the name of a directory, this will result in URLs changing. If redirects are not implemented, this will result in 404 errors.

How 404 errors affect usability and SEO

404 errors can have a negative effect on your website regarding both usability and the search engine. A high number of 404 error pages on a website will give your user a negative user experience, and they’re more likely to leave your website and go to a competitor’s website. A high number of 404 errors wastes crawl resources, and raise the risk of search engine crawlers not being able to access valuable content through the link structure.

However, 404 errors do not impact the ranking of your website, and Google will not penalise your website due to a high number of 404 errors. Gary Illyes confirmed this with a tweet in 2015:

Figure 1: Tweet from Gary Illyes in 2015

How Can You Find 404 Errors on Your Website?

Analyze 404 error pages in Website Success

Ryte's module Website Success can help you to easily identify 404 errors that have been found on your website by the Ryte bot. Go to the Website Success module, select "Indexability" → "Status Codes", and click on 4xx status codes.

Figure 2: Identify 404 errors with Ryte

If you want to get to the root of the problem, you also need to also analyze what pages link to these inaccessible URLs. These can be identified by clicking on "Links" → "Overview" → "List of all links", and then setting up the following filters:

Click on "Add new filter", select "is local file (source)", and set the option to "Local file".

Figure 3: Add filter to view local files

This filter lists all links that point to internal pages. To limit the results to just the inaccessible pages, i.e. where a 404 error will occur, click on "Add new filter", then "Status Code (source)", and select "is" "404".

Figure 4: Filter to show all 404 pages

Once you have successfully created and applied both filters, you will see a list of all internal 404 errors and the pages that link to them.

Tip: If you have linked Ryte with Google Analytics, the Ryte bot also analyzes all the URLs from Analytics. This raises your chances of identifying all 404 error pages and gives you a clear overview of the number of visitors who have visited the various URLs in the last 30 days. This means that you can prioritize which 404 errors to correct first based on their traffic.

Identify 404 errors using the Google Search Console

The Google Search Console provides you with a lot of useful information about your domain. By clicking on "Index" → "Coverage" you see a list of issues that the Googlebot found when crawling. Clicking on an issue will open a list with the affected URLs.

Figure 5: Identify 404 errors in the Google Search Console

Tip: Taking a look at soft 404 pages often pays off. Soft 404 pages are faulty or non-existent URLs that still return a "200 OK" or "302 Found" status code.

The Google Search Console lists all 404 pages that have been detected on your website, both now and in the past. When analyzing the 404 pages, you should firstly check the date of the page, and see if it still exists.

When analyzing these key figures, note that the data is partially generated by comparing the URLs stored in the Sitemap.xml. This means that 404 errors that have already been corrected can be displayed here, or URLs that are no longer present in Google’s search index. Often, 404 errors appear in the list suddenly even though you have already noted them as fixed. That happens because Google crawls each website multiple times.

How to Handle 404 error pages

You can deal with 404 errors either by creating a redirect, or setting up an error page offering users alternative information. If you do not set up an error page or a redirect, you will send negative signals to users and to Google. The Googlebot will try to access a non-existent resource, but will not get the correct status code 404 or 410 that the URL is no longer available. Users will also receive negative signals as they do not know why the URL isn’t working.

Redirects

If there is similar content elsewhere on your website to the page that has been removed, you can redirect users to this URL. This is the best solution if you think that you can fulfill the user’s needs with the alternative page.

You may be tempted to implement a redirect so as to not lose out on link power, in some cases, a 404 error should be left as it is. This is particularly the case when you want to remove content permanently, and there are no other pages with similar content. Implementing a redirect here could create a negative user experience - the user clicks on a URL expecting to find content to fulfill their search intent, and if they are redirected to content that does not fulfill their needs, they will leave your page and go to your competitor’s website.

These pages should be set to provide a "404 not found" page, so that users are informed that the content no longer exists.

Creating a 404 error page

Creating a 404 error page means you can show users a message that the content is no longer available, and offer them an opportunity to search elsewhere on your website for the content they need.

To create a 404 error page, you firstly need to configure your server correctly. You can do this by adding the following code in the .htaccess file:

ErrorDocument 404/404.php

You shouldn’t use the domain name in this .htaccess line, as search engines will often interpret this as a soft 404 error. This means that if there’s a server request for a URL that is no longer available (Error document 404), the following page is displayed. The next step is to create this page.

404 Error Page Examples

When setting up a 404 page, there are many several basic elements that you should provide. The aim is to make sure that users stay on your website. As well as showing a notification that the original page is no longer available, you could provide a link to the homepage, suggestions for the users to navigate to similar content, or a search function. Here are some examples showing how you can provide creative 404 pages to keep your users on your website.

1. Refer to the homepage

The most simple way to keep your users on your website is linking to the homepage.

Figure 6: Referring to the homepage of ryte.com

2. Refer to other categories and the homepage

Figure 7: Referring to other categories on the 404 page (source)

3. Offer a search function

Figure 8: Search function on the 404 page (source)

4. Offer an option for interaction

Figure 9: Opportunity to interact

Google hardly shows particular creativity with their 404 error page. However, as the provider of the biggest search engine world wide, it’s obvious which page the user will call up next…

Figure 10: Google's 404-page

Conclusion

You should keep your 404 errors pages to a minimum due to the negative effects they can have for usability and for search engines. However, the negative effects of 404 errors on usability can be easily negated with a creative 404 page. If you offer navigation options or a search function so that users can find the content they need, the chances are good that they will stay on your page.