BotLogs: The Smart Alternative to Log File Analysis

Log file analysis is an important part of website quality management and of search engine optimization. It shows optimization potentials and critical errors that would not be found any other way.

What Is Log File Analysis?

A log file or server log records every “hit”, or request to a server. A log file contains information such as:

• IP address and hostname
• User agent
• HTTP status code
• Date and time
• Method (Get or post)
• Referrer (site from which a user comes)
• Requested URL

Log file analysis describes the process of analyzing these requests to the server, extracting valuable information regarding user and bot behavior on websites.

Advantages of Log File Analysis

Log file analysis makes it possible to see your website exactly how search engine bots see it, and provides information impossible to find any other way. Many SEO crawlers simulate bot action and therefore display data extracted at the time of analysis, but log file analysis provides real time data into how search engines bots behave on your site. With log file analysis, you can:

• Identify how frequently pages are crawled by search engine bots, meaning you can prioritize these pages when optimizing your website.
• Find and fix errors specific to the mobile and desktop versions of your site.
• See the status code generated when bots visit your site, showing you if bots have issues in accessing certain areas of your website.

Analyzing bot behavior on your website will help you optimize your website fully for search engines, ensuring good rankings and a solid online presence.

Disadvantages of log file analysis

SEOs who are used to log file analysis will know that there are certain drawbacks. Above all, log file analysis can be complex and time consuming. To even be able to start analyzing your log files, you need to access them, which could involve either talking to your development team, or contacting your server administrators. Once you’ve got hold of the server logs, you can start analyzing. But, looking through and analyzing your server logs manually can be a full time job.

The solution: BotLogs

We at Ryte have developed tool with which you can analyze your log files without the complexity. With Ryte BotLogs, you can easily see what search engine bots see, helping you to better understand your website.

What can you do with Ryte BotLogs?

• See which bots have crawled your website
• Identify crawled URLs
• Understand crawl frequency
• Gain clarity into the relationship between Desktop Bot and Mobile Bot
• Discover broken links and page errors
• Find unnecessary redirects
• See details about server performance
• Work in compliance with GDPR

Ryte only records logs from search engine bots, not from users, so there’s no need to worry about GDPR data regulations. Furthermore, you won’t need to access, manually upload or store your server logs. All the information is there for you with just a few clicks.

Ryte also carries out a double DNS check with your log files to verify that these are genuine log files generated from search engine bots’ requests to servers.

Why Do You Need Ryte BotLogs?

1. Identify which pages are crawled, and how frequently

Identifying which pages are most frequently crawled by search engine crawlers helps you to prioritize important pages, and to identify issues. Pages that are crawled most frequently should be optimized and updated regularly to increase traffic. Make sure that your most important content exists on these pages.

Finding which pages are crawled most frequently can also help you identify issues. For example:

• You can identify important pages that are not crawled as often as they should be. Improving your site architecture and internal linking structure will ensure that these pages are more easily found by the search engine crawlers.
• Search engine bots might be crawling pages you didn’t realise existed, for example URLs that have more than one version. Letting this issue go unnoticed could lead to issues with duplicate content which makes it difficult for the Google indexer to see which page is most important. If bots are crawling more than one version of a URL, this also wastes Crawl budget - Google allocates a certain amount of resources to a website, so it shouldn’t waste resources crawling multiple versions of URLs.
• A non-indexable page might be crawled more frequently than you would expect, considering that it should not be visible in the SERPs. Check that there are no issues with the meta tags on this page, or that it is not linked too often internally.

Seeing which URLs are crawled by bots, and how frequently, can help you find and fix structural and critical errors on your website which makes it easier for Google to crawl and index your site.

With Ryte BotLogs, you can easily see which pages are crawled most frequently. In the "All Pages" report, the number of bot visits is shown in a bar chart, with the crawled URLs listed below. The list shows information regarding response time, status code, and number of bot visits. Sort according to bot visits to find pages that are most/least frequently crawled, or sort according to status code to easily find errors. Use the search function to find individual URLs.

Figure 1: See how frequently bots visited your pages, including number of visits, response time, and status code.

Tip:

By clicking on any URL in any report, the Super Inspector will pop-up, providing further information regarding frequency of bot visits, which bots visited the page, an overview of the load time, and a timeline of bot visits. This is the case if you click on a URL in any report, making it easy for you to identify and fix any issues.

Figure 2: The BotLogs Super Inspector

2. Find errors specific to your mobile and desktop sites

Log file analysis helps to easily identify and distinguish errors on your desktop and mobile sites. A Log file analysis shows the exact search engine bot that visited your site, meaning you can distinguish between visits from the Google desktop bot and the google mobile bot. If visits from the mobile bot encountered more errors, this is a clear indication that you need to focus on optimizing your mobile website.

In the "Mobile vs Desktop" report in Ryte BotLogs, you can see how many times a page was visited by the Google mobile and desktop bots, and the respective status code. Click on any title to filter according to the respective metric.

Figure 3: Mobile / Desktop report in Ryte BotLogs.

The example above shows that the mobile bot is crawling the pages more frequently than the desktop bot, and the yellow exclamation mark shows issues.

3. Easily find critical errors with response times and status codes

With BotLogs, you can monitor real-time critical errors. As soon as 404 errors are encountered by bots, these will show up in your reports. An additional overview of status codes and errors found can be seen in the "Bot Timeline". This timeline gives an overview of bot visits from all days, including the number of issues found when crawling.

Figure 4: Bot Timeline

This report also clearly shows which bots are crawling the page - the fact that the Google mobile bot is crawling the website more frequently than the desktop mob is a clear indication to optimize for the mobile version.

Log files provide concrete information regarding the response times of pages. With Ryte BotLogs, you can see a clear overview of response times in the Server Performance report. You can see the average response time, as well as the response time of the last bot’s visit.

Figure 5: Mobile / Desktop report in Ryte BotLogs

Conclusion

With Ryte BotLogs, you can analyze your log files without the complexity. You won’t have to access or analyze them manually, and you won’t even have to think about data protection as only bot visits are recorded. All information you need to see your website from the search engine crawlers’ perspective is available with a few clicks. Find which pages are crawled and how often; see how your desktop and mobile websites are crawled; find critical issues. Carry out a log file analysis easily, and optimize your website so it can be crawled and indexed efficiently for a solid online presence.