Including log file analysis into your website strategy will help you optimize your website to ensure that it can be crawled and indexed efficiently so that your business can be found online easily. Find out how you can carry out a log file analysis easily with Ryte.
Log file analysis is an important part of website quality management and of search engine optimization. It shows optimization potentials and critical errors that would not be found any other way.
A log file or server log records every “hit”, or request to a server. A log file contains information such as:
Log file analysis describes the process of analyzing these requests to the server, extracting valuable information regarding user and bot behavior on websites.
Log file analysis makes it possible to see your website exactly how search engine bots see it, and provides information impossible to find any other way. Many SEO crawlers simulate bot action and therefore display data extracted at the time of analysis, but log file analysis provides real time data into how search engines bots behave on your site. With log file analysis, you can:
Analyzing bot behavior on your website will help you optimize your website fully for search engines, ensuring good rankings and a solid online presence.
SEOs who are used to log file analysis will know that there are certain drawbacks. Above all, log file analysis can be complex and time consuming. To even be able to start analyzing your log files, you need to access them, which could involve either talking to your development team, or contacting your server administrators. Once you’ve got hold of the server logs, you can start analyzing. But, looking through and analyzing your server logs manually can be a full time job.
We at Ryte have developed tool with which you can analyze your log files without the complexity. With Ryte BotLogs, you can easily see what search engine bots see, helping you to better understand your website.
What can you do with Ryte BotLogs?
Ryte only records logs from search engine bots, not from users, so there’s no need to worry about GDPR data regulations. Furthermore, you won’t need to access, manually upload or store your server logs. All the information is there for you with just a few clicks.
Ryte also carries out a double DNS check with your log files to verify that these are genuine log files generated from search engine bots’ requests to servers.
Identifying which pages are most frequently crawled by search engine crawlers helps you to prioritize important pages, and to identify issues. Pages that are crawled most frequently should be optimized and updated regularly to increase traffic. Make sure that your most important content exists on these pages.
Finding which pages are crawled most frequently can also help you identify issues. For example:
Seeing which URLs are crawled by bots, and how frequently, can help you find and fix structural and critical errors on your website which makes it easier for Google to crawl and index your site.
With Ryte BotLogs, you can easily see which pages are crawled most frequently. In the "All Pages" report, the number of bot visits is shown in a bar chart, with the crawled URLs listed below. The list shows information regarding response time, status code, and number of bot visits. Sort according to bot visits to find pages that are most/least frequently crawled, or sort according to status code to easily find errors. Use the search function to find individual URLs.
Figure 1: See how frequently bots visited your pages, including number of visits, response time, and status code.
By clicking on any URL in any report, the Super Inspector will pop-up, providing further information regarding frequency of bot visits, which bots visited the page, an overview of the load time, and a timeline of bot visits. This is the case if you click on a URL in any report, making it easy for you to identify and fix any issues.
Figure 2: The BotLogs Super Inspector
Log file analysis helps to easily identify and distinguish errors on your desktop and mobile sites. A Log file analysis shows the exact search engine bot that visited your site, meaning you can distinguish between visits from the Google desktop bot and the google mobile bot. If visits from the mobile bot encountered more errors, this is a clear indication that you need to focus on optimizing your mobile website.
In the "Mobile vs Desktop" report in Ryte BotLogs, you can see how many times a page was visited by the Google mobile and desktop bots, and the respective status code. Click on any title to filter according to the respective metric.
Figure 3: Mobile / Desktop report in Ryte BotLogs.
The example above shows that the mobile bot is crawling the pages more frequently than the desktop bot, and the yellow exclamation mark shows issues.
With BotLogs, you can monitor real-time critical errors. As soon as 404 errors are encountered by bots, these will show up in your reports. An additional overview of status codes and errors found can be seen in the "Bot Timeline". This timeline gives an overview of bot visits from all days, including the number of issues found when crawling.
Figure 4: Bot Timeline
This report also clearly shows which bots are crawling the page - the fact that the Google mobile bot is crawling the website more frequently than the desktop mob is a clear indication to optimize for the mobile version.
Log files provide concrete information regarding the response times of pages. With Ryte BotLogs, you can see a clear overview of response times in the Server Performance report. You can see the average response time, as well as the response time of the last bot’s visit.
Figure 5: Mobile / Desktop report in Ryte BotLogs
With Ryte BotLogs, you can analyze your log files without the complexity. You won’t have to access or analyze them manually, and you won’t even have to think about data protection as only bot visits are recorded. All information you need to see your website from the search engine crawlers’ perspective is available with a few clicks. Find which pages are crawled and how often; see how your desktop and mobile websites are crawled; find critical issues. Carry out a log file analysis easily, and optimize your website so it can be crawled and indexed efficiently for a solid online presence.
Let BotLogs make your log file analysis easy!Request BotLogs!
Published on 02/11/2019 by Olivia Willson.
Who writes here
After studying at King’s College London, Olivia moved to Munich, where she joined the Ryte team in 2017. She is in charge of product marketing and CRO, and also helps out with SEO and content marketing. When she's not working, you can usually find her outside, either running around a track, or hiking up a mountain.
Get more traffic and customers by optimizing your website, content and search performance. What are you waiting for?
Do you want more SEO traffic?
Improve your rankings for free by using Ryte.Register for free