JavaScript crawling with the new High-Performance Crawler

Ryte has had one of the best crawlers in the industry for the past seven years, and uses the most advanced crawling technology to crawl over 1.5 billion URLs per month for our 700,000 users. Now it’s time to prepare for the future. Our developers have spent the past year writing 15,000 lines of brand new code to bring you the new High-Performance Crawler, which can crawl your site as fast as fast as you can deliver URLs.

The best part: it can even crawl and render JavaScript to provide you with the data you need to ensure a technically perfect dynamic website.

JavaScript and SEO: The Challenge

Using JavaScript in designing and building websites makes it possible to create interactive, dynamic content for a great user experience. Visual animations, navigations, and displaying dynamic content to users based on previous purchase behavior are just a few of the things that can be achieved with JavaScript.

However, JavaScript has its drawbacks: it can be hard to combine with good search engine optimization. JavaScript means inefficiency. It uses up crawl budget because Google requires more resources to render, crawl and index JavaScript websites than HTML websites, as it adds a layer of complexity to the process. Experiments by Bartosz Goralewicz have shown that this extra complexity can lead to the Googlebot not always rendering and crawling JavaScript pages. If a website is not rendered, it can’t be crawled, and therefore won’t be indexed. The result: your website won’t be visible in the SERPs, and you’ll lose out on valuable traffic.

Analyze Your Dynamic Website with Ryte

That’s where we come in. We have created the High-Performance Crawler that can execute, render and crawl JavaScript to give you the data you need to ensure that your dynamic website is free of mistakes. We use an optimized Chrome browser to render JavaScript like you do with your browser. Compared to other products which use third-party render services, for example, this gives you real insights and all the details you need to optimize your web presence.

The High-Performance Crawler can crawl all JavaScript libraries and frameworks, whether JQuery, Ember.js, plain js, React, Vue.js, Angular or even libraries and frameworks that you have built yourself. It executes the JavaScript on each page for up to 30 seconds, so all elements that are triggered when the page is first loaded are rendered and crawled.

If there are any errors stopping users or Google from accessing content, you will be able to find them and fix them, for example broken links that are dynamically inserted with JavaScript. If you use JavaScript to change or add content elements such as page titles and meta descriptions, you will be able to identify missing or duplicate descriptions and titles.

The Ryte dashboard and reports in Website Success make it easy to find the information you need. For example, the "Critical errors" report, which you can reach from your project Dashboard, will show you immediately if any serious errors were found. All data regarding content, headings and meta descriptions is available in the Content report.

Figure 1: See your website’s critical errors on your project dashboard

JavaScript crawling is currently available for all Business Suite customers. Basic Suite customers can upgrade their accounts by contacting our sales team. If you’re not a Ryte user yet, you can contact our sales team to request an offer, or have a look at our pricing page for an overview of our packages.

Activate JavaScript Crawling

JavaScript crawling can be enabled in the project settings by account owners. Just click on "Enable JavaScript crawling".

Figure 2: Enable JavaScript Crawling in your project settings

Crawler Settings

We have made sure that the most important settings are available even with JavaScript crawling. However, a few are not yet available, or have been slightly adjusted. Here’s a quick overview of the settings that are not yet available:

Setting	Availability
Ignore/include URLs	Coming soon
Accept cookies	Not available
Login data	Coming soon
Ignore GET parameters	Coming soon
Analyze subfolder	Coming soon
Crawler waiting time	Not available
Analyze images	Automatic, no option to deactivate
URL normalization	Not available
Custom snippets	Coming soon
Additional request header	Coming soon

Single Page Analysis is available, but cannot crawl and render JavaScript as it is not yet using the High-Performance Crawler.

Tip:

Before you enable JavaScript crawling, you should filter out the RyteBot from your web analytics tools’ settings, as the crawler executes tracking scripts. If you want to give the RyteBot a custom name, you can do this in "Advanced analysis >> What to analyze >> Analysis User Agent". If you need any extra advice regarding the project settings, don’t hesitate to get in touch with our Customer Success Team.

Figure 3: Name the RyteBot

Conclusion

JavaScript is great for users, but not for search engines. With its ability to execute, render and crawl JavaScript, the High-Performance Crawler will help provide you with more transparency into your website, making it easy for you to extract the data you need to ensure a technically perfect dynamic website. Analyze your website with our fast, precise and powerful High-Performance Crawler, and don’t let JavaScript stand in the way of success in the SERPs.

This article is also available in German and French