How to Easily Crawl Your JavaScript Website

Contents

• JavaScript and SEO: The Challenge
• How to crawl your dynamic website
• Find and fix errors
• Activate JavaScript crawling
• Conclusion

JavaScript and SEO: The Challenge

Using JavaScript in designing and building websites makes it possible to create interactive, dynamic content for a great user experience. Visual animations, navigations, and displaying dynamic content to users based on previous purchase behavior are just a few of the things that can be achieved with JavaScript. However, JavaScript has its drawbacks: it can be hard to combine with good search engine optimization. JavaScript means inefficiency. It uses up crawl budget because Google requires more resources to render, crawl and index JavaScript websites than HTML websites, as it adds a layer of complexity to the process. Experiments by Bartosz Goralewicz have shown that this extra complexity can lead to the Googlebot not always rendering and crawling JavaScript pages. If a website is not rendered, it can’t be crawled, and therefore won’t be indexed. The result: your website won’t be visible in the SERPs, and you’ll lose out on valuable traffic.

Analyze Your Dynamic Website with the Ryte crawler

That’s where our new High-Performance Crawler can help. It can execute, render and crawl JavaScript to give you the data you need to ensure that your dynamic website is free of mistakes. We use an optimized Chrome browser to render JavaScript like you do with your browser. Compared to other products which use third-party render services, for example, this gives you real insights and all the details you need to optimize your web presence. The Crawler now also uses Google Lighthouse audits which means we can provide you with access to multiple Lighthouse metrics for example in our Web Vitals Report, and we can simulate mobile devices and mobile networks like 4G to provide you with accurate and actionable data for mobile performance. The High-Performance Crawler can crawl all JavaScript libraries and frameworks, whether JQuery, Ember.js, plain js, React, Vue.js, Angular or even libraries and frameworks that you have built yourself. It executes the JavaScript on each page for up to 30 seconds, so all elements that are triggered when the page is first loaded are rendered and crawled.

To summarise, with the Ryte crawler, you can:

• Render pages with proper javascript execution like a normal user would see it
• Access more than 200 different metrics including Lighthouse performance metrics
• Simulate mobile devices and mobile network like 4G for analysing mobile performance

Our cutting edge streaming technology allows:

• up to 100 parallel requests
• crawling websites of any size
• crawling any number of projects in parallel

Find and fix errors

If there are any errors stopping users or Google from accessing content, you will be able to find them and fix them, for example broken links that are dynamically inserted with JavaScript. If you use JavaScript to change or add content elements such as page titles and meta descriptions, you will be able to identify missing or duplicate descriptions and titles. The Ryte dashboard and reports in Website Success make it easy to find the information you need. The Dashboard provides all the information you need to help you get started with your website optimizations.

Figure 1: Start your website optimizations on your project dashboard

JavaScript crawling is currently available for all Business Suite customers. Basic Suite customers can upgrade their accounts by contacting our sales team. If you’re not a Ryte user yet, you can contact our sales team to request an offer, or have a look at our pricing page for an overview of our packages.

Activate JavaScript Crawling

JavaScript crawling can be enabled in the project settings by account owners. Just click on "Enable JavaScript crawling".

Figure 2: Enable JavaScript Crawling in your project settings

Crawler Settings

We have made sure that the most important settings are available even with JavaScript crawling. However, a few are not yet available, or have been slightly adjusted. Here’s a quick overview of the settings that are not yet available:

Setting	Availability
Ignore/include URLs	Available (Learn more)
Accept cookies	Not available
Login data	Available
Ignore GET parameters	Available
Crawler waiting time	Not available
Analyze images	Automatic, no option to deactivate
URL normalization	Not available
Custom snippets	Coming soon

Single Page Analysis is available, but cannot crawl and render JavaScript as it is not yet using the High-Performance Crawler. Tip: Before you enable JavaScript crawling, you should filter out the RyteBot from your web analytics tools’ settings, as the crawler executes tracking scripts. If you want to give the RyteBot a custom name, you can do this in "Advanced analysis >> What to analyze >> Analysis User Agent". If you need any extra advice regarding the project settings, don’t hesitate to get in touch with our Customer Success Team.

Figure 3: Name the RyteBot

Conclusion

JavaScript is great for users, but not for search engines. With its ability to execute, render and crawl JavaScript, the High-Performance Crawler will help provide you with more transparency into your website, making it easy for you to extract the data you need to ensure a technically perfect dynamic website. Analyze your website with our fast, precise and powerful High-Performance Crawler, and don’t let JavaScript stand in the way of success in the SERPs. This article is also available in German and French