The Ryte Crawler uses an optimized Chrome browser with Google Lighthouse audits. Analyze your entire website with the Ryte crawler to get the most comprehensive analysis and make sure your website is free of mistakes.
Using JavaScript in designing and building websites makes it possible to create interactive, dynamic content for a great user experience. Visual animations, navigations, and displaying dynamic content to users based on previous purchase behavior are just a few of the things that can be achieved with JavaScript.
However, JavaScript has its drawbacks: it can be hard to combine with good search engine optimization. JavaScript means inefficiency. It uses up crawl budget because Google requires more resources to render, crawl and index JavaScript websites than HTML websites, as it adds a layer of complexity to the process. Experiments by Bartosz Goralewicz have shown that this extra complexity can lead to the Googlebot not always rendering and crawling JavaScript pages. If a website is not rendered, it can’t be crawled, and therefore won’t be indexed. The result: your website won’t be visible in the SERPs, and you’ll lose out on valuable traffic.
That’s where our new High-Performance Crawler can help. It can execute, render and crawl JavaScript to give you the data you need to ensure that your dynamic website is free of mistakes. We use an optimized Chrome browser to render JavaScript like you do with your browser. Compared to other products which use third-party render services, for example, this gives you real insights and all the details you need to optimize your web presence.
The Crawler now also uses Google Lighthouse audits which means we can provide you with access to multiple Lighthouse metrics for example in our Web Vitals Report, and we can simulate mobile devices and mobile networks like 4G to provide you with accurate and actionable data for mobile performance.
The High-Performance Crawler can crawl all JavaScript libraries and frameworks, whether JQuery, Ember.js, plain js, React, Vue.js, Angular or even libraries and frameworks that you have built yourself. It executes the JavaScript on each page for up to 30 seconds, so all elements that are triggered when the page is first loaded are rendered and crawled.
If there are any errors stopping users or Google from accessing content, you will be able to find them and fix them, for example broken links that are dynamically inserted with JavaScript. If you use JavaScript to change or add content elements such as page titles and meta descriptions, you will be able to identify missing or duplicate descriptions and titles.
The Ryte dashboard and reports in Website Success make it easy to find the information you need. The Dashboard provides all the information you need to help you get started with your website optimizations.
Figure 1: Start your website optimizations on your project dashboard
JavaScript crawling is currently available for all Business Suite customers. Basic Suite customers can upgrade their accounts by contacting our sales team. If you’re not a Ryte user yet, you can contact our sales team to request an offer, or have a look at our pricing page for an overview of our packages.
JavaScript crawling can be enabled in the project settings by account owners. Just click on "Enable JavaScript crawling".
Figure 2: Enable JavaScript Crawling in your project settings
We have made sure that the most important settings are available even with JavaScript crawling. However, a few are not yet available, or have been slightly adjusted. Here’s a quick overview of the settings that are not yet available:
| Setting | Availability |
| Ignore/include URLs | Available (Learn more) |
| Accept cookies | Not available |
| Login data | Coming soon |
| Ignore GET parameters | Coming soon |
| Crawler waiting time | Not available |
| Analyze images | Automatic, no option to deactivate |
| URL normalization | Not available |
| Custom snippets | Coming soon |
Single Page Analysis is available, but cannot crawl and render JavaScript as it is not yet using the High-Performance Crawler.
Tip:Before you enable JavaScript crawling, you should filter out the RyteBot from your web analytics tools’ settings, as the crawler executes tracking scripts. If you want to give the RyteBot a custom name, you can do this in "Advanced analysis >> What to analyze >> Analysis User Agent". If you need any extra advice regarding the project settings, don’t hesitate to get in touch with our Customer Success Team.
Figure 3: Name the RyteBot
JavaScript is great for users, but not for search engines. With its ability to execute, render and crawl JavaScript, the High-Performance Crawler will help provide you with more transparency into your website, making it easy for you to extract the data you need to ensure a technically perfect dynamic website. Analyze your website with our fast, precise and powerful High-Performance Crawler, and don’t let JavaScript stand in the way of success in the SERPs.
Analyze your website with the High-Performance Crawler!
Request JavaScript crawling!Published on 02/21/2019 by Olivia Willson.
Who writes here
After studying at King’s College London, Olivia moved to Munich, where she joined the Ryte team in 2017. She is in charge of product marketing and CRO, and also helps out with SEO and content marketing. When she's not working, you can usually find her outside, either running around a track, or hiking up a mountain.
Get more traffic and customers by optimizing your website, content and search performance. What are you waiting for?