OnPageBot

Introduction of our crawler and service

We are Ryte - a Website Quality Assurance Software. We fight for a cleaner web. No more broken pages, no more links to deleted contents and better performance for all. Please help us to make the web a little better.

Why did Ryte crawl my website?

• Ryte is not a search engine
• We only crawl websites at the behest of our clients
• We check the technical status of our clients website, covering internal as well as external linking.
• If our client linked to your website, we will only access this particular URL. This check is being made to give our client the info, that the reference is still valid.
• We gather this data so that our clients can keep their websites up to date.

Why you should not block our crawler

If you block our crawler, our clients might think that they are referencing (aka linking to) a broken page. In order to keep their website up to date and clean, they might be inclined to remove this reference to your page based on this false information.

Why do you use the "Googlebot" UserAgent?

By default we use the UserAgent "Googlebot 2.1 (compatible; OnPageBot; +https://ryte.com/bot/)". But our users can set any useragent in their project settings.

We recommend our clients to use the Useragent "Googlebot" in order to reveal so called "cloaking" which is prohibited in the Webmaster Guidelines of Google. By visiting the website as Googlebot we offer our clients a way to identify if this technique is used on their domain (or referenced pages).

How often does Ryte crawl my website?

Our clients can schedule the crawler. By default the crawler starts on demand only. Clients can schedule a time slot for crawling their website via the crawl settings. Crawling is possible at any time.

What IP address will Ryte use to crawl?

We use a wide-range of IP-addresses to crawl the website of our clients. If you need further guidance regarding whitelisting our service please contact support@ryte.com.

If your page is beeing crawled without your permission and you wold like to stop then please contact us at support@ryte.com.

How many requests are send? How do you determine the crawling speed?

The speed of our crawlers is also determined by the project settings. The default setting are 10 parallel requests - simulating 10 users clicking from page to page on our clients domain. This setting can be decreased. If a user has verified the ownership of a domain he can also increase the amount of parallel requests (and with that the amount of requests per second).

Why are some pages crawled every 5 minutes?

This is our monitoring service. Our clients can define their most important pages (by default their homepage) which get constantly monitored. This is done to inform our users if their website is facing an issue (like a server downtime, bad database performance, etc.).