RyteBot

Introduction of our crawler and service

We are Ryte - a software suite for website quality management. We want a cleaner internet: no more broken pages, no more links to deleted contents, and better performance for all.

Why did Ryte crawl my website?

• Ryte is not a search engine
• We only crawl websites at the behest of our clients
• We check the technical status of our clients website, covering internal as well as external linking.
• If our client linked to your website, we will only access this particular URL. This check is being made to give our client the info, that the reference is still valid.
• We gather this data so that our clients can keep their websites up to date.

Why you should not block our crawler

If you block our crawler, our clients might think that they are linking to a broken page, and they are therefore likely to remove the link to your page.
By default, we use the RyteBot, but users also have a choice of using the Googlebot, or they can customize the name in their project settings.

Why do you use the "Googlebot" UserAgent?

By default we use the UserAgent "Googlebot 2.1 (compatible; OnPageBot; +https://ryte.com/bot/)". But our users can set any useragent in their project settings.

We recommend our clients to use the Useragent "Googlebot" in order to reveal so called "cloaking" which is prohibited in the Webmaster Guidelines of Google. By visiting the website as Googlebot we offer our clients a way to identify if this technique is used on their domain (or referenced pages).

How often does Ryte crawl my website?

Our clients can schedule the crawler to crawl at regular intervals via the project settings. By default, the crawl starts only when activated. Crawling is possible at any time.

What IP address will Ryte use to crawl?

We use a wide-range of IP-addresses to crawl the website of our clients. If you need further guidance regarding whitelisting our service, please contact support@ryte.com.

If your page is being crawled without your permission and you would like it to stop, please contact us at support@ryte.com.

How many requests are sent? How do you determine the crawling speed?

The speed of our crawler is also determined by the client in their project settings. The default setting is 10 parallel requests. This simulates 10 users clicking from page to page on our client’s domain. This can be decreased, and if a user has verified the ownership of a domain, they can also increase the amount of parallel requests (and the amount of requests per second).