With the whitelist/blacklist function, you can choose to include or exclude parts of your domain from your crawl, giving you more flexibility in your analysis. You can find the function in the project settings, under advanced analysis -> Include or exclude URLs.
When building the new function, we took user feedback into account to ensure everyone could use it easily.
We have created pre-defined filters for the different components of a URL: protocol, domain, path, and query string. So, all you need to do is choose from the dropdown list, and enter the respective protocol, domain, URL path, or query string that you want to include or exclude.
The following filter options are available:
When you’ve set up your desired filters, click "save".
We recommend setting up the whitelist/blacklist function when you set up a new project so that you can track the progress of your optimizations more easily. Suddenly changing the number of URLs crawled would skew your data.
The whitelist/blacklist feature can help you tailor your website analysis specifically to your needs. It will also help you save crawl budget as you can exclude URLs that are not relevant for your individual work.
Some specific cases where the whitelist/blacklist feature can help make your analysis tailored specifically to your needs:
Analyze specific subdomains
In this example, both the English and German subdomains of the Ryte website will be analyzed, using "Domain is en.ryte.com" and "Domain is de.ryte.com"
Analyze a subdomain, but focus on or exclude a directory
In this example, we analyze the subdomain en.ryte.com and the subdirectory "wiki":
Ignore specific file types such as .jpg .pdf .gif
In this example, we exclude URLs with file types .jpg, .pdf. and .gif:
Published on 04/15/2020 by Olivia Willson.
Who writes here
After studying at King’s College London, Olivia moved to Munich, where she joined the Ryte team in 2017. She is in charge of product marketing and CRO, and also helps out with SEO and content marketing. When she's not working, you can usually find her outside, either running around a track, or hiking up a mountain.
Get more traffic and customers by optimizing your website, content and search performance. What are you waiting for?
Do you want more SEO traffic?
Improve your rankings for free by using Ryte.Register for free