Page Comparison

The following regex will allow you to exclude at fetch time all the urls beyond a certain depth, to be set in the exclusions tab of your job: (/[^\/]+){6}/?.* (change 6 with the integer of your choice, the higher the deeper the crawl will be in terms of slashes in the url)

This regex has to be interpreted with a shift of 2 from the number X declared here (/[^\/]+){X}/?.* , for instance:

• If you crawl www.datafari.com and use (/[^\/]+){2}/?.* , Datafari will only fetch one page, the seed url www.datafari.com/ (and NOT any page exposed at the root), so a depth of 0.
• If you crawl www.datafari.com and use (/[^\/]+){3}/?.* , Datafari will be fetching pages that are directly at the root, such as https://www.francelabs.com/solutions.html, so a depth of 1.
• If you crawl www.datafari.com and use (/[^\/]+){4}/?.* , Datafari will be fetching pages described above, plus those one slash deeper, such as https://www.francelabs.com/en/big_data.html, so a depth of 2.