Note : You can use the standard configuration on Datafari for crawling a website (pages and files).
If you want to crawl only webpages (without files), you can customize the part of the page to be indexed. This configuration is done for you.
Starting from Datafari X.X version, we add the possibility to crawl easily a website into Datafari.
...