| Info | ||
|---|---|---|
| ||
Except for particular cases, do not use this content limiter with your Datafari Enterprise Edition solution, because if you are using the available Tika Server Connector, it is already equipped and preconfigured with an optimised content limiter. |
Since the 4.0.0 version of Datafari, introducing ManifoldCF v2.8.1, a new transformation connector is available : the Content limiter.
The purpose of this connector is to truncate the content stream of a crawled file if its size is above the limit configured, instead of ignoring and not indexing the file. This helps to improve the stability of Solr in case the amount of pure text to index is so big that it causes huge CPU and memory load which can lead to an OOM from Solr or the Operating System.
...