Versions Compared

Old Version 18

changes.mady.by.user Cedric

Saved on

compared with

New Version 19

changes.mady.by.user Olivier Tavard

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Warning

First of all : Do not underestimate the importance of SWAP memory !  Be sure your SWAP fits the recommandations according to your physical memory ! 

We recommend a minimium minimum of
- 1,5 time the amount of RAM as SWAP memory . for servers that have less than 32 GB of RAM
- 1 time the amount of RAM as SWAP memory for servers that have 32 GB of RAM and more

Note that if you server is properly sized in terms of RAM, you should almost never see any usage of SWAP. This recommended mininum minimum amount is a safety measure in case you end up in scenarii where you push your Datafari to the limit of your physical hardware.

...

  1. Crawl

    During a crawl phase, the component that will need a lot of RAM is Tika, because it extracts the content of documents and needs, for certain doc types, to fully load them in memory. Depending on your job configuration, Tika may be :

    • Used in MCF if your job is configured to use the "TikaOCRTikaServerRmetaConnector"  transformation connector 

    • Used in Solr if your job is configured to use the "DatafariSolr" output connector instead of the "DatafariSolrNoTika" output connector

    • Used in its own JVM if your job is configured to use the "TikaServer" transformation connector (only available in the Enterprise edition) 

    If you used the Simplified MCF UI of Datafari to create your job, it is automatically configured with the "TikaOCR" connector for the Community Edition, and the "TikaServer" connector for the Enterprise Edition.

    Knowing this, you will need to allocate more RAM to the component that handle Tika to ensure the stability of your crawls. Tika needs at least 5GB to be stable. So if you use Tika into MCF or into Solr, you will need to add 5GB to the default configuration of those components.

  2. Search

    During the search phase a lot of RAM may be used by Solr to improve performances. Solr uses its own JVM allocated RAM but it also relies on the system cache to perform searches, so it is important to NOT allocate all the available physical memory to Solr or any other Datafari component to ensure best performances ! 

    We recommend to allocate between at least 1GB of RAM and a maximum of 12GB of RAM to Solr depending on the available RAM of your Server. For best search performances, try to let a number of un-allocated RAM that matches your Solr index size (size of the DATAFARI_HOME/solr/solr_home directory). 

...