Versions Compared

Old Version 4

changes.mady.by.user Julien Massiera

Saved on

compared with

New Version Current

changes.mady.by.user Cedric

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Valid from Datafari X.X

Datafari periodically generates monitoring logs about the main Solr core. Those logs are intended to provide general informations on the core contents (like number of indexed doc by types, by source, etc.).

By default, the generation rate is once per hour, and can be currently modified through the java class "com.francelabs.datafari.monitoring.IndexMonitoring".
Here is how monitoring logs look like:

monitoring logs
Code Block
languagetext
titlemonitoring logs
linenumberstrue
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:175 - ec6d7ff440530d73e020766403a3e058|2015-11-06T18:14:00.000+0100|40|no|no
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - dd3fb399e47dced897b708d9c6e78d2f|2015-11-06T18:14:00.000+0100|11|docx|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 039435d1512eb763234a2690fb06c63c|2015-11-06T18:14:00.000+0100|6|pdf|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 611555203c75e7880c8dc67afd41062d|2015-11-06T18:14:00.000+0100|4|txt|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 2ce23ed298b044a3eda0a0e695921024|2015-11-06T18:14:00.000+0100|3|png|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 0fc7bbcad0efcfc2037ae2a2f1fbfe04|2015-11-06T18:14:00.000+0100|2|doc|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 31f8884b788c7872e5b2a286faa42747|2015-11-06T18:14:00.000+0100|2|html|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - e155fbb2f018035d65553e763e0d8e63|2015-11-06T18:14:00.000+0100|1|gif|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - c9dc854a334e1dc339d6cdfd10cb37cf|2015-11-06T18:14:00.000+0100|1|jar|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - c10e628086bdc9fe4126c5f9c3b222c6|2015-11-06T18:14:00.000+0100|1|jpg|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 24ca7afe128eb7b33b36c4919e830509|2015-11-06T18:14:00.000+0100|16|en|language
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 8879abd5830038c4a630d07526857faf|2015-11-06T18:14:00.000+0100|15|fr|language
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - dc8834aa358f02cff45e71052dd648c0|2015-11-06T18:14:00.000+0100|30|file|source

...

They respect a specific format which is:
[log_id] |  [timestamp] | [num_doc] | [facet_value] | [facet_field]
 
Let's explain each field:
  • [log_id] : the log 
     id which must be 
    id. This id IS UNIQUE per 
     day
    time event, per facet value and field
     and value, in order to exploit those logs with ELK
    , and is generated from these 3 values. The reason behind this is the [DEPRECATED] ELK.
  • [timestamp] : the timestamp of the log. It is formatted and could be different from the log4j timestamp so it is mandatory
  • [num_doc] : the number of documents found according to the facet
  • [facet_value] : the facet value
     represented by the log
  • [facet_field] : the facet field used
...
  • , in relation with the facet value
At each monitoring log generation iteration, a log line is created for each facet value of the selected facet fields, plus a log line for the global number of documents indexed (in that case, the [facet_value] and [facet_field] are both set to "no").
The default selected facet fields are "extension", "language" and "source". One can change the facet fields selected in the class "com.francelabs.monitoring.IndexMonitoring.FileShareMonitoringLog".