Core Monitoring logs

Owned by Cedric
Last updated: 11 Apr, 2024 by Cedric
2 min read

Valid from Datafari X.X

Datafari periodically generates monitoring logs about the main Solr core. Those logs are intended to provide general informations on the core contents (like number of indexed doc by types, by source, etc.).

By default, the generation rate is once per hour, and can be currently modified through the java class "com.francelabs.datafari.monitoring.IndexMonitoring".
Here is how monitoring logs look like:

monitoring logs
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:175 - ec6d7ff440530d73e020766403a3e058|2015-11-06T18:14:00.000+0100|40|no|no 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - dd3fb399e47dced897b708d9c6e78d2f|2015-11-06T18:14:00.000+0100|11|docx|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 039435d1512eb763234a2690fb06c63c|2015-11-06T18:14:00.000+0100|6|pdf|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 611555203c75e7880c8dc67afd41062d|2015-11-06T18:14:00.000+0100|4|txt|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 2ce23ed298b044a3eda0a0e695921024|2015-11-06T18:14:00.000+0100|3|png|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 0fc7bbcad0efcfc2037ae2a2f1fbfe04|2015-11-06T18:14:00.000+0100|2|doc|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 31f8884b788c7872e5b2a286faa42747|2015-11-06T18:14:00.000+0100|2|html|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - e155fbb2f018035d65553e763e0d8e63|2015-11-06T18:14:00.000+0100|1|gif|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - c9dc854a334e1dc339d6cdfd10cb37cf|2015-11-06T18:14:00.000+0100|1|jar|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - c10e628086bdc9fe4126c5f9c3b222c6|2015-11-06T18:14:00.000+0100|1|jpg|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 24ca7afe128eb7b33b36c4919e830509|2015-11-06T18:14:00.000+0100|16|en|language 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 8879abd5830038c4a630d07526857faf|2015-11-06T18:14:00.000+0100|15|fr|language 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - dc8834aa358f02cff45e71052dd648c0|2015-11-06T18:14:00.000+0100|30|file|source

They respect a specific format which is:
[log_id] |  [timestamp] | [num_doc] | [facet_value] | [facet_field]

Let's explain each field:

  • [log_id] : the log id. This id IS UNIQUE per time event, per facet value and field, and is generated from these 3 values. The reason behind this is the [DEPRECATED] ELK.

  • [timestamp] : the timestamp of the log. It is formatted and could be different from the log4j timestamp so it is mandatory

  • [num_doc] : the number of documents found according to the facet

  • [facet_value] : the facet value

  • [facet_field] : the facet field used, in relation with the facet value

At each monitoring log generation iteration, a log line is created for each facet value of the selected facet fields, plus a log line for the global number of documents indexed (in that case, the [facet_value] and [facet_field] are both set to "no").
The default selected facet fields are "extension", "language" and "source". One can change the facet fields selected in the class "com.francelabs.monitoring.IndexMonitoring.FileShareMonitoringLog".