Valid from Datafari X.X
Datafari periodically generates monitoring logs about the main Solr core. Those logs are intended to provide general informations on the core contents (like number of indexed doc by types, by source, etc.).
By default, the generation rate is once per hour, and can be currently modified through the java class "com.francelabs.datafari.monitoring.IndexMonitoring".
Here is how monitoring logs look like:
monitoring logs
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:175 - ec6d7ff440530d73e020766403a3e058|2015-11-06T18:14:00.000+0100|40|no|no 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - dd3fb399e47dced897b708d9c6e78d2f|2015-11-06T18:14:00.000+0100|11|docx|extension 2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 039435d1512eb763234a2690fb06c63c|2015-11-06T18:14:00.000+0100|6|pdf|extension 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - 611555203c75e7880c8dc67afd41062d|2015-11-06T18:14:00.000+0100|4|txt|extension 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - 2ce23ed298b044a3eda0a0e695921024|2015-11-06T18:14:00.000+0100|3|png|extension 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - 0fc7bbcad0efcfc2037ae2a2f1fbfe04|2015-11-06T18:14:00.000+0100|2|doc|extension 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - 31f8884b788c7872e5b2a286faa42747|2015-11-06T18:14:00.000+0100|2|html|extension 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - e155fbb2f018035d65553e763e0d8e63|2015-11-06T18:14:00.000+0100|1|gif|extension 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - c9dc854a334e1dc339d6cdfd10cb37cf|2015-11-06T18:14:00.000+0100|1|jar|extension 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - c10e628086bdc9fe4126c5f9c3b222c6|2015-11-06T18:14:00.000+0100|1|jpg|extension 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - 24ca7afe128eb7b33b36c4919e830509|2015-11-06T18:14:00.000+0100|16|en|language 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - 8879abd5830038c4a630d07526857faf|2015-11-06T18:14:00.000+0100|15|fr|language 2015-11-06 0918:1714:5500 MONITORING IndexMonitoring$FileShareMonitoringLog:187184 - dc8834aa358f02cff45e71052dd648c0|2015-11-06T18:14:00.000+0100|30|file|source |
...
They respect a specific format which is:[log_id] | [timestamp] | [num_doc] | [facet_value] | [facet_field]
Let's explain each field:
id which must be[log_id]: the logid. This id IS UNIQUE per
daytime event, per facet value and field
and value, in order to exploit those logs with ELK, and is generated from these 3 values. The reason behind this is the [DEPRECATED] ELK.
[timestamp]: the timestamp of the log. It is formatted and could be different from the log4j timestamp so it is mandatory[num_doc]: the number of documents found according to the facet
represented by the log[facet_value]: the facet value[facet_field]: the facet field used
...
, in relation with the facet value
At each monitoring log generation iteration, a log line is created for each facet value of the selected facet fields, plus a log line for the global number of documents indexed (in that case, the [facet_value] and [facet_field] are both set to "no").
The default selected facet fields are "extension", "language" and "source". One can change the facet fields selected in the class "com.francelabs.monitoring.IndexMonitoring.FileShareMonitoringLog".