Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

Datafari periodically generates monitoring logs about the main Solr core. Those logs are intended to provide general informations on the core contents (like number of indexed doc by types, by source, etc.).

By default, the generation rate is once per hour, and can be currently modified through the java class "com.francelabs.datafari.monitoring.IndexMonitoring".
Here is how monitoring logs look like:

monitoring logs
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:175 - ec6d7ff440530d73e020766403a3e058|2015-11-06T18:14:00.000+0100|40|no|no
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - dd3fb399e47dced897b708d9c6e78d2f|2015-11-06T18:14:00.000+0100|11|docx|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 039435d1512eb763234a2690fb06c63c|2015-11-06T18:14:00.000+0100|6|pdf|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 611555203c75e7880c8dc67afd41062d|2015-11-06T18:14:00.000+0100|4|txt|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 2ce23ed298b044a3eda0a0e695921024|2015-11-06T18:14:00.000+0100|3|png|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 0fc7bbcad0efcfc2037ae2a2f1fbfe04|2015-11-06T18:14:00.000+0100|2|doc|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 31f8884b788c7872e5b2a286faa42747|2015-11-06T18:14:00.000+0100|2|html|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - e155fbb2f018035d65553e763e0d8e63|2015-11-06T18:14:00.000+0100|1|gif|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - c9dc854a334e1dc339d6cdfd10cb37cf|2015-11-06T18:14:00.000+0100|1|jar|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - c10e628086bdc9fe4126c5f9c3b222c6|2015-11-06T18:14:00.000+0100|1|jpg|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 24ca7afe128eb7b33b36c4919e830509|2015-11-06T18:14:00.000+0100|16|en|language
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 8879abd5830038c4a630d07526857faf|2015-11-06T18:14:00.000+0100|15|fr|language
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - dc8834aa358f02cff45e71052dd648c0|2015-11-06T18:14:00.000+0100|30|file|source

 

They respect a specific format which is:
[log_id] |  [timestamp] | [num_doc] | [facet_value] | [facet_field]

 

Let's explain each field:

  • [log_id] : the log id which must be UNIQUE per day, per facet field and value, in order to exploit those logs with ELK
  • [timestamp] : the timestamp of the log. It is formatted and could be different from the log4j timestamp so it is mandatory
  • [num_doc] : the number of documents found according to the facet
  • [facet_value] : the facet value
  • [facet_field] : the facet field used

 

At each monitoring log generation iteration, a log line is created for each facet value of the selected facet fields, plus a log line for the global number of documents indexed (in that case, the [facet_value] and [facet_field] are both set to "no").
The default selected facet fields are "extension", "language" and "source". One can change the facet fields selected in the class "com.francelabs.monitoring.IndexMonitoring.FileShareMonitoringLog".

  • No labels