Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Datafari periodically generates monitoring logs about the main Solr core. Those logs are intended to provide general informations on the core contents (like number of indexed doc by types, by source, etc.).

By default, the generation rate is once per hour, and can be currently modified through the java class "com.francelabs.datafari.monitoring.IndexMonitoring".
Here is how monitoring logs look like:

monitoring logs
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:175 - ec6d7ff440530d73e020766403a3e058|2015-11-06T18:14:00.000+0100|40|no|no
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - dd3fb399e47dced897b708d9c6e78d2f|2015-11-06T18:14:00.000+0100|11|docx|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 039435d1512eb763234a2690fb06c63c|2015-11-06T18:14:00.000+0100|6|pdf|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 611555203c75e7880c8dc67afd41062d|2015-11-06T18:14:00.000+0100|4|txt|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 2ce23ed298b044a3eda0a0e695921024|2015-11-06T18:14:00.000+0100|3|png|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 0fc7bbcad0efcfc2037ae2a2f1fbfe04|2015-11-06T18:14:00.000+0100|2|doc|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 31f8884b788c7872e5b2a286faa42747|2015-11-06T18:14:00.000+0100|2|html|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - e155fbb2f018035d65553e763e0d8e63|2015-11-06T18:14:00.000+0100|1|gif|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - c9dc854a334e1dc339d6cdfd10cb37cf|2015-11-06T18:14:00.000+0100|1|jar|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - c10e628086bdc9fe4126c5f9c3b222c6|2015-11-06T18:14:00.000+0100|1|jpg|extension
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 24ca7afe128eb7b33b36c4919e830509|2015-11-06T18:14:00.000+0100|16|en|language
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - 8879abd5830038c4a630d07526857faf|2015-11-06T18:14:00.000+0100|15|fr|language
2015-11-06 18:14:00 MONITORING IndexMonitoring$FileShareMonitoringLog:184 - dc8834aa358f02cff45e71052dd648c0|2015-11-06T18:14:00.000+0100|30|file|source

 

They respect a specific format which is:
[log_id] |  [timestamp] | [num_doc] | [facet_value] | [facet_field]

 

Let's explain each field:

  • [log_id] : the log id. This id IS UNIQUE per time event, per facet value and field. The time event, which is "daily" by default, correspond to the unity that will be used to visualize data in Kibana. For example, with the default "daily" unity, you will have one Elasticsearch document by facet value and field, by day. If you set the time event unity to "hourly" you will have one Elasticsearch document by facet value and field by hour. The time event unity can be changed in the code. 
  • [timestamp] : the timestamp of the log. It is formatted and could be different from the log4j timestamp so it is mandatory
  • [num_doc] : the number of documents found according to the facet
  • [facet_value] : the facet value
  • [facet_field] : the facet field used, in relation with the facet value

 

At each monitoring log generation iteration, a log line is created for each facet value of the selected facet fields, plus a log line for the global number of documents indexed (in that case, the [facet_value] and [facet_field] are both set to "no").
The default selected facet fields are "extension", "language" and "source". One can change the facet fields selected in the class "com.francelabs.monitoring.IndexMonitoring.FileShareMonitoringLog".

  • No labels