[DEPRECATED] Custom ELK stats

By default, Datafari generates two kinds of logs that are meant to provide data to the ELK stack in order to create visualizations and dashboard : 

• The Core Monitoring logs that provide index related data

• The Statistics logs that provide search usages related data

The Kibana visualizations and dashboards you can create with a default installation of Datafari are limited to the data provided by those two kind of logs. 

So to add more data to the ELK stack, you will need either to modify those logs or to create your own logs.

1. Modify the existing logs
   The Core Monitoring logs are generated by the com.francelabs.datafari.monitoring.IndexMonitoring class. This class runs a scheduled thread that will query the Solr index each day with some facet queries and format the results into log entries. So it is in this class that you should add your own additionnal facet queries and modify the log format if needed in case you want to add more data.
   
   The Statistics logs are generated by the com.francelabs.datafari.statistics.StatsPusher class that is called whenever a user performs an action in the search view (search, click on result/page/facet). This class contains two main methods:
   - pushDocument : this method is called by the com.francelabs.datafari.servlets.URL servlet which is triggered when a user clicks on a result in the search view
   - pushQuery : this method is called by the com.francelabs.datafari.servlets.SearchProxy servlet which is triggered when a user performs a search in the search view
   In each method, the com.francelabs.datafari.statistics.StatsUtils class is used to format and produce the logs.
   So, you can either modify the existing logs to add more data or produce your own logs in the StatsPusher class.
   
   To add more data (ie Solr fields) to the logs generated from the com.francelabs.datafari.statistics.StatsPusher class or com.francelabs.datafari.servlets.SearchProxy servlet, you will need to add the wanted fields to the queries field list parameter. This can be done in the DATAFARI_HOME/tomcat/webapps/Datafari/js/main.js. Add the desired fields in the existing list :

   Manager.store.addByValue("fl", 'title,url,id,extension,preview_content');

   Once this modification done, you will be able to access those fields values in the query responses and so, add the to the logs.
   
   
   
   

2. Create your own logs 
   If the logs you want to create are meant to be generated on a user action in the search view, we strongly recommend you to generate them either from the com.francelabs.datafari.servlets.URL servlet (when a user clicks on a result) or the com.francelabs.datafari.servlets.SearchProxy servlet (when a user performs a search). 
   
   If the logs you want to create are meant to extract infos (fields etc.) from a Solr index at a defined frequency, we strongly suggest you to take example on the com.francelabs.datafari.monitoring.IndexMonitoring and com.francelabs.datafari.monitoring.IndexMonitoringLauncher classes.

Whatever choice you will do, you you create or add new data to existing log, you will need to do some other modifications in order to take them in account into ELK :

• Update the logstash conf

• Update/create logstash templates

• Update/create index patterns in Kibana

1. Update the logstash conf
   In DATAFARI_HOME/elk/logstash/ you will need to edit the file logstash-datafari.conf. This file contains all the instructions to retrieve log files (input section), how to parse them (filter section) and where to put them (output section). You can rely on the official logstash documentation to understand how to update the configuration.
   
   
   

2. Update/create logstash templates
   If you have modified an existing log format or created one, you will also need to update/create the logstash templates. These templates are in fact Elasticsearch templates, they are used to tell Elasticsearch how to index the incoming data for a specific index. Depending on the Elasticsearch index the logs are pushed (refer to the "output" section of the logstash-datafari.conf file), you will need to update/create the corresponding template to handle the data. Check the official Elasticsearch documentation for more details about templates.
   
   
   

3. Update/create index patterns in Kibana
   Once you have updated or created an Elasticsearch template, you will need to add/refresh it in Kibana. To do so, simply connect to Kibana, and, in the "Management" => "Index patterns" section, create a new pattern in case you created a new template/index, or update an existing index pattern by selecting it in the list and clicking on the refresh button 

   Open

   

    :
   
   
   

   Open