[DEPRECATED] ELK logs exploitation

Deprecated as of Datafari 5.2

Please refer to [DEPRECATED] Analytic Stack (Apache Zeppelin)

Starting with Datafari 3.1

When ELK is activated in Datafari, every log file generated by Datafari and its components is pushed to Elasticsearch thanks to Logstash. The configuration file of Logstash is the following: DATAFARI_HOME/elk/logstash/logstash-datafari.conf and the template used to create the Elasticsearch logs indices is DATAFARI_HOME/elk/logstash/templates/datafari-logs-template.json

If you open the logstash-datafari.conf file, you will notice that the configuration also includes the statistics and monitoring logs. It is because logstash can only manage one config file by process and one of the best practice is to limit the number of logstash processes.
This is the reason why you will also notice that the inputs are "marked" with a "type" which can be 'log' or 'exploit' and is used to apply a different parsing and insertion strategy for the inputs.

The other thing to notice in the logstash conf file is that the logs are inserted in a daily index and the logs with a "DEBUG" level are ignored. These lines are used to specify that the logs must be pushed to the index corresponding to the log date (variable %{index_date} which is created earlier in the conf file):

if [type] == "log" {
	if [level] != "DEBUG" {
		elasticsearch {
			hosts => ["localhost:9200"]
			index => "logs-%{index_date}"
			template => "./templates/datafari-logs-template.json"
			template_name => "datafari-logs"
			template_overwrite => true
			flush_size => 100
			idle_flush_time => 1
		}
	}
}

For example, the following log line:

INFO 2016-07-25 16:38:08,936 (localhost-startStop-1) - Tomcat|Datafari|datafari.alerts.AlertsManager|Alert config file successfully read

will be inserted in the Elasticsearch index named "logs-2016.07.25". This behavior has been implemented to facilitate the purge of Elasticsearch when necessary. All you need to do is to delete the indices that you consider too old and all the logs corresponding to the index date will be deleted. This is a quick way, with a low resource consumption, to purge Elasticsearch and it is recommended by the Elasticsearch best practices.

Once the logs are inserted in Elasticsearch, you will be able to consult them and create dashboards with Kibana which make it a powerful tool for monitoring. To create dashboards or visualizations for the logs, simply connect to Kibana either by the direct url http://[HOSTNAME]:5601 or through the admin UI of Datafari.
Go to the indices settings, click on the 'Add New' button, and add the new pattern 'logs-*'

Check that Kibana detects the field 'date' as in the above screenshot and click on 'Create'. You should obtain the following result:

Now you can create visualizations and dashboards, but you can already some visualizations and dashboards made by France Labs by importing the file datafari-logs-kibana.json located in DATAFARI_HOME/elk/save. To import them, go to Setting => Objects in Kibana, click on the import button then select the json file.