Logging
- Olivier Tavard
- Julien Massiera
- Cedric
From Datafari 4.6
All the logs generated by Datafari and the other components (Manifoldcf, Solr, ELK) have been formated in order to be exploited by ELK.
Here is the new format :
[level] [timestamp] [thread name] - [process name]|[component name]|[package name]|[message]It is implemented through a log4j properties file for each Datafari component. The corresponding log4j pattern is the following:
%5p %d{ISO8601} (%t) - [process name]|[component name]|%c{3}|%m%nTo understand the specific log4j patterns used, refer to the official doc: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html
For example, the following log4j conversion pattern has been defined in the log4j.properties file of Tomcat for the Datafari appender:
log4j.appender.datafari.layout.ConversionPattern=%5p %d{ISO8601} (%t) - Tomcat|Datafari|%c{3}|%m%nThis configuration will generate log lines that look like this:
With this kind of format, Datafari can push every generated log line to Elasticsearch thanks to Logstash, and Kibana can be used to visualize and exploit them with efficiency.
List of log files in Datafari :
Component | Location | Config file | Product version | Rotation | Log files | Max size for each file | Max number of occurences |
|---|---|---|---|---|---|---|---|
Apache | $DATAFARI_HOME/logs/apache/*.log | Â | EE | Yes daily | Â | Â | Â |
Cassandra | $DATAFARI_HOME/logs/cassandra-*.log | $DATAFARI_HOME/cassandra/conf/logback.xml | CE/EE | Â Yes :max size | cassandra-gc.log cassandra_system.log cassandra_debug/log | 10 Mb 10 10 | 10 10 10 |
Datafari webapp | $DATAFARI_HOME/logs/datafari-*.log | $DATAFARI_HOME/tomcat/conf/log4j2.properties | CE/EE | Yes daily Yes daily and max size Yes daily and max size | datafari.log datafari-stats.log datafari-monitoring.log datafari-crawl.log | 10 Mb 10 Mb 10 Mb 10 Mb | 10 10 10 10 |
MCF | $DATAFARI_HOME/logs/manifoldcf.log | $DATAFARI_HOMEmcf/mcf_home/logging.xml | CE/EE | Yes daily | manifoldcf.log | 10 | 10 |
MCF agent | $DATAFARI_HOME/logs/mcf-agent.log | $DATAFARI_HOMEmcf/bin/mcf_crawler_agent.sh /etc/logrotate | CE/EE | Yes daily | mcf-agent.log | 10 | 10 |
PostgreSQL | $DATAFARI_HOME/logs/pgsql.log $DATAFARI_HOME/logs/postgresql-*.log | $DATAFARI_HOME/pgsql/postgresql.save.conf | CE/EE | Yes daily and max size | pgsql.log | 10 Mb | / |
Solr | $DATAFARI_HOME/logs/solr*.log | $DATAFARI_HOME/solr/conf/log4j2.xml | CE/EE | Yes daily | solr.log | 10 | 10 |
Tika server | $DATAFARI_HOME/tika-server/logs/tika*.log | $DATAFARI_HOME/tika-server/conf/log4.properties $DATAFARI_HOME/tika-server/conf/log4child.properties | CE/EE | Yes daily Yes daily | tika-server.log tika-server-child.log | 10 10 | 10 10 |
Tomcat webapp | $DATAFARI_HOME/logs/tomcat*.log | $DATAFARI_HOME/tomcat/conf/server.xml | CE/EE | Yes | localhost_access/log | 10 | 10 |
Tomcat MCF | $DATAFARI_HOME/logs/tomcat-mcf.log $DATAFARI_HOME/logs/tomcat-mcf-[0-9]*.log | $DATAFARI_HOME/tomcat-mcf/conf/log4j2.properties.xml | CE/EE | Yes daily Yes daily | tomcat-mcf.log datafari-manifoldcf.log | 10 10 | 10 10 |
ZK Solr | $DATAFARI_HOME/logs/zookeeper.log* | $DATAFARI_HOME/zookeeper/conf/log4j.properties | CE/EE | Yes daily | zookeeper.log | 10 | 10 |
ZK MCF | $DATAFARI_HOME/logs/zookeeper-mcf.log* | $DATAFARI_HOME/zookeeper-mcf/conf/log4j.properties | CE/EE | Yes daily | zookeeper-mcf.log | 10 | 10 |
ELK - ElasticSearch | $DATAFARI_HOME/elk/logs/*.log | $DATAFARI_HOME/elk/elasticsearch/config/log4j2.properties | CE/EE | Yes daily and max size | elasticsearch*.log | 10 | 10 |
ELK - Logstash | $DATAFARI_HOME/elk/logs/*.log | $DATAFARI_HOME/elk/logstash/config/log4j2.properties | CE/EE | Yes daily | logstash*.log | 10 | 10 |
ELK - Kibana | $DATAFARI_HOME/elk/logs/kibana.log | /etc/logrotate | CE/EE | Yes daily | Â | 10 | 10 |