Logging


From Datafari 4.6


All the logs generated by Datafari and the other components (Manifoldcf, Solr, ELK) have been formated in order to be exploited by ELK.

Here is the new format :

[level] [timestamp] [thread name] - [process name]|[component name]|[package name]|[message]



It is implemented through a log4j properties file for each Datafari component. The corresponding log4j pattern is the following:

%5p %d{ISO8601} (%t) - [process name]|[component name]|%c{3}|%m%n

To understand the specific log4j patterns used, refer to the official doc: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html


For example, the following log4j conversion pattern has been defined in the log4j.properties file of Tomcat for the Datafari appender:

log4j.appender.datafari.layout.ConversionPattern=%5p %d{ISO8601} (%t) - Tomcat|Datafari|%c{3}|%m%n

This configuration will generate log lines that look like this:

 INFO 2016-07-25 16:38:08,374 (localhost-startStop-1) - Tomcat|Datafari|datafari.startup.UserManagementLauncher|UserManagement info
 INFO 2016-07-25 16:38:08,851 (localhost-startStop-1) - Tomcat|Datafari|service.db.DBContextListerner|Cassandra client initialized successfully
 INFO 2016-07-25 16:38:08,936 (localhost-startStop-1) - Tomcat|Datafari|datafari.alerts.AlertsManager|Alert config file successfully read
 INFO 2016-07-25 16:38:08,945 (localhost-startStop-1) - Tomcat|Datafari|datafari.alerts.AlertsManager|Alert scheduler started


With this kind of format, Datafari can push every generated log line to Elasticsearch thanks to Logstash, and Kibana can be used to visualize and exploit them with efficiency.


List of log files in Datafari :


ComponentLocationConfig fileProduct versionRotationLog filesMax size for each fileMax number of occurences
Apache$DATAFARI_HOME/logs/apache/*.log
EEYes daily


Cassandra$DATAFARI_HOME/logs/cassandra-*.log$DATAFARI_HOME/cassandra/conf/logback.xmlCE/EE Yes :max size

cassandra-gc.log

cassandra_system.log

cassandra_debug/log


10 Mb

10

10

10

10

10

Datafari webapp$DATAFARI_HOME/logs/datafari-*.log$DATAFARI_HOME/tomcat/conf/log4j2.propertiesCE/EE

Yes daily

Yes daily and max size

Yes daily and max size

datafari.log

datafari-stats.log

datafari-monitoring.log

datafari-crawl.log


10 Mb

10 Mb

10 Mb

10 Mb

10

10

10

10

MCF$DATAFARI_HOME/logs/manifoldcf.log

$DATAFARI_HOMEmcf/mcf_home/logging.xml


CE/EEYes dailymanifoldcf.log1010
MCF agent$DATAFARI_HOME/logs/mcf-agent.log

$DATAFARI_HOMEmcf/bin/mcf_crawler_agent.sh

/etc/logrotate


CE/EEYes dailymcf-agent.log1010
PostgreSQL

$DATAFARI_HOME/logs/pgsql.log

$DATAFARI_HOME/logs/postgresql-*.log

$DATAFARI_HOME/pgsql/postgresql.save.confCE/EEYes daily and max sizepgsql.log10 Mb/
Solr$DATAFARI_HOME/logs/solr*.log$DATAFARI_HOME/solr/conf/log4j2.xmlCE/EEYes dailysolr.log1010
Tika server$DATAFARI_HOME/tika-server/logs/tika*.log

$DATAFARI_HOME/tika-server/conf/log4.properties

$DATAFARI_HOME/tika-server/conf/log4child.properties

CE/EE

Yes daily

Yes daily

tika-server.log

tika-server-child.log

10

10

10

10

Tomcat webapp

$DATAFARI_HOME/logs/tomcat*.log

$DATAFARI_HOME/tomcat/conf/server.xmlCE/EEYeslocalhost_access/log1010
Tomcat MCF

$DATAFARI_HOME/logs/tomcat-mcf.log

$DATAFARI_HOME/logs/tomcat-mcf-[0-9]*.log

$DATAFARI_HOME/tomcat-mcf/conf/log4j2.properties.xmlCE/EE

Yes daily

Yes daily

tomcat-mcf.log

datafari-manifoldcf.log

10

10

10

10

ZK Solr$DATAFARI_HOME/logs/zookeeper.log*$DATAFARI_HOME/zookeeper/conf/log4j.propertiesCE/EEYes dailyzookeeper.log1010
ZK MCF$DATAFARI_HOME/logs/zookeeper-mcf.log*$DATAFARI_HOME/zookeeper-mcf/conf/log4j.propertiesCE/EEYes dailyzookeeper-mcf.log1010
ELK - ElasticSearch$DATAFARI_HOME/elk/logs/*.log$DATAFARI_HOME/elk/elasticsearch/config/log4j2.propertiesCE/EEYes daily and max size

elasticsearch*.log

1010
ELK - Logstash$DATAFARI_HOME/elk/logs/*.log$DATAFARI_HOME/elk/logstash/config/log4j2.propertiesCE/EEYes dailylogstash*.log1010
ELK - Kibana$DATAFARI_HOME/elk/logs/kibana.log/etc/logrotateCE/EEYes daily
1010





From Datafari 4.5 :

Datafari 3.0 comes with a revised way of logging. All the logs generated by Datafari and the other components (Manifoldcf, Solr, ELK) have been formated in order to be exploited by ELK.

Here is the new format :

[level] [timestamp] [thread name] - [process name]|[component name]|[package name]|[message]



It is implemented through a log4j properties file for each Datafari component. The corresponding log4j pattern is the following:

%5p %d{ISO8601} (%t) - [process name]|[component name]|%c{3}|%m%n

To understand the specific log4j patterns used, refer to the official doc: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html


For example, the following log4j conversion pattern has been defined in the log4j.properties file of Tomcat for the Datafari appender:

log4j.appender.datafari.layout.ConversionPattern=%5p %d{ISO8601} (%t) - Tomcat|Datafari|%c{3}|%m%n

This configuration will generate log lines that look like this:

 INFO 2016-07-25 16:38:08,374 (localhost-startStop-1) - Tomcat|Datafari|datafari.startup.UserManagementLauncher|UserManagement info
 INFO 2016-07-25 16:38:08,851 (localhost-startStop-1) - Tomcat|Datafari|service.db.DBContextListerner|Cassandra client initialized successfully
 INFO 2016-07-25 16:38:08,936 (localhost-startStop-1) - Tomcat|Datafari|datafari.alerts.AlertsManager|Alert config file successfully read
 INFO 2016-07-25 16:38:08,945 (localhost-startStop-1) - Tomcat|Datafari|datafari.alerts.AlertsManager|Alert scheduler started


With this kind of format, Datafari can push every generated log line to Elasticsearch thanks to Logstash, and Kibana can be used to visualize and exploit them with efficiency.


List of log files in Datafari :


ComponentLocationConfig fileProduct versionRotationLog filesMax size for each fileMax number of occurences
Apache$DATAFARI_HOME/logs/apache/*.log
EENo


Cassandra$DATAFARI_HOME/logs/cassandra-*.log$DATAFARI_HOME/cassandra/conf/logback.xmlCE/EE Yes :max size

cassandra-gc.log

cassandra_system.log

cassandra_debug/log


10 Mb

10

10

10

10

10

Datafari webapp$DATAFARI_HOME/logs/datafari-*.log$DATAFARI_HOME/tomcat/conf/log4j2.propertiesCE/EE

Yes daily

Yes daily and max size

Yes daily and max size

datafari.log

datafari-stats.log

datafari-monitoring.log

datafari-crawl.log


10 Mb

10 Mb

10 Mb

10 Mb

10

10

10

10

MCF$DATAFARI_HOME/logs/manifoldcf.log

$DATAFARI_HOMEmcf/mcf_home/logging.xml


CE/EEYes dailymanifoldcf.log1010
MCF agent$DATAFARI_HOME/logs/mcf-agent.log

$DATAFARI_HOMEmcf/bin/mcf_crawler_agent.sh

/etc/logrotate


CE/EENomcf-agent.log1010
PostgreSQL

$DATAFARI_HOME/logs/pgsql.log

$DATAFARI_HOME/logs/postgresql-*.log

$DATAFARI_HOME/pgsql/postgresql.save.confCE/EEYes daily and max sizepgsql.log10 Mb/
Solr$DATAFARI_HOME/logs/solr*.log$DATAFARI_HOME/solr/conf/log4j2.xmlCE/EEYes dailysolr.log1010
Tika server$DATAFARI_HOME/tika-server/logs/tika*.log

$DATAFARI_HOME/tika-server/conf/log4.properties

$DATAFARI_HOME/tika-server/conf/log4child.properties

CE/EE

Yes daily

Yes daily

tika-server.log

tika-server-child.log

10

10

10

10

Tomcat webapp

$DATAFARI_HOME/logs/tomcat*.log

$DATAFARI_HOME/tomcat/conf/server.xmlCE/EEYeslocalhost_access/log1010
Tomcat MCF

$DATAFARI_HOME/logs/tomcat-mcf.log

$DATAFARI_HOME/logs/tomcat-mcf-[0-9]*.log

$DATAFARI_HOME/tomcat-mcf/conf/log4j2.properties.xmlCE/EE

Yes daily

Yes daily

tomcat-mcf.log

datafari-manifoldcf.log

10

10

10

10

ZK Solr$DATAFARI_HOME/logs/zookeeper.log*$DATAFARI_HOME/zookeeper/conf/log4j.propertiesCE/EEYes dailyzookeeper.log1010
ZK MCF$DATAFARI_HOME/logs/zookeeper-mcf.log*$DATAFARI_HOME/zookeeper-mcf/conf/log4j.propertiesCE/EEYes dailyzookeeper-mcf.log1010
ELK - ElasticSearch$DATAFARI_HOME/elk/logs/*.log$DATAFARI_HOME/elk/elasticsearch/config/log4j2.propertiesCE/EEYes daily and max size

elasticsearch*.log

1010
ELK - Logstash$DATAFARI_HOME/elk/logs/*.log$DATAFARI_HOME/elk/logstash/config/log4j2.propertiesCE/EEYes dailylogstash*.log1010
ELK - Kibana$DATAFARI_HOME/elk/logs/kibana.log/etc/logrotateCE/EEYes daily
1010






Datafari 3.0 comes with a revised way of logging. All the logs generated by Datafari and the other components (Manifoldcf, Solr, ELK) have been formated in order to be exploited by ELK.

Here is the new format :

[level] [timestamp] [thread name] - [process name]|[component name]|[package name]|[message]



It is implemented through a log4j properties file for each Datafari component. The corresponding log4j pattern is the following:

%5p %d{ISO8601} (%t) - [process name]|[component name]|%c{3}|%m%n

To understand the specific log4j patterns used, refer to the official doc: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html


For example, the following log4j conversion pattern has been defined in the log4j.properties file of Tomcat for the Datafari appender:

log4j.appender.datafari.layout.ConversionPattern=%5p %d{ISO8601} (%t) - Tomcat|Datafari|%c{3}|%m%n

This configuration will generate log lines that look like this:

 INFO 2016-07-25 16:38:08,374 (localhost-startStop-1) - Tomcat|Datafari|datafari.startup.UserManagementLauncher|UserManagement info
 INFO 2016-07-25 16:38:08,851 (localhost-startStop-1) - Tomcat|Datafari|service.db.DBContextListerner|Cassandra client initialized successfully
 INFO 2016-07-25 16:38:08,936 (localhost-startStop-1) - Tomcat|Datafari|datafari.alerts.AlertsManager|Alert config file successfully read
 INFO 2016-07-25 16:38:08,945 (localhost-startStop-1) - Tomcat|Datafari|datafari.alerts.AlertsManager|Alert scheduler started


With this kind of format, Datafari can push every generated log line to Elasticsearch thanks to Logstash, and Kibana can be used to visualize and exploit them with efficiency.


List of log files in Datafari :


ComponentLocationConfig fileProduct versionRotationLog filesMax size for each fileMax number of occurences
Apache$DATAFARI_HOME/logs/apache/*.log
EENo


Cassandra$DATAFARI_HOME/logs/cassandra-*.log$DATAFARI_HOME/cassandra/conf/logback.xmlCE/EE Yes :max size

cassandra-gc.log

cassandra_system.log

cassandra_debug/log


20 Mb

20

20

10

20

20

Datafari webapp$DATAFARI_HOME/logs/datafari-*.log$DATAFARI_HOME/tomcat/conf/log4j2.propertiesCE/EE

Yes daily

Yes daily and max size

Yes daily and max size

datafari.log

datafari-stats.log

datafari-monitoring.log

datafari-crawl.log


/

100 Mb

100 Mb

100 Mb

/

10

10

10

MCF$DATAFARI_HOME/logs/manifoldcf.log

$DATAFARI_HOMEmcf/mcf_home/logging.xml


CE/EEYes dailymanifoldcf.log//
MCF agent$DATAFARI_HOME/logs/mcf-agent.log

$DATAFARI_HOMEmcf/bin/mcf_crawler_agent.sh

/etc/logrotate


CE/EENomcf-agent.log//
PostgreSQL

$DATAFARI_HOME/logs/pgsql.log

$DATAFARI_HOME/logs/postgresql-*.log

$DATAFARI_HOME/pgsql/postgresql.save.confCE/EEYes daily and max sizepgsql.log10 Mb/
Solr$DATAFARI_HOME/logs/solr*.log$DATAFARI_HOME/solr/conf/log4j2.xmlCE/EEYes dailysolr.log//
Tika server$DATAFARI_HOME/tika-server/logs/tika*.log

$DATAFARI_HOME/tika-server/conf/log4.properties

$DATAFARI_HOME/tika-server/conf/log4child.properties

CE/EE

Yes daily

Yes daily

tika-server.log

tika-server-child.log

/

/

/

/

Tomcat webapp

$DATAFARI_HOME/logs/tomcat*.log

$DATAFARI_HOME/tomcat/conf/server.xmlCE/EEYeslocalhost_access/log//
Tomcat MCF

$DATAFARI_HOME/logs/tomcat-mcf.log

$DATAFARI_HOME/logs/tomcat-mcf-[0-9]*.log

$DATAFARI_HOME/tomcat-mcf/conf/log4j2.properties.xmlCE/EE

Yes daily

Yes daily

tomcat-mcf.log

datafari-manifoldcf.log

/

/

/

/

ZK Solr$DATAFARI_HOME/logs/zookeeper.log*$DATAFARI_HOME/zookeeper/conf/log4j.propertiesCE/EEYes dailyzookeeper.log//
ZK MCF$DATAFARI_HOME/logs/zookeeper-mcf.log*$DATAFARI_HOME/zookeeper-mcf/conf/log4j.propertiesCE/EEYes dailyzookeeper-mcf.log//
ELK - ElasticSearch$DATAFARI_HOME/elk/logs/*.log$DATAFARI_HOME/elk/elasticsearch/config/log4j2.propertiesCE/EEYes daily and max size

elasticsearch*.log

256 Mb/
ELK - Logstash$DATAFARI_HOME/elk/logs/*.log$DATAFARI_HOME/elk/logstash/config/log4j2.propertiesCE/EEYes dailylogstash*.log//
ELK - Kibana$DATAFARI_HOME/elk/logs/*.log/CE/EENo