HTTPS / SSL Protocol - Enterprise Edition

Datafari enterprise edition comes out of the box with the possibility to enable the SSL protocol. This protocol allows to strongly encrypt the data transfert between different systems.

The protocol is used on Datafari, Solr, and Kibana which are, according to us, the critical parts of the product concerning the data. Unfortunately, Elasticsearch does not provide a simple way to use the SSL protocol , so either you have to purchase their "Shield" extension which provides a very easy way to implement security for ES, either you have to manually configure a proxy server.

The following documentation describes how the SSL protocol has been implemented and how you can configure it to fits with your needs.

The first thing to take into consideration is that Datafari is using a single self signed certificate having "localhost" as CN (Common Name). This is the reason why the first thing you should do is to replace this certificate by your/yours.

To do this, let's begin to see where the certificate is located. In the installation directory of Datafari you will find a folder named "ssl-keystore" which contains 5 files:

• datafari-cert.csr : this is the self signed certificate used by Datafari. It is extracted from the 'jks' keystore so it is a 'JAVA' format. It is used by ManifoldCF to push documents to Solr
• datafari-cert.pem : this is the self signed certificate used by Datafari, but it is extracted from the 'p12' keystore so it is a 'PEM' format. It is used by Kibana
• datafari-key.pem : this is the private key associated with the certificate. It is not encrypted and in the 'PKCS12/PEM' format. It is used by Kibana
• datafari-keystore.jks : it is the JAVA keystore of Datafari. It is used by Datafari and Solr
• datafari-keystore.p12 : it is the equivalent of the JAVA keystore of Datafari but in the 'PKCS12/PEM' format
  
  

Two different versions of the keystore and the certificate have been generated to cover the JAVA and PKCS12 formats. The reason is that some applications only support PKCS12/PEM format, like Kibana for example. 

Now how those files are used ? Let explain it for each application

1. Datafari
   To begin with, let's be clear, when speaking of Datafari we are speaking of Tomcat. As it is the container of the Datafari webapp, Tomcat is configured to force the SSL protocol. If you look at the 'server.xml' file located in '{Datafari_Install_Dir}/tomcat/conf' and search for the connector of the 8433 port, you will find these lines:

   <Connector
              protocol="org.apache.coyote.http11.Http11Protocol"
              port="8443"
              scheme="https" secure="true" SSLEnabled="true"
              keystoreFile="../ssl-keystore/datafari-keystore.jks" keystorePass="DataFariAdmin"
              clientAuth="false" sslProtocol="TLS"/>

   They tell Tomcat to activate the SSL protocol on the 8433 port and provide the keystore to use, along with its password. The provided keystore of Datafari enterprise contains a self-signed certificate which has 'localhost' as CommonName (CN) and Tomcat uses it to claim itself when a client connects to it. The CN is used to identify the server and is compared to the domain name that the client has typed to reach the server. That means that with the out of the box Datafari enterprise edition, the SSL connection will only work for the localhost domain. To change this you can import into the keystore a certificate with a CN corresponding to the domain name you want to use for your Datafari server (or use your own keystore, but only a certificate's CN matters for the domain).
   
   Now to force the client to be automatically redirected to the SSL protocol even if he has entered the standard address (8080 port by default), changes have been made into the 'web.xml' file located in '{Datafari_Install_Dir}/tomcat/webapps/Datafari/WEB-INF/'. Search for the security constraint concerning the 'Datafari' web resource:

   <security-constraint>
       <web-resource-collection>
           <web-resource-name>Datafari</web-resource-name>
           <url-pattern>/*</url-pattern>
       </web-resource-collection>
   
       <user-data-constraint>
           <transport-guarantee>CONFIDENTIAL</transport-guarantee>
       </user-data-constraint>
   </security-constraint>

   This rule which concerns every url pattern, tells Tomcat that the data exchanged between the client and the server must be CONFIDENTIAL. In other terms the SSL protocol is mandatory.
   
   
   We saw how SSL is implemented between the client browser and Datafari but there is another configuration for Tomcat. Indeed, you will notice that to connect to Datafari you will have either to put one of the two certificates in your browser truststore, or to add a security exception, otherwise it will be impossible for you to access Datafari. Solr and Kibana use the same protocol with the same certificate, so when you will use Datafari, some requests will be sent to these applications, Tomcat will have to trust their certificate in order to allow the requests. This is the reason why you will find into the 'set-datafari-env.sh' script, located in '{Datafari_Install_Dir}/bin/', a JAVA_OPTS variable that references the 'datafari-keystore.jks' as the trustStore:

   export JAVA_OPTS="${JAVA_OPTS} -Djavax.net.ssl.trustStore=${DATAFARI_HOME}/ssl-keystore/datafari-keystore.jks -Djavax.net.ssl.trustStorePassword=DataFariAdmin"

   The trustStore is used to identifies the trusted certificates. When a request will be sent to Solr or Kibana, both applications will provide their certificate info and Tomcat will use the trustStore to determine if the certificate is trusted or not.

   For this reason, if you decide to use another keystore or certificate than the one provided, do not forget either to reference the correct keystore as trustStore in the set-datafari-env.sh script, or to add the new certificate to the existing datafari-keystore.jks

   
   
   

2. Solr
   Activate the SSL protocol for Solr is quite simple. Search for 'SOLR_SSL' in the solr.in.sh file located in '{Datafari_Install_Dir}/solr/bin/' and you should find these lines:

   SOLR_SSL_KEY_STORE=${DATAFARI_HOME}/ssl-keystore/datafari-keystore.jks
   SOLR_SSL_KEY_STORE_PASSWORD=DataFariAdmin
   SOLR_SSL_TRUST_STORE=${DATAFARI_HOME}/ssl-keystore/datafari-keystore.jks
   SOLR_SSL_TRUST_STORE_PASSWORD=DataFariAdmin
   SOLR_SSL_NEED_CLIENT_AUTH=false
   SOLR_SSL_WANT_CLIENT_AUTH=false

   So if you want to use your own keystore, replace theses values with your own ones, but do not change the SOLR_SSL_NEED_CLIENT_AUTH and SOLR_SSL_WANT_CLIENT_AUTH parameter values.
   
   
   

3. ManifoldCF
   The SSL protocol is not activated for MCF but as Solr is using this protocol, the configuration of the DatafariSolr output connector has been modified to specify the https protocol and add the certificate to the trustStore of MCF. Without this configuration, MCF will not trust Solr and all the requests to push crawled documents to Solr will be rejected:
   
   
   
   So if you decide to use your own certificate, do not forget to import it into the SSL trust certificate list of the DatafariSolr output connector. To do this, go to the admin UI of Datafari, Connectors, MCF Administration, List Output Connections and then edit the DatafariSolr output connector to import your SSL certificate in the 'Server' tab.
   
   

4. Zookeeper
   Zookeeper is configured by default to use the HTTP protocol. To enable the HTTPS protocol you have to run the following command:
   

   /opt/datafari/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd clusterprop -name urlScheme -val https

   
   
   

5. Kibana
   The Kibana configuration to activate the SSL protocol is as simple as the Solr one. Search for 'server.ssl' in the 'kibana.yml' file located in '{Datafari_Install_Dir}/elk/kibana/config/':

   server.ssl.enabled: true
   server.ssl.cert: /opt/datafari/ssl-keystore/datafari-cert.pem
   server.ssl.key: /opt/datafari/ssl-keystore/datafari-key.pem

   Here the difference is that Kibana only accepts the 'PKCS12/PEM' format for the certificate and its private key. So if you use your own certificate, you will have to generate it and its associated NON ENCRYPTED private key (otherwise you will not be able to import it) in this format, thanks to a tool like openssl.

Now you have a better understanding of the behavior of Datafari concerning the SSL protocol. As said in the beginning of this documentation, Elasticsearch does not provide a simple way to enable the SSL. If you don't want to purchase their 'Shield' extension, the best approach to implement SSL is to create a proxy (apache proxy for example), and redirect every query addressed to ES to force the protocol. You can contact France Labs in case you need help for this.