Introduction

As we have been working on the implementation of a RAG solution into Datafari, we came up with a new feature: “Datafari RagAPI”. RagAPI is a collection of Java classes and methods designed to handle various AI-related processes within Datafari. Currently, RagAPI supports RAG (Retrieval Augmented Generation) and document summarization, with future plans to integrate translation capabilities.

RagAPI is used in two main contexts:

• Search API: Enables RAG functionality within the search engine.

• AiPowered API: Supports both RAG and additional AI-powered features, such as document summarization.

All our AI features can be used calling the proper API endpoint, or by using the AI Assistant tool from Datafari UI.

At the core of RagAPI are LLM Services, a set of classes that act as interfaces between Datafari and external APIs leveraging Large Language Models (LLMs). These services allows integration with third-party AI providers like OpenAI API, as well as Datafari AI Agent, our in-house LLM API solution.

What is RAG?

RAG stands for Retrieval-Augmented Generation. It is the exploitation of a Large Language Model in order to generate the response to a user question or prompt leveraging only some contextual information provided with the prompt (these contextual information can be relevant documents or chunks of documents, coming from sources such as Datafari search results).

Here are some sources, for a better understanding of RAG:

• What Is Retrieval-Augmented Generation aka RAG?

• What is Retrieval-Augmented Generation (RAG)?

• What is retrieval-augmented generation (RAG)?

Classic search VS Vector Search

The “Retrieve” part of the RAG is an important step. In this step, a search is processed to identify a list of document that may contain the wanted information, and extract relevant fragments that can be interpreted by the LLM. In our own terms, the “classic” method is the keywords-based search, implemented in Datafari. The vector search is based on Machine Learning technologies to capture the meaning and the context of unstructured data, converted into digital vectors. The advantage of vector search is to “understand” natural language queries, thus finding more relevant documents, that may not necessarily use the same terms as the ones in the query.

Datafari currently offers three different approaches to RAG retrieval:

• Keyword-based Search (classic BM25): Documents are retrieved using a traditional BM25 Datafari search, followed by a chunking process.

• InMemory Vector Search (hydrid solution): An initial BM25 search is performed, retrieving relevant documents. These documents are then chunked and stored in an "InMemoryEmbeddingsStore." A secondary vector search is executed within this in-memory store to refine the results, and extract a limited number of relevant chunks.

• Solr Vector Search: During indexing, documents are pre-chunked, and each chunk is vectorized. The classic keyword-based search is replaced by a fully vector-based retrieval process, using Text to Vector Solr features.

How does RAG works in Datafari ?

The RAG process can be started through two differents Datafari API endpoints:

• “/search/*” from Search API (GET) (see details in the “Endpoints” section).

• “/rag” from AiPowered API (POST), documented in https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/3619946497.

1. Query reception

A “RAG” query is received from the user, through one of the API endpoints.

2. Retrieval

The first step of the process is to retrieve a list of relevant chunks, using one of the three method precedently listed. Depending on the selected method, search results may need to be processed.

3. Prompting

A list of prompts (including instructions for the model, relevant documents chunks and the user query) is prepared and sent to the LLM External Service.

4. Response Generation

The LLM generates a text response, citing the source he used to find the response to the user query.

5. Response formatting

Datafari will format the webservice response into a standard JSON, attach the relevant sources, and sent it to the user.

Open

What is this external webservice?

Our solution currently supports OpenAI-compatible APIs. For example:

• OpenAI API is a service provided by OpenAI. Its chat completion endpoint can be used to process RAG searches.

• Datafari AI Agent is an experimental solution developped by France Labs. It is a simple Python OpenAI-like API, hosting at least one Large Language Model. This solution currently supports text generation and vector embeddings.

In both cases, we use the LLM to extract a response to the user question from the provided document.

Endpoints

Parameters

The endpoint above handles multiple parameters.

action : Mandatory, must be set to “rag”.

prompt : Mandatory. It contains the prompt or the search written by the user.

format : Optional. This parameters allows the user to define the format of the generated response. Allowed values for this field are "bulletpoint", "text", "stepbystep" or “default”. It can also be left blank or unset.

lang : Optional. The expected language of the response. Requests from DatafariUI should specify the user’s preferred language.

• Allowed values are “en”, “fr”, “it”, “pt”, “pt_br”, “de”, “es”, and “ru”.

• If no language is selected, the API will try to retrieve the logged user’s preferred language.

• If the user is not logged in, English will be used by default.

Response structure

The responses are formatted using the following template:

In this case, the “message” contains the text generated by the Large Language Model. The “documents” section is a JSONArray containing information about the document sent to the LLM and used to retrieve the answer. Each document has an ID (id), URL (url), a title (title) and a content. The content may be the “exactContent” of the document, the “preview_content”, or the Solr “highlightings” depending on the RAG configuration. Some fields may be added in the future developments.

For errors, the content follows the following structure:

• Example of a valid response:

• Example of an error response:

The “message” field is not localized. It only has an informative value. In order to allow translations, we recommend using the “label” thing. Here are the different labels, and the associated message.

• Example of a valid formatted response

Configuration

Multiple configurable parameters can be set in the rag.properties file. This file can be found:

• In git repository:

  datafari-ce/datafari-tomcat/conf-datafari/rag.properties

• On the Datafari server:

  /opt/datafari/tomcat/conf/rag.properties

This file contains the parameters you need to call your LLM solution, enable or disable AI features, and configure their processes.

Global AI/RAG properties

• ai.enable.rag: Set to “true” to enable RAG features (default: true).

• ai.enable.summarization: Enable processing logs (default: false).

Web services parameters

• rag.api.endpoint: The URL of the API you want to call. By default, OpenAI LlmService uses https://api.openai.com/v1/. If you are using Datafari AI Agent, consider using http://[your-aiagent-ws-address]:8888

• rag.api.token: Your API token. Required to use OpenAI services. Please use your own token.

• rag.llm.service: The API you are using. Currently, the only accepted value is "openai" (default: openai).

LLM parameters

• rag.model: The LLM model to be used. Can be left blank to use the API’s default model. (default: gpt-3.5-turbo).

• rag.temperature: Temperature controls the randomness of the text that the LLM generates. With GPT, you can set a Temperature of between 0 and 2 (the default is 1). With others LLM, it can be set between 0 and 1. We recommand setting it to 0. (default: 0)

• rag.maxTokens: Integer value. The maximum number of tokens in the LLM response. (default: 200)

Datafari RAG pre-processing properties

• rag.enable.vector.search: Enable vector search using a local vector store. This significantly increased processing time, but also highly improves the quality of the response. Recommanded, unless you are using Solr Vector Search. (default: true).

• rag.enable.logs: Enable processing logs. Must log to DEBUG level. (default: false).

• rag.enable.chunking: Enable chunking. Highly recommended if vector search is disabled. (default: true).

• rag.maxFiles : Integer value. The maximum number of files that should be included in the process. Allowing too many files decreases performances. (default: 3).

• rag.maxChunks : Integer value. The maximum number of document chunks that should be sent to the LLM when Vector Search is enabled. Allowing too many chunks decreases performances. (default: 5)

• rag.chunk.size : Integer value. The maximum length in character of a chunk. Setting an exceeding value may cause errors or malfunction. (default: 30000)

• rag.max.request.size : Integer value. The maximum total length in character of all the messages that should be handled by the LLM in a single request. Setting an exceeding value may cause errors or malfunction. (default: 40000)

• rag.operator : “AND” or “OR”. This fields defines the operator (q.op in Solr) used in the Search process in Datafari. Using “AND” may increase the relevancy of a response, but significantlty decreases the chance to find relevant sources. (default: OR)

Solr Vector Search

• solr.enable.vector.search : Set to “true” to enable Solr Vector Search into RAG process (default: false).

• solr.embeddings.model : The name of the model uploaded to Solr through a JSON file. (default: default_model).

• solr.vector.field : The Solr field name containing the vector. Must be an existing DenseVectorField, with a dimension that is compatible with the provided model. Datafari comes with a list of commons vector fields (“vector_1536”, “vector_512”…), but you may need to create a new one depending on the embeddings model you are using. (default: vector_1536).

• solr.topK : The expected number of search results (default: 10)

Examples of configuration

Technical specification

Process description

Depending on the configuration and on the Retrieval approach, the global RAG process can take three forms.

Open

Open

Open

1. The client sends a query to the Datafari API, using one of the “RAG” endpoint:

   GET https://{DATAFARI_HOST}/Datafari/rest/v2.0/search/select?q={prompt}&action=rag
   POST https://{DATAFARI_HOST}/Datafari/rest/v2.0/ai/rag

Parameters are extracted from the HTTPS request, and configuration is retrieved from rag.properties.

2. A search query is processed using Search API methods, based on the user prompt, in order to retrieve a list of potentially relevant documents. This search can be either a keyword-based BM25 search, or a Solr Vector Search.

In the first case, the search will return whole documents, that will require chunking to be processed by the LLM.

In the case of Vector Search, Solr will return a number of length-limited excerpts.

3. The “InMemory Vector Search” is managed in the VectorUtils class. It receives results from a BM25 documents, chunk them into short semantic excerpts, embeds them and store them into an “In Memory Embeddings Store”. Then, it runs a vector search using the embedded user query. Finally, it returns a list of N relevant excerpts (N is defined in rag.properties, as rag.maxChunks). Since the excerpts are short, a chunking strategy is not required if “InMemory Vector Search” is enabled.

Enable or disable this step by editing the rag.enable.vector.search property in rag.properties. This step is not related to Solr Vector Search.

4. Documents from BM25 search might be to big to be handled in one time by the LLM. Chunking allows to cut large documents into smaller pieces to process them sequentially.

Enable or disable this step by editing the rag.enable.chunking property in rag.properties.

5. During prompting, the list of documents/snippets is converted into a list of prompts that will be processed by the LLM. Each prompt contains instructions (instructions are defined in the /opt/datafari/tomcat/webapps/Datafari/WEB-INF/classes/prompts folder), documents excerpts, and the user prompt as a question.

If prompts are short enough, they might be sent to the LLM into one single request to improve performances. If that is not the case, we use a “Stuff Chain” method to process all chunks.

6. Our solution is designed to be able to interact with various LLM API. The dispatcher selects the proper connector (LlmService) to interact with the configured LLM API.

A connector is a Java class implementing our LlmService interface. It contains the “invoke” method, taking as parameter a list of prompts, and returning a simple String response from the LLM.

To this day, we one provide one LlmService:

• OpenAILlmService, compatible with OpenAI API and any other OpenAI-like API (including Datafari AI Agent)

7. The selected connector prepares one or multiple HTTP/HTTPS queries, one for each prompt from the list. Then, it calls the external LLM API, and extracts the response.

If the list of prompts contains multiple entries, all the responses are concatenated and sent to the LLM again, to generate a final summary.

8. The response is formatted in JSON and sent back to the user.

Chunking

Most documents stored in the FileShare Solr collection are too large to be processed in a single request by a Large Language Model. To address this, implementing a chunking strategy is essential, allowing us to work with manageable, concise, and contextually relevant text snippets.

The chunking strategy depends on the Retrieval method. The three cases are detailed below.

Prompts

Prompts are a collection of "Message" objects sent to the LLM. Each "Message" contains:

• A role: "user" for the user query and document content, "assistant" for AI-generated messages, or "system" for instructions.

• Content: The body of the message, which may include instructions, the user query, or document content.

If the RAG process needs to manage too many or too large snippets, it may not be able to fit all of the into one single LLM request. In this situation, a prompt strategy is required. Our current approach is the Stuff Chain method, but that is subject to change. Read more about chunking management strategies in the LLM Tranformation Connector documentation.

Below are three prompt chain, associated with a RAG query.

Case 1: All prompts can fit in a single LLM Request	Case 2 : Prompts are to large to be processed at once (Stuff Chain method)	Case 3 : Prompts have to large (Refine method)

Case 1: All prompts can fit in a single LLM Request	Case 2 : Prompts are to large to be processed at once (Stuff Chain method)	Case 3 : Prompts have to large (Refine method)
The LLM is call only once, with the following prompt chain. Open	First, the LLM is called one per chunk, each time with the following Message list: Open Then, the LLM is call one final time: Open	This feature has not been implemented yet, and the prompts have not been tested. First, the LLM is called with the first N chunks (as many chunks as a request can fit) Open Then, it will recursively call the LLM for each chunk (or pack of chunks), with the following Message list: Open

Available LlmServices

An LlmService is a class that implements our “LlmService.java” interface, and acts as an interface between Datafari and an external APIs leveraging Large Language Models (LLMs).

All “LlmService” classes should implement the invoke() method.

The generate() method takes, as parameter, a list of String prompts ready to be sent. All the prompts are sent to the associated LLM API, and a single String response is returned.

Message is a Datafari Java class, with the following attributes:

• String role: The role associated to the Message. Either “user”, “system”, or “assistant”.

• String content: The content of the message.

The LlmService must fulfill multiple tasks:

• Override the generate() method.

• Provide default configuration if relevant (default model, maxToken, specific configuration…)

• Call an external LLM Service, using the proper configuration as defined in the rag.properties file (endpoint temperature, maxTokens, API key…).

• Format the LLM response to a simple String.

• Implement a constructor taking at least a RagConfiguration object as parameter.

OpenAI LLM service

This connector can be used to access OpenAI API, or any other API that uses OpenAI signature, including our Datafari AI Agent . The default model, gpt-3.5-turbo, can be changed by editing the rag.model property.

If you are planning to use your own OpenAI-like solution, edit the rag.api.endpoint property. Default value is https://api.openai.com/v1/

Vector Search

To enhance the relevance of document excerpts sent to the LLM, we have implemented vector search solutions. This machine learning-based approach represents semantic concepts as vectors, offering more accurate results than traditional keyword-based search. Additionally, vector search improves retrieval quality in multilingual datasets, as it relies on semantic meaning rather than exact wording.

InMemory Vector Search

In a first time, we implemented a simple and local vector-store solution, provided by Langchain4j : InMemoryEmbeddingStore.

In this scenario, documents are first retrieved with a keyword BM25 search. Then the are processed by the EmbeddingStoreIngestor : they are chunked, chunks are translated to vectors, and stored in the local vector database.

Then, the user query is embedded and used to retrieve relevant chunks. This solution can be considered as an Hybrid Search approach, since it combines keywords-based search and vector search.

Solr Vector Search

Solr Vector Search uses the new text-to-vector feature provided by Solr 9.8. The purpose is to replace the current BM25 search and the local vectore store by a full vector search solution (and in the future, an hybrid search solution for even more relevant results).

Our VectorUpdateProcessor process all documents that are indexed into the FileShare Solr collection. Documents are chunks, embedded, and store into the VectorMain collection.

Those chunks can know be searched using our new “/vector” handler.

The following query can be used to process a vector search through the API.

• queryrag or q (required) : The user query. The "queryrag" parameter is required by Solr; however, if it is missing, Datafari will automatically populate it with the value of "q".

• topK (optional) : The number of results to return. (default: 10)

Security

Security is a major concern, and was a central element in our technical decisions. One of the main advantages of the Retrieval Augmented Generation is that the LLM only uses the data he is provided to answer a question. As long as we control the data sent to the model, we can prevent any leaks.

The “prompt injection” is a set of technique used to override original instructions in the prompt, through the user input. As our prompt does not contain secret, confidential or structural elements, we consider that it is not a serious issue if a user is able to read it.

Datafari Enterprise Edition provides a security solution, allowing enterprises to set up access restrictions on their files in Datafari. In any case, our RAG solution must not break this security. If security is configured, any user can process a RAG search on the server. However, a search (BM25 or Vector) will be processed through Datafari SearchAPI first to retrieve available documents. If the user is not allowed to see a document, this document will not be retrieved and won’t be sent to the external LLM service. That way, it is impossible for a user to use the RAG tools to retrieve information he should not be able to access.

More information in Datafari Enterprise Edition here.