Introduction

As we have been working on the implementation of a RAG (Retrieval Augmented Generation) solution into Datafari, we came up with a new feature: “Datafari RagAPI”. RagAPI is a collection of Java classes and methods designed to handle RAG-related processes within Datafari. For more AI-related features, see also AI Powered Datafari API .

RAG processes can be triggered from two different API contexts:

• AiPowered API: Supports both RAG and additional AI-powered features, such as document summarization.

• Search API: Enables RAG functionality within the search engine. RAG through Search API is deprecated.

All our AI features can be used calling the proper API endpoint, or by using the AI chatbot widget available on Datafari UIv2.

At the core of RagAPI are LLM Services, a set of classes that act as interfaces between Datafari and external APIs leveraging Large Language Models (LLMs). These services allow integration with third-party AI providers like OpenAI API, as well as Datafari AI Agent, our in-house LLM (and embedding models) API solution.

What is RAG?

RAG stands for Retrieval-Augmented Generation. It is the exploitation of a Large Language Model in order to generate the response to a user question or prompt leveraging (or potentially limiting to) extra contextual information provided with the prompt (these contextual information can be relevant documents or chunks of documents, coming from sources such as Datafari search results). In our Datafari case, we purposedly ask the LLM to restrict its answer to the knowledge contained in the extra contextual information (or at least we try to).

Here are some sources, for a better understanding of RAG:

• https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/

• https://cloud.google.com/use-cases/retrieval-augmented-generation

• https://research.ibm.com/blog/retrieval-augmented-generation-RAG

Classic search (BM25) VS Vector Search

The “Retrieval” part of the RAG is an important step. In this step, a search is processed to identify a list of documents that may contain the desired information, and extract relevant fragments that can be interpreted by the LLM. In our own terms, the “classic” method is the keywords-based search, implemented in Datafari. The vector search is based on Machine Learning algorithms to capture the meaning and the context of unstructured data, converted into vectors. The advantage of vector search is to “understand” natural language queries, thus finding more relevant documents, that may not necessarily use the same terms as the ones in the query.

Datafari currently offers two different approaches to RAG retrieval:

• Keyword-based Search (classic BM25): Full documents are retrieved using a traditional BM25 Datafari search, followed by a chunking process.

• Solr Vector Search: During indexing, documents are pre-chunked, and each chunk is vectorized. The classic keyword-based search is replaced by a fully vector-based retrieval process, using Text to Vector Solr features. Short size chunks are returned, instead of whole documents.

How does RAG work in Datafari ?

The RAG process can be started through two different Datafari API endpoints:

• “/rag” from AiPowered API (POST), documented in AI Powered Datafari API.

• “/search/*” from Search API (GET) (see details in the “Endpoints” section). This endpoint is deprecated. The /rag endpoint is the recommended way.

1. Query reception

A “RAG” query is received from the user, through one of the API endpoints.

2. History retrieval (optional)

If “chat memory” is enabled, the chat history is retrieved from the request to be used in the prompts.

3. Query rewriting (optional)

If “query rewriting” is enabled, the search query is rewritten by the LLM before the Solr search (source retrieval). The rewritten query is only used during the retrieval step. The initial user query is still used in RAG process, in the “Prompting” step, and provided to the LLM as context for response generation. If “chat memory” is enabled, the conversation history is used for query rewriting.

4. Source retrieval

Documents are retrieved from Solr using Datafari search. The retrieval process can use Vector Search technology, or classic BM25 Search. If the “query rewriting” feature is enabled, the rewritten query is used for the search. Otherwise, the initial user query used.

5. Chunking

Any document content (or document extract, in case of vector search) larger than the maximum chunk size defined in configuration is chunked into smaller pieces. Each piece is called a “chunk”.

6. Prompting

A list of prompts (including instructions for the model, relevant documents chunks and the user query) is prepared and sent to the LLM External Service.

6. Response Generation

The LLM generates a text response, citing the sources it used to generate the response to the user query.

7. Response formatting

Datafari will format the webservice response into a standard JSON, attach the relevant sources, and send it to the user.

Open

Simplified view of the RAG process with Vector Search

LLM external webservice?

Our solution currently supports OpenAI-compatible APIs. For example:

• OpenAI API is a service provided by OpenAI. Its chat completion endpoint can be used to process RAG searches.

• Datafari AI Agent is an experimental solution developped by France Labs. It is a Python-based OpenAI-like API, hosting at least one Model (LLM for the RAG case, but can also host an embedding model for vector search). This solution currently supports text generation and vector embeddings.

In both cases, we use the LLM to generate a response to the user question, using the provided documents chunks.

Endpoints

Configuration

Via the Admin UI

The easiest and fastest way to configure RAG and other AI-powered features is to use the dedicated page on the Admin interface. This page can be found in the section Extra Functionalities > RAG & AI configuration. See the section below for more information about each parameter.

Open

Open

Via properties file

Configuration properties related to RAG and other AI features in Datafari are stored in the rag.properties file. Those can be directly edited without using the dedicated AdminUI. This file can be found:

• In git repository:

  datafari-ce/datafari-tomcat/conf-datafari/rag.properties

• On the Datafari server:

  /opt/datafari/tomcat/conf/rag.properties

Technical specification

Process description

Depending on the configuration and on the Retrieval approach, the global RAG process can take three forms.

Open

Open

1. The client sends a query to the Datafari API, using the POST /rag endpoint:

   POST https://{DATAFARI_HOST}/Datafari/rest/v2.0/ai/rag

   RAG process can also be started with the GET /search/* endpoint, even though it is deprecated:

   GET https://{DATAFARI_HOST}/Datafari/rest/v2.0/search/select?q={prompt}&action=rag

   Parameters are extracted from the HTTPS request, and configuration is retrieved from rag.properties.

2. A search query is processed using Search API methods, based on the user prompt, in order to retrieve a list of potentially relevant documents. This search can be either a keyword-based BM25 search, or a Solr Vector Search.
   In the first case, the search will return entire documents, that will require chunking to be processed by the LLM.
   In the case of Vector Search, Solr will return a number of length-limited document excerpts (chunks).

3. Retrieved documents (in particular from BM25 search) might be too big to be handled in one call to the LLM. Chunking allows to cut large documents into smaller pieces to process them sequentially. The maximum size of the chunks can be configured in the “RAG & AI configuration” AdminUI (chunk.size property in rag.properties).

4. During prompting, the list of documents/snippets is converted into a list of prompts that will be processed by the LLM. Each prompt contains instructions (instructions are defined in the /opt/datafari/tomcat/webapps/Datafari/WEB-INF/classes/prompts folder), documents excerpts, and the user prompt as a question.
   
   If prompts are short enough, they might be sent to the LLM into one single request to potentially improve performances. If that is not the case, we use either “Iterative refine” or “Map-Reduce” methods (configurable in AdminUI) to process all chunks.
   The prompt in a single request to the LLM contains as many chunks as possible (minimum 1) without exceeding the limit (in character) set in prompt.max.request.size (instructions and history included).

5. Our solution is designed to be able to interact with various LLM API. The dispatcher selects the proper connector (LlmService) to interact with the configured LLM API.
   A connector is a Java class implementing our LlmService interface. It contains the “invoke” method, taking as parameter a list of prompts, and returning a simple String response from the LLM.
   To this day, we one provide one LlmService:
   - OpenAILlmService, compatible with OpenAI API and any other OpenAI-like API (including Datafari AI Agent)

6. The selected connector prepares one or multiple HTTP/HTTPS queries, one for each prompt from the list. Then, it calls the external LLM API, and extracts the response.
   If the list of prompts contains multiple entries, all the responses are concatenated and sent to the LLM again, to generate a final summary (see prompting strategies).

7. The response is formatted in JSON and sent back to the user.

Chunking

Many documents stored in the FileShare Solr collection are too large to be processed in a single request by a Large Language Model. To address this, implementing a chunking strategy is essential, allowing us to work with manageable, concise, and contextually relevant text snippets.

The chunking strategy depends on the Retrieval method. The two cases are detailed below.

Case 1: BM25 Search	Case 2: Solr Vector Search

Case 1: BM25 Search	Case 2: Solr Vector Search
The BM25 Search returns large and whole documents from FileShare. Those documents are chunked into smaller pieces during the chunking step of the RAG process. All retrieved documents are processed by the ChunkUtils Java class. The chunkContent() method uses a Langchain4j solution: Recursive DocumentSplitters. This splitter recursively divides a document into paragraphs (defined by two or more consecutive newline characters), lines, sentences, words (…), in order to fit as many content as possible without exceeding the configured chunk size limit[1]. [1] The size of the chunks (currently in character, but should be in tokens in the future) can be configured in the AdminUI, or in rag.properties. Open “RAG & AI configuration” AdminUI rag.properties: chunking.chunk.size=4000	In this scenario, chunking occurs during document indexing within the VectorUpdateProcessor. All files uploaded to FileShare are processed and split into smaller chunks using the DocumentByParagraphSplitter. These chunks are then stored as new "child" documents, inheriting their parent's metadata. The chunked content replaces the original content in the child documents. The child documents are stored in a separate Solr collection, VectorMain. Once created, each child’s content is embedded using the Solr TextToVectorUpdateProcessor. When Vector Search is executed in Datafari, it retrieves documents from the VectorMain collection instead of FileShare, eliminating the need for additional chunking steps. The chunking step described in Case 1 is still applied on Vector Search retrieved documents. However, depending on your configuration, this may have no effect since the retrieved contents are probably short enough.
Detailed chunking workflow 1. Indexing Chunking: No chunking here Indexed documents: - Whole documents are indexed into FileShare 2. Retrieval The search returns a maximum of N whole documents from FileShare N corresponds to the chunking.maxFiles property. Default value for N is 3. 3. RAG process Chunking All documents are chunked (if necessary) with the following conditions: - A chunk must not exceed the limit of {chunking.chunk.size} characters - A recursive splitter is used (paragraphs, then lines, then sentences…). This is not configurable. Prompting and chunk management Prompt is generated accordingly to the chunk management strategy (Map-Reduce or Iterative Refining) set in prompt.chunking.strategy - We stuff as many chunks as possible (minimum 1) in the prompt, without exceeding the limit of {prompt.max.request.size} characters - Every chunks from the N documents must be processed. - If all chunks do not fit in a single prompt (one prompt per LLM request), we send as many request as necessary.         property.name: Property configurable in the « RAG & AI configuration » AdminUI, or in the   rag.properties file.	Detailed chunking workflow 1. Indexing Chunking: Documents are chunked in the VectorUpdateProcessor with the following conditions: - A chunk must not exceed the limit of {vector.chunksize} tokens - A recursive splitter is used by default. This is configurable by editing the FileShare property vector.splitter (default value is 300). - You can set an optional maximum overlap by editing the FileShare property vector.maxoverlap (default value is 0). Indexed documents: - Whole documents are indexed into FileShare - Chunks (subdocuments) are indexed into VectorMain. Their content is embedded during the indexing in VectorMain by the TextToVectorUpdateProcessor. 2. Retrieval The search returns a maximum of {solr.topK} subdocuments from VectorMain Default value for solr.topK is 10. 3. RAG process Chunking All documents are chunked (if necessary, which is probably not the case if {chunking.chunk.size} > {vector.chunksize} ) with the following conditions: - A chunk must not exceed the limit of {chunking.chunk.size} characters - A recursive splitter is used (paragraphs, then lines, then sentences…). This is not configurable. Prompting and chunk management Prompt is generated accordingly to the chunk management strategy (Map-Reduce or Iterative Refining) set in prompt.chunking.strategy - We stuff as many chunks as possible (minimum 1) in the prompt, without exceeding the limit of {prompt.max.request.size} characters - Every chunks from the N documents must be processed. - If all chunks do not fit in a single prompt (one prompt per LLM request), we send as many request as necessary.     property.name: Property configurable in the « RAG & AI configuration » AdminUI, or in the   rag.properties file. property.name: Property configurable in the « Solr Vector Search » AdminUI, or directly in   Solr with a Curl command.

Prompts

Prompts are a collection of "Message" objects sent to the LLM. Each "Message" contains:

• A role: "user" for the user query and document content, "assistant" for AI-generated messages, or "system" for instructions. In “mono-message” prompts (one message per LLM request), we only use the “user” role.

• Content: The body of the message, which may include instructions, the user query, and/or document content.

If the RAG process needs to manage too many or too large snippets, it may not be able to fit all of the into one single LLM request. In this situation, a chunk management strategy is required. Datafari provides two options: Iterative Refining method, and Map-Reduce method. You can pick one in the “RAG & AI configuration” AdminUI. Read more about chunking management strategies in the LLM Tranformation Connector documentation.

Below are the prompt chains associated with a RAG query, for each chunk management strategy.

Available LlmServices

An LlmService is a class that implements our “LlmService.java” interface, and acts as an interface between Datafari and an external APIs leveraging Large Language Models (LLMs).

All “LlmService” classes should implement the invoke() method.

The generate() method takes, as parameter, a list of String prompts ready to be sent. All the prompts are sent to the associated LLM API, and a single String response is returned.

Message is a Datafari Java class, with the following attributes:

• String role: The role associated to the Message. Either “user”, “system”, or “assistant”.

• String content: The content of the message.

The LlmService must fulfill multiple tasks:

• Override the generate() method.

• Provide default configuration if relevant (default model, maxToken, specific configuration…)

• Call an external LLM Service, using the proper configuration as defined in the rag.properties file (endpoint temperature, maxTokens, API key…).

• Format the LLM response to a simple String.

• Implement a constructor taking at least a RagConfiguration object as parameter.

OpenAI LLM service

This connector can be used to access OpenAI API, or any other API that uses OpenAI signature, including our Datafari AI Agent . The default model, gpt-3.5-turbo, can be changed by editing the rag.model property.

If you are planning to use your own OpenAI-like solution, edit the rag.api.endpoint property. Default value is https://api.openai.com/v1/

Vector Search

To enhance the relevance of document excerpts sent to the LLM, we have implemented vector search solutions. This machine learning-based approach represents semantic concepts as vectors, offering more accurate results than traditional keyword-based search. Additionally, vector search improves retrieval quality in multilingual datasets, as it relies on semantic meaning rather than exact wording.

Solr Vector Search

Solr Vector Search uses the new text-to-vector feature provided by Solr 9.8. The purpose is to replace the current BM25 search and the local vectore store by a full vector search solution (and in the future, an hybrid search solution for even more relevant results).

Our VectorUpdateProcessor process all documents that are indexed into the FileShare Solr collection. Documents are split into chunks, those are embedded, and stored into the VectorMain collection.

Those chunks can know be searched using our new “/vector” handler, as long as the feature is enabled in the dedicated AdminUI.

The following query can be used to process a vector search through the API.

• queryrag or q (required) : The user query. The "queryrag" parameter is required by Solr; however, if it is missing, Datafari will automatically populate it with the value of "q".

• topK (optional) : The number of results to return. (default: 10, editable in “RAG & AI confirugation” AdminUI)

• model (optional) : The active embeddings model name, as defined in Solr. By default, Datafari automatically uses the value stored in solr.embeddings.model in rag.properties (editable in “Solr Vector Search” AdminUI). Unless you are experimenting with multiple models, or you are directly requesting Solr API (and bypassing Datafari API), you probably don’t need to use this parameter.

Chat Memory (for RAG)

For models that support conversational context, it is possible to enable chat memory within the RAG process.

Enable Chat Memory

To activate chat memory:

1. Enable the option in the AdminUI or in rag.properties:

In “RAG & AI configuration” AdminUI: Open

In rag.properties: chat.memory.enabled=true

2. Define the maximum number of messages to include in the context with:

In “RAG & AI configuration” AdminUI: Open

In rag.properties: chat.memory.history.size=8

By default, 6 messages are included: 3 user messages + 3 assistant responses.

Using Chat Memory in API calls

To include chat history when calling the /ai/rag endpoint, use the optional history field in your JSON payload. Chat history will be added to the LLM context during RAG Generation processes.

Example:

This chat history will be included in the prompt and passed to the LLM (in each request), providing contextual awareness for more coherent and personalized responses.

Datafari will run a full RAG process, based on the query "What is my dog's name ?". Optionnaly, the query may be dynamically rewritten to include the chat history based on the query rewriting method. But if not, the chat history will be used AFTER the chunks have been retrieved. This also means that the documents chunks retrieved from earlier questions of the user (in the same discussion or any other) are NOT used again. Only the documents chunks retrieved at the nth query, are used for the nth answer.

To summarise:

Case 1: query rewriting is not activated

• Step 1.1: only the query is used for retrieving documents chunks

• Step 1.2: the query is combined with past couples [query/generated response]

• Step 1.3: this modified query is sent to the LLM

Case 2: query rewriting is activated

• Step 2.1: the initial query is rewritten to take into account past couples [query/generated response]

• Step 2.2: this modified initial query is used for retrieving documents chunks

• Step 2.3: the initial query is combined with past couples [query/generated response]

• Step 2.4: this initial query is sent to the LLM

Here is the response to the example request.

Query rewriting

The user queries sent to the RAG endpoint are written into a chatbot, and therefore may not be a proper search query. That is why we added an optional “query rewriting” step, that call the LLM in order to generate a new search query, based on the chat history (if provided) and the user initial query. We use the following prompt (template-rewriteSearchQuery.txt):

The {conversation} tag is replaced with multiple lines (one per message from the provided history), each using the following format:

If no history in provided, the “conversation” remains empty.

Enable query rewriting in the “RAG & AI configuration” AdminUI, or in rag.properties configuration file:

In “RAG & AI configuration” AdminUI: Open

In rag.properties: chat.query.rewriting.enabled=true

At current stage, we do not implement Langchain4j query re-writing solution. This may be considered in a future evolution.

Security

Document security

Security is a major concern, and was a central element in our technical decisions. One of the main advantages of the Retrieval Augmented Generation is that the LLM only uses the data it is provided with to answer a question. As long as we control the data sent to the model, we can prevent any leaks.

“Prompt injection” is a set of techniques used to override original instructions in the prompt, through the user input. As our prompt does not contain secret, confidential or structural elements, we consider that it is not a serious issue if a user is able to read it.

Datafari Enterprise Edition provides a security solution, allowing enterprises to set up access restrictions on their files in Datafari. Our RAG solution respects this security. If security is enabled, any user can run a RAG search on Datafari Enterprise Edition. However, the retrieval part of the RAG (BM25 or Vector) will be processed through Datafari SearchAPI to retrieve available documents. If the user is not allowed to see a document, this document will not be retrieved and won’t be sent to the external LLM service. That way, it is impossible for a user to use the RAG tools to retrieve information he should not be able to access.

More information in Datafari Enterprise Edition here.

Prompt security

To reduce the risk of malformed prompt due to bad poor quality (or malicious) user input or indexed content, those are cleaned when added to the prompt.

The following method is applied to RAG sources (document chunks):

The user query is sanitize with the following method: