Introduction

The Datafari API is trying to respect the principles of RESTFUL applications as much as possible and uses json as the mean to exchange data. This means that everything returned by the API will be in JSON, and any payload in POST and PUT requests made to the API will be in JSON too.

On GET and DELETE requests, some parameters may be passed through url parmeters (either as part of the URL or in the search parameter String - anything that is after the “?”)

Prefix

Every endpoint in this API is prefixed by /rest/VX.Y/ where X and Y are the major and minor versions of this API.

Do also remember to add the relevant path to your Datafari web app (https://datafaridomain/Datafari/rest/VX.Y/endpoint)

Response Structure

All responses are formatted using the following template:

In the following, any object presented as the response is showing the format of the content key in the above template.

For errors, the content follows the following structure:

1. code: an integer using the HTTP response code nomenclature

2. reason: a String in English giving some details if relevant to help solve the problem or give context

3. extra: a JSON object providing some extra information from the endpoint if relevant, can be null or empty

Do keep in mind that on some occasions the server might respond with an HTTP error code without any JSON payload, you must handle this case in your applications.

Endpoints

V2.0

GET /rest/v2.0/search/*

This is the endpoint to perform a search request. This endpoint expects a search handler to be provided as well as all the elements of a solr query as URL parameters. A simple example for a search query would be:

Note that {DATAFARI_BASE_URL} is by default of the following shape: [DATAFARI_DOMAIN_NAME]/Datafari

This endpoints allows the use of only a few selected search handlers for security reasons. The handlers that can be called are different depending on the value of the special “action” URL parameter.

Keyword-based search (BM25)

The main handlers to keep in mind are /select and /suggest.

The /select handler is used to perform search queries while the /suggest handler is used to get suggestions for autocomplete.

If you wonder how to format your queries (either for search or query), please refer to the Solr documentation as well as the rest of Datafari’s documentation for the fields available.

Proposals

The /proposals handler works just like /suggest. However, instead of providing autocomplete suggestions, it processes a search based on the query. This one is meant to be used for autoproposals. The service only returns those fields:

• title

• exactContent

• url

Vector search

The /vector handler is used to perform vector search. Vector search features must be enabled before indexing (see https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/3920297985 ). Instead of returning whole documents from the main Solr collection, this handler returns documents snippets from the VectorMain collection. This handler accepts the following parameters:

• q or queryrag (required): a keywords-based or natural language-based query

• topK: (Optional) The number of results returned by the vector distance calculation. Must be greater than start + rows (default: 100).

• rows, start: (Optional) Pagination parameter. Refer to Solr documentation.

• fl: (Optional) The list of fields that must be returned in the results. Refer to Solr documentation.

• model: (Optional) The ID of the embeddings model used for query embeddings. By default, the Active Embeddings Model is used.

• vectorField: (Optional) The Dense Vector Field (in Solr) used for vector search. If you override this parameter, make sure that the vectors contained in this field have been generated by the provided model. By default, the Active Vector Field is used.

Usual Solr parameters (including facets, fq…) can be used here, except for those associated to eDisMax Query Parser.

Example:

In this example, a vector search wil be processed using the query “who are France Labs founders”, returning title, content and url of the 5 first entries from the top 10 results.

The resulting json will look as follows:

Hybrid search (RRF)

The /rrf endpoint is used to perform two searches through Datafari API: one keyword-based (BM25, using /select handler), and one vector search (using /vector handler). Both searches are targeting the VectorMain collection. Then, the results of both searches are merged using a “Reciprocal Rank Fusion” algorithm.

Instead of returning whole documents from the main Solr collection, this handler returns documents snippets from the VectorMain collection. This handler accepts the following parameters:

• q or queryrag (required): a keywords-based or natural language-based query. If both parameters are provided, q is used for BM25 search, and queryrag is used for vector search. If only one of these is provided, it is used by both searches.

• topK: (Optional) The number of results returned by both initial searches. Must be greater than start + rows (default: 100).

• rows, start: (Optional) Pagination parameter. Refer to Solr documentation.

• fl: (Optional) The list of fields that must be returned in the results. Refer to Solr documentation.

• model: (Optional) The ID of the embeddings model used for query embeddings. By default, the Active Embeddings Model is used.

• vectorField: (Optional) The Dense Vector Field (in Solr) used for vector search. If you override this parameter, make sure that the vectors contained in this field have been generated by the provided model. By default, the Active Vector Field is used.

Usual Solr parameters (including facets, fq…) can be used here, except for those associated to eDisMax Query Parser.

Example:

In this example:

• A vector search wil be processed using the query “what is enron address”enron%20address

• A BM25 search wil be processed using the query “enron%20address”

• Datafari merges the results from both searches, and returns the 4 most relevant ones.

The resulting json will look as follows:

Response format

The response format of this endpoint does not follow the standard response format of this API.

The payload of the response is the response from solr (please refer to the Solr documentation to see what those response look like or experiment with the endpoint).
If solr encountered an error, the endpoint will most of the time respond with 200 status, the error being in the solr response itself.

If an error occurred while Datafari is querying Solr, the response may be a 500 or 5xx http response with possibly no JSON (and maybe an html page as a response body).

For each document in the response, they have two fields for the url:

1. doc.url : the original url of the document

2. doc.click_url : The url to use for the href link to send the user to the document (and will track user clicks to get statistics in the search engine)

Starting from Datafari 6.1, there is also a specific url to open folder links :

1. doc.folder_url

GET /rest/v2.0/results/export?*

This is the endpoint to perform an export of the result of a provided query. Here are the parameters to provide:

• query: the main query to perform. ex:
  query=*:*

• facetQuery: An array containing all the facet queries. ex:
  facetQuery[]={!key=From%20100KB%20To%2010MB}original_file_size:[102400 TO 10485760]&facetQuery[]={!key=More%20Than%2010MB}original_file_size:[10485760 TO *]

• facetField: An array containing all the facet fields. ex:
  facetField[]=extension&facetField[]=language&facetField[]=source

• fq: An array containing all the active filter queries. ex:
  fq[]={!tag=query}original_file_size:[102400 TO 10485760]&fq[]={!tag=author}(author:"agence de l'eau artois picardie")

• sort: The sort method to apply to results, can be either "score desc", or "score asc". ex:
  sort=score desc

• fl: The list of fields that must be returned in the results. The fields must be separated by a coma. ex:
  fl=title,last_modified,url

• nbResults: The number of results that the export file must contain. ex:
  nbResults=500
  Be careful with this parameter, because the higher is the number, the more the query will take time and the export file will be big

• type: The type of the export file to generate. Currently (December 2022) the only supported type is “excel”:
  type=excel

Here is an example of a GET request to the export API:

The response of this endpoint is a bytes input stream of the generated export file. In a standard web browser like chrome, it will trigger a download process

V1.0

Bellow is a set of API endpoints that are implemented or in the roadmap. Endpoints that are not implemented are clearly identified by the “NOT IMPLEMENTED” string in the response column. Those endpoints return a 501 HTTP Response code when queried (which is the code for the NOT_IMPLEMENTED http error). For implemented endpoints, query body and format for the content object of the response in case of success are provided.