Spacy Transformation Connector

The spacy transformation connector allows using spacy models exposed through a web API using fastapi to be used to extract entities and add them to the documents metadata.

Lets take an example to use as a guideline throughout this documentation: we want to automatically extract person names from the documents we are indexing and add this information to the index. This requires the setup of a spacy web service with a model able to extract this information, the configuration of a job with the necessary transformation connector to call the web service, a dedicated field in Solr to store the information.

1. Setting up the fastapi server with spacy model

A dedicated server dedicated to running spacy and the fastapi interface is advised as most NER models require several gigabytes of memory to run smoothly. Each model has its own requirements and limitations so it is advised to test the model you plan on using before hand to check its requirements. And then make sure that the server you plan on running the NER model on can handle that.

You can find information on how to deploy a fastapi webservice hosting some spacy models here: https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/2657517573

2. Using the Transformation Connector

Now that the spacy web service is running, we need to configure a job with a transformation connector to call this service.

2.1 Create a job

• Using Datafari’s admin interface create a job for the source you want to extract entities on WITHOUT checking the box to start the job immediately

2.2 Defining a new transformation connector

• In the MCF administration interface, go to the output → list transformation connections menu entry

• Click on the Add a new transformation connection button at the bottom

Open

• Name the connector and click on the type tab

Open

• Choose datafari spacy fastapi connector and click continue

• Then go to the Spacy Fastapi tab and fill in the address of your spacy fastapi server

• Finally click save, if your fastapi server is running and accessible you should see a connection working statement on the next screen

2.3 Add the connector to the job

• Go to your jobs list and select edit on the job you want to add entity extraction on

• In the connection tab, on the bottom row stating “Transformation” with a select dropdown, select the transformation connector you just created

• Then click Insert transformation before on the output line right above

• You should end up with this

• Click on the Spacy Fastapi tab at the top and fill in the required information:

• Name of the model: If a specific model name should be included in the query with the endpoint you are using, precise it here. Can be left blank if no model must be provided.

• Endpoint: The endpoint on the entity extraction web service that you want your request to be sent to. Defaults to /process/ if none is provided. We recommend using /split_detect_and_process/ to avoid running into OOM for large documents, and to benefit from automatic language detection to pick the appropriate model. SpaCy will analyze files content and identify named entities. See an example blow for /split_detect_and_process/.

• Prefix: The prefix you want to use for the metadata that will be added to the document for the entities. It is strongly recommended to set one. Metadata will be named [prefix][entity_label]. No defaults so you have to set one. For example entity_.

3. Store the information in solr

For the entities to be stored in solr, you will need to create fields with the same name as the metadata created in MCF containing your entities. If you configured a prefix for your metadata, you can use one dynamic field in Solr to catch all the entities at once.

The field must have the following properties:

• Be multivalued

• Store strings unaltered as those are entities

We recommend that values are stored and indexed.

In our case, we want to store information about the detected persons names. To do so, we create a field entities_PER with the above parameters so that this entity type is stored into Solr (if you want to keep all entity types, you can create a dynamic field entities_*).