Advanced search feature

Datafari provides an advanced search feature with more ways to refine searches.

In the advanced search interface, you can refine your search by adding filters that apply to specific fields:

And when you execute the advanced search, the filters will be converted into a query understandable by Solr and the results will be displayed in the standard search view :

So how does this feature work ? There are two possibilities, either you perform a simple search and you refine it in the advanced UI, or you perform a complex search as if it were generated by the advanced search and you can use the advanced UI to have a better view/understanding of the query and adjust it as you like.

1. Refine a simple search
   Before going further into explanations, let's put the light on something that may seems obvious : you can start with an empty query and then build a complex query in the advanced search UI

   a) The static part of the advanced search
   The basical UI shown in the adanced search is made of 4 filters that represent the 'basic search' (as we like to call it) and will be applied to all the indexed data of the Solr index. This is the reason why these filters are filled with the last executed simple query and will be converted into a simple query if they are the only filled filters of the advanced UI.

   Here are the descriptions of each filter:

      - All words : Will search in the indexed data for the presence of each typed word in this filter. The words must be separated by a space. This filter is the default applied filter of what you enter in the simple search view. There is no conversion for this filter, everything in it will be added to the query as it is.

      - Exact expression : Will search in for the exact expression entered, in two dedicated index fields: exactContent et exactTitle. This means that it will search for an exact match only in the title and content of a document, not in other fields. This is a default behavior, you can always create new exact fields and change the query executed by this UI. It means that if you enter ‘solar energy' for example (without the single quotes), Datafari will search documents having the phrase ‘solar energy’ (without the quotes) within their title or content. Note that by default, the search is case insensitive: a search for ‘solar Energy’ or 'SOLar EnerGY’ would yeld the same results. 
      - At least one word : Will search for the presence in the indexed data of at least one word of the provided words list. As for the exact expression filter, if you enter only one word,  it will be considered as a "all words" filter and added to the "all words" filter, as the purpose of this filter is to find at least one word among several ones.
   As for the other filters, this one must be filled with words separated by a space. It will be converted by words separated by a literral 'OR' operator. Example: ship OR boat

      - None of these words : Will asserts that none of the indexed data of the documents in the results contains the words of this filter.
   Again, this filter must be filled with words separated by a space. It will be converted by words prefixed by an hyphen. Example: -car -bike

    

   b) The modular part of the advanced search
   Besides the static part, a '+' button is present on the bottom of it, and will allow you to add new filters based on specific indexed Solr fields:
   
   

   
   Once you click on this button, a new line will be added to the UI, containing a dropdown box to select an operator, and a dropdown box filled with all the indexed Solr fields that you can use:
   
   

   
   The selectable fields are only fields that are indexed, not starting by '_' or 'allow_' or 'deny_', and not presents in the denied fields list which is in the 'advanced-search.properties' file located in /opt/datafari/tomcat/conf
   The little red cross allows you to remove the filter.

   So this is how everything works:
   - Firstly select the wanted operator for your filter. The default operator is 'AND' because Datafari is configured by default to use the 'AND' operator between words and filters. This is the reson why you will notice that when you select the 'AND' operator, it will not appear in the converted query although the 'OR' operator does.
   - Secondly select your field. Once selected the UI will be updated depending on the type of the selected field:

          i) The type of the field is text based
   The UI will be updated with the same 4 fields as for the base search. But of course, everything you will enter in those ones will only be applied to the selected field and not on all the indexed data.
   However there is a little difference on the exact expression filter. Indeed, if as of some clients you would like to apply a "true" exact expression to the selected field (which means with case sensitivity and very less stemming), there is a solution.
   First you need to create in Solr a clone field named [original_field_name]_exact that is of string type or custom type with minimum (only WhiteSpaceTokenizerFactory for example) or no stemming at all. This clone field can be specially used for this filter by adding the original field name to the EXACTFIELDS variable (which is a list of fields separated by a coma) in the 'advanced-search.properties' file located in /opt/datafari/tomcat/conf. Once these steps are done, you will notice that if you fill the exact expression filter in the advanced search UI for this specific field, the converted query will automaticaly contain an aditional filter which will be an exact expression filter on the clone field with the AND operator.

          ii) The type of the field is a date
   The UI will be updated with a DateSelectorFacetModule allowing you to select a date range for the field.

          iii) The type of the field is numeric
   The UI will be updated with a range filter UI where you can select a 'from' numeric value and/or a 'to' numeric value. The behavior of this filter is almost the same as the DateSelectorFacetModule as if you want to perform a 'greater than or equals' filter, you have to empty the 'to' value, and if you want to perform a 'less than or equals' filter, you will have to empty the 'from' value. To search for a specific numeric value, just set the same value as 'from' and 'to' value.

   
   You can customize the label of fields if you want them to be more user friendly than the original field name. To do so, you will need to edit the DATAFARI_HOME/tomcat/conf/advanced-search.properties file and update the LABELED_FIELDS parameter. This parameter must be a json containing key/value pairs where the keys are the original field name you want to label, and the value is the wanted label name. For example, if you want to replace the field “artifact_id” by the label “Artifact ID” and the field “artifact_last_modified” by the label “Artifact Last Modified”, then you will set the LABELED_FIELDS parameter like this:
   
   

   LABELED_FIELDS={"artifact_id":"Artifact ID", "artifact_last_modified":"Artifact Last Modified"}

   And here will be the result after reloading the search UI:
   
   

   
   
   
   Now that you know the basis, you can add as many filters as you want, test different combinations with the operators, and try to learn how to directly build a complex query in the simple search view.

    

2. Adjust a complex search
   The advanced search widget does not exclusively perform the conversion of the advanced interface into a query, but it is also able to re-interprete the last executed query to build the corresponding UI in the advanced interface, even if it was not a query issued from the advanced UI.
   So if you master the 'advanced query format' and use it in the simple search view, the advanced search widget will be more or less able to create the corresponding UI and you will have the possibility to 'tune' the query graphically. The 'more or less' expression is used here because it is of course a complex process and we did not thinked it to be perfect or to correctly handle the query in 100% of the cases.
   
   

3. Other configurable things (For Enterprise Edition)
   There are several things that you can configure in the advanced search thanks to the conf file “advanced-search.properties” located in DATAFARI_HOME/tomcat/conf.
   You can determine predefined values for some fields so that the users will only be able to set one of those values from a dropdown list. Follow this documentation to configure it: https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/331415555
   You can also configure some fields so that they have an autocompletion. Follow that documentation to configure it:
   
   

The 3.2 version of Datafari comes with a fresh new advanced search feature that provides users a much more powerful way to refine their searches.

Indeed, you can now, in the new advanced search interface, refine your base search by adding filters that apply to specific fields:

And when you execute the advanced search, the filters will be converted into a query understandable by Solr and the results will be displayed in the standard search view :

So how does this great feature works ? There are two possibilities, either you perform a simple search and you refine it in the advanced UI, either you perform a complex search like it would be generated by the advanced search and you can use the advanced UI to have a better view/understanding of the query and adjust it as you like.

1. Refine a simple search

   Before going further into explanations, let's put the light on something that may seems obvious but that is harmless to remember : you can of course start with an empty query and build a complex query in the advanced search UI upon it !

   a) The static part of the advanced search
   The basical UI shown in the adanced search is made of 4 filters that represent the 'basic search' (as we like to call it) and will be applied to all the indexed data of the Solr index. This is the reason why these filters are filled with the last executed simple query and will be converted into a simple query if they are the only filled filters of the advanced UI.
   Here are the descriptions of each filter:
      - All words : Will search in the indexed data for the presence of each typed word in this filter. The words must be separated by a space. This filter is the default applied filter of what you enter in the simple search view. There is no conversion for this filter, everything in it will be added to the query as it is
      - Exact expression : Will search in the indexed data for the presence of the exact expression entered. It means that if you enter 'solar energy' for example (without the single quotes), Solr will search documents having the word 'solar' immediately followed by the word 'energy'. However be careful, the search will not necessarily take care of the case sensitivity. It all depends on the way you have indexed the data and on the fields types and analyzers used for global search. Also, because of this and on the applied stemming in Datafari's default configuration, an exact expression search on the global indexed data will not necessarily only return exact expression of expected words, but also exact expression of words having the same stemming result ! 
   Another thing to remember is that if you enter only one word in this filter, it will be considered as a "all words" filter and added to the "all words" filter, as the purpose of an exact expression filter is to search for an exact combination of several words.
   This filter must be filled with several words separated by a space and will be converted for the Solr query by words surrounded by quotes. Example : "solar energy"
      - At least one word : Will search for the presence in the indexed data of at least one word of the provided words list. As for the exact expression filter, if you enter only one word,  it will be considered as a "all words" filter and added to the "all words" filter, as the purpose of this filter is to find at least one word among several ones.
   As for the other filters, this one must be filled with words separated by a space. It will be converted by words separated by a literral 'OR' operator. Example: ship OR boat
      - None of these words : Will asserts that none of the indexed data of the documents in the results contains the words of this filter.
   Again, this filter must be filled with words separated by a space. It will be converted by words prefixed by an hyphen. Example: -car -bike
   b) The modulable part of the advanced search
   Besides the static part, a '+' button is present on the bottom of it, and will allow you to add new filters based on specific indexed Solr fields:
   
   

   Once you click on this button, a new line will be added to the UI, containing a dropdown box to select an operator, and a dropdown box filled with all the indexed Solr fields that you can use:

   The selectable fields are only fields that are indexed, not starting by '_' or 'allow_' or 'deny_', and not presents in the denied fields list which is in the 'advanced-search.properties' file located in /opt/datafari/tomcat/conf
   The little red cross allows you to remove the filter.
   So this is how everything works:
   - Firstly select the wanted operator for your filter. The default operator is 'AND' because Datafari is configured by default to use the 'AND' operator between words and filters. This is the reson why you will notice that when you select the 'AND' operator, it will not appear in the converted query although the 'OR' operator does.
   - Secondly select your field. Once selected the UI will be updated depending on the type of the selected field:
          i) The type of the field is text based
   The UI will be updated with the same 4 fields as for the base search. But of course, everything you will enter in those ones will only be applied to the selected field and not on all the indexed data.
   However there is a little difference on the exact expression filter. Indeed, if as of some clients you would like to apply a "true" exact expression to the selected field (which means with case sensitivity and very less stemming), there is a solution.
   First you need to create in Solr a clone field named [original_field_name]_exact that is of string type or custom type with minimum (only WhiteSpaceTokenizerFactory for example) or no stemming at all. This clone field can be specially used for this filter by adding the original field name to the EXACTFIELDS variable (which is a list of fields separated by a coma) in the 'advanced-search.properties' file located in /opt/datafari/tomcat/conf. Once these steps are done, you will notice that if you fill the exact expression filter in the advanced search UI for this specific field, the converted query will automaticaly contain an aditional filter which will be an exact expression filter on the clone field with the AND operator.
          ii) The type of the field is a date
   The UI will be updated with a DateSelectorFacetModule allowing you to select a date range for the field.
          iii) The type of the field is numeric
   The UI will be updated with a range filter UI where you can select a 'from' numeric value and/or a 'to' numeric value. The behavior of this filter is almost the same as the DateSelectorFacetModule as if you want to perform a 'greater than or equals' filter, you have to empty the 'to' value, and if you want to perform a 'less than or equals' filter, you will have to empty the 'from' value. To search for a specific numeric value, just set the same value as 'from' and 'to' value.
   Now that you know the basis, you can add as many filters as you want, test different combinations with the operators, and try to learn how to directly build a complex query in the simple search view.

2. Adjust a complex search

The advanced search widget does not exclusively perform the conversion of the advanced interface into a query, but it is also able to re-interprete the last executed query to build the corresponding UI in the advanced interface, even if it was not a query issued from the advanced UI.
So if you master the 'advanced query format' and use it in the simple search view, the advanced search widget will be more or less able to create the corresponding UI and you will have the possibility to 'tune' the query graphically. The 'more or less' expression is used here because it is of course a complex process and we did not thinked it to be perfect or to correctly handle the query in 100% of the cases.