{"type":"doc","content":[{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Valid from Datafari 6","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"},{"text":"The documentation below is valid from Datafari v6.0 upwards, for both the CE and EE editions","type":"text"}]}]},{"type":"paragraph","content":[{"text":"By default a Tika server does not perform \"OCRization\" on incoming docs. To do so, you will need two things:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install Tesseract","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Configure the Tika Server","type":"text"}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"You may use the OCR Tika in two variations:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Direct use, as described here, and the OCR phase takes place at the indexing phase","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Indirect use, by going through a ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/1832484865"}},{"text":" , and the OCR phase takes place asynchronously.","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"1. Install Tesseract","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Tesseract is an open source application that \"OCRize\" documents. Tika uses this tool to retrieve the content of images.","type":"text"},{"type":"hardBreak"},{"text":"To install it on a Debian/Ubuntu system, run the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install tesseract-ocr","type":"text"}]},{"type":"paragraph","content":[{"text":"Congrats ! Tesseract is now installed and ready to be used. However by default, Tesseract is only able to deal with english language, if you want to deal with other languages you will need to install the corresponding package (if available). The pattern of a language paclage is: tesseract-ocr-[country_code]","type":"text"},{"type":"hardBreak"},{"text":"The country code is based on three letters. For example the Tesseract package for the french language is: tesseract-ocr-fra","type":"text"},{"type":"hardBreak"},{"text":"So to install the french package you will need to run the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install tesseract-ocr-fra","type":"text"}]},{"type":"paragraph","content":[{"text":"You can find the list of available language packages for Tesseract on the web or here: ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/tesseract-ocr/tessdoc/blob/master/Data-Files-in-different-versions.md"}},{"text":" ","type":"text"},{"type":"hardBreak"},{"text":"You can also install all the available languages in a single command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install tesseract-ocr-all","type":"text"}]},{"type":"paragraph","content":[{"text":"At this step you can test if tesseract is correctly working with your desired languages by sending it a TIF image file containing text to OCR through the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"tesseract /path/to/my-test-image.tif /path/to/my-output","type":"text"}]},{"type":"paragraph","content":[{"text":"No need to add a file extension for the output file as Tesseract automatically adds “.txt”","type":"text"},{"type":"hardBreak"},{"text":"The “my-output.txt” file should contain the extracted text from your TIF file","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"2. Configure the Tika Server","type":"text","marks":[{"type":"strong"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"This section is optional if you have been using this procedure with the OCR option : ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/2768601089"}},{"text":" ","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Once Tesseract is installed on the same machine than the Tika Server, it usually uses it automatically to parse image files but in the Tika server provided with Datafari, it is disabled by default. ","type":"text"}]},{"type":"paragraph","content":[{"text":"To enable OCR in Tika Server you will need to configure the ","type":"text"},{"text":"TesseractOCRParser","type":"text","marks":[{"type":"strong"}]},{"text":" and the ","type":"text"},{"text":"PDFParser","type":"text","marks":[{"type":"strong"}]},{"text":" in the ","type":"text"},{"text":"tika-config.xml","type":"text","marks":[{"type":"strong"}]},{"text":" file located in ","type":"text"},{"text":"[DATAFARI_HOME]/tika-server/conf/","type":"text","marks":[{"type":"strong"}]},{"text":". By default, the TesseractOCRParser is excluded from the parsers and its configuration is commented to disable it. ","type":"text"}]},{"type":"paragraph","content":[{"text":"So first you will need to comment its exclusions: ","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"\n \n \n","type":"text"}]},{"type":"paragraph","content":[{"text":"Then to un-comment its configuration section to enable it:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"\n\n \n false\n gray\n 300\n 4\n false\n triangle\n \n eng+fra\n 2147483647\n 1000\n 1\n \n false\n 900\n false\n \n \n 420\n \n","type":"text"}]},{"type":"paragraph","content":[{"text":"Once the section un-commented, you can tweak the different parameters of the parser to fit your needs (it is not the purpose of this documentation to explain them: Google is you friend !","type":"text"},{"type":"emoji","attrs":{"id":"1f609","text":"😉","shortName":":wink:"}},{"text":"). In any case, ","type":"text"},{"text":"YOU MUST","type":"text","marks":[{"type":"strong"}]},{"text":" at least pay close attention to the “timeoutSeconds” and the “language” parameters. ","type":"text"},{"type":"hardBreak"},{"text":"The “timeoutSeconds” parameter corresponds to the max allowed time to perform OCR on a document before giving up. In our experience most of documents of less than 10 pages fulfilled with images will take a bit more than 10 minutes to be processed, thus we set the default value to 1200 seconds which corresponds to 20 minutes in order to cover most of the documents. You may want to adapt this parameter according to your needs and/or experience but keep in mind that the value you set here ","type":"text"},{"text":"MUST BE THE SAME","type":"text","marks":[{"type":"strong"}]},{"text":" than the value of the ","type":"text"},{"text":"taskTimeoutMillis","type":"text","marks":[{"type":"code"}]},{"text":" parameter of the Tika server parameters section (mentioned further in this documentation).","type":"text"},{"type":"hardBreak"},{"text":"The “language” parameter corresponds to the languages the Tesseract parser may encounter and that it will try to recognize in documents. Any language encountered not listed here will not be recognized and the OCRization will fail or be inaccurate ! The parameter is configured by default on “eng+fra”. If you want to deal with documents in other languages, you will need to add the language code (which is the same than the one present in the package name). Each specified language must be separated by a '+'.","type":"text"},{"type":"hardBreak"},{"text":"For example, if you want to deal with english, french and deutch documents, you will need to set the \"language\" property like this: ","type":"text"}]},{"type":"codeBlock","content":[{"text":"eng+fra+deu","type":"text"}]},{"type":"paragraph","content":[{"text":"Now, to perform OCR process on images embeded in PDF files, you need to tell the PDF parser of Tika to do so because it is disabled by default (for resources consumption and processing time reasons). This can be done in the ","type":"text"},{"text":"PDFParser","type":"text","marks":[{"type":"strong"}]},{"text":" configuration section of the tika-config.xml file which is by default:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"\n \n \n true\n 0.3\n true\n false\n \n false\n 2.5\n true\n true\n false\n true\n true\n false\n false\n true\n false\n \n 524288000\n \n 300\n \n png\n 1.0\n \n gray\n ALL\n \n no_ocr\n better\n \n false\n false\n 0.5\n false\n \n","type":"text"}]},{"type":"paragraph","content":[{"text":"As you can notice, there is an 'ocrStrategy' parameter which is by default set to 'no_ocr'. You can set it with three other options:","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Our recommendation is to set to ","type":"text"},{"text":"auto","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"paragraph","content":[{"text":"- ","type":"text"},{"text":"auto ","type":"text","marks":[{"type":"strong"}]},{"text":": try to extract text, but run OCR if fewer than 10 characters were extracted of if there are more than 10 characters with unmapped Unicode values. This is the most efficient mode but you will loose OCR text on pages containing one or several images on which to perform OCR and a text of more than 10 characters ","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"ocr_only","type":"text","marks":[{"type":"strong"}]},{"text":" : don't bother extracting text, just run OCR on each page. So if you only have text in a pdf and no images it will be a huge amount of process time loss for nothing cause the OCR process will take time but will not retunr any text. Use this mode only if you are absolutely sure that you will only feed Tika with files containing only images to OCR process","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"ocr_and_text_extraction","type":"text","marks":[{"type":"strong"}]},{"text":" : Run both OCR and text extraction on each page. This is the mode that will ensure to retrieve everything (text + ocr processed text) but it is also the more expensive mode in terms of resources/processing time","type":"text"},{"type":"hardBreak"},{"text":"So for example if I want the PDF parser to return both \"OCRized\" text and \"natural\" text of PDF files, I will set the 'ocrStrategy' parameter like this:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"ocr_and_text_extraction","type":"text"}]},{"type":"paragraph","content":[{"text":"NOTE: There is an “extractInlineImages” parameter in the PDFParser conf, it is used to change the OCR mode. There are two OCR modes:","type":"text"},{"type":"hardBreak"},{"text":"1/ ","type":"text"},{"text":"render each page and then run OCR on that rendered image","type":"text","marks":[{"type":"strong"}]},{"text":": This is the default mode. In this mode, each pdf page is rendered as a unique image and the OCR process is applied to this image. The advantage is that there is only one OCR process per page so it consumes less resources and is potentially faster than the other mode. The inconvenient is that if there are several images next to each other in the page, the semantic analysis of the OCR process may be wrong and/or have difficulties to process the page, resulting in bad accuracy and lack of performances ","type":"text"},{"type":"hardBreak"},{"text":"2/ ","type":"text"},{"text":"run OCR against inline images","type":"text","marks":[{"type":"strong"}]},{"text":": This is the alternative mode that is enabled be setting the “extractInlineImages” parameter to true. Instead of rendering each pdf page as a unique image, the parser will extract the images as they are in the page and OCR process each one of them. The advantage of this method is that the semantic analysis will be 100% correct in any case but the inconvenient is that if there are many images in a page, the performances will be much worse than the other mode","type":"text"},{"type":"hardBreak"},{"text":"The second mode “run OCR against inline images” MUST NOT BE ENABLED with Datafari when the option “extract archive content” is enabled on the Tika Server connector in a job. Otherwise Tika will process twice the PDF pages in case this mode is enabled, resulting in DRASTIC JOB PERFORMANCE LOSSES ! (at least x3 times slower)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"strong"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Our recommendation is to keep the default configuration so the properties are set like that : ","type":"text"},{"type":"hardBreak"},{"text":"false","type":"text"}]},{"type":"paragraph","content":[{"text":"true","type":"text"}]}]},{"type":"paragraph","content":[{"text":"And ","type":"text"},{"text":"LAST BUT NOT LEAST","type":"text","marks":[{"type":"strong"}]},{"text":", as the OCR process can also take a very long time, you will also need to tell the Tika server not to kill its task too soon. By default a timeout of 120 seconds is set to a Tika task which means after 2 minutes, if the text extraction of a file is not finished, the task is killed. You can change the value of this timeout thanks to the ","type":"text"},{"text":"taskTimeoutMillis ","type":"text","marks":[{"type":"strong"}]},{"text":"parameter of the “server” section in the tika-config.xml file, remember that this parameter ","type":"text"},{"text":"MUST BE EQUAL","type":"text","marks":[{"type":"strong"}]},{"text":" to the “timeoutSeconds” parameter of the TesseractOCRParser section (mentioned earlier in this documentation) :","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Ou recommandation is : ","type":"text"}]}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"1200000\n\n1200","type":"text"}]},{"type":"paragraph","content":[{"text":"In our experience with OCR, many files can take over than 10 minutes to be OCR processed so we strongly recommend to set at least 20 minutes of task timeout so a value of 1200000 !","type":"text"}]},{"type":"paragraph","content":[{"text":"Once the above instructions are done, ","type":"text"},{"text":"don’t forget to restart the Tika server","type":"text","marks":[{"type":"strong"}]},{"text":". You can now test the OCR process with Tika thanks to a PDF file containing images to OCR process with the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"curl -T /path/to/my-pdf-file.pdf http://localhost:9998/rmeta/text > /path/to/extract-result.json","type":"text"}]},{"type":"paragraph","content":[{"text":"Once processed, you should retrieve the extracted text in the “X-TIKA:content” parameter of the ‘extract-result.json’ file.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"3.Configure the MCF job(s)","type":"text"}]},{"type":"paragraph","content":[{"text":"Go to the MCF admin UI (Data Crawlers Expert Mode) and:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Edit the TikaServerRMetaConnector transformation connector in the “List Transformation Connections” menu and set the “Max connections” parameter to “1” in the “Throttling” tab. This will limit the number of documents simultaneously sent to the Tika OCR server to 1 and so, guarantee the stability of the OCR process.","type":"text"},{"type":"hardBreak"},{"text":"You also need to change the ‘Connection timeout’ and ‘Socket timeout’ parameters in the ‘Tika Server’ tab to set them to the value you set to the timeOut parameter in the previous section + 5 seconds ","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":70.0},"content":[{"type":"media","attrs":{"width":1913,"id":"27e0aadd-9e2b-41b6-9488-572b18e34031","collection":"contentId-123830282","type":"file","height":591}}]},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":70.0},"content":[{"type":"media","attrs":{"width":1899,"id":"e183e0a9-521c-4a13-b58b-baa23190da70","collection":"contentId-123830282","type":"file","height":607}}]},{"type":"paragraph"},{"type":"mediaSingle","attrs":{"layout":"center","width":70.0},"content":[{"type":"media","attrs":{"width":960,"id":"906ef551-3428-42c0-86d3-256fee7e7739","collection":"contentId-123830282","type":"file","height":771}}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Edit the repository connector your are using in your job in the “","type":"text"},{"text":"List Repository Connections","type":"text","marks":[{"type":"em"}]},{"text":"” menu and set the “","type":"text"},{"text":"Max connections","type":"text","marks":[{"type":"em"}]},{"text":"” parameter to “1” in the “Throttling” tab.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center"},"content":[{"type":"media","attrs":{"width":1154,"id":"0690822d-63e6-4c61-a8a0-81f6f761ca42","collection":"contentId-123830282","type":"file","height":671}}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Add a filter to your job in order to limit the files that will be crawled to those that need to be OCR processed. If you don’t know or understand how to do that, please contact Francelabs.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"4. Enjoy !","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Once you have completed the two previous sections, you can run your Tika Server and enjoy the power of the OCR !","type":"text"}]},{"type":"rule"},{"type":"expand","attrs":{"title":"Valid from 4.3 to 5.5"},"content":[{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Valid from 4.3 to 5.5","type":"text"}]},{"type":"paragraph","content":[{"text":"The documentation below is valid from Datafari v4.3 upwards","type":"text"}]}]},{"type":"paragraph","content":[{"text":"By default a Tika server does not perform \"OCRization\" on incoming docs. To do so, you will need two things:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install Tesseract","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Configure the Tika Server","type":"text"}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"You may use the OCR Tika in two variations:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Direct use, as described here, and the OCR phase takes place at the indexing phase","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Indirect use, by going through an annotator (see ","type":"text"},{"text":"OCR Annotator Configuration - Enteprise Edition","type":"text","marks":[{"type":"link","attrs":{"href":"https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/116195560"}}]},{"text":") or a ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/1832484865"}},{"text":" , and the OCR phase takes place asynchronously.","type":"text"}]}]}]}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install Tesseract","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"},{"text":"Tesseract is an open source application that \"OCRize\" documents. Tika uses this tool to retrieve the content of images.","type":"text"},{"type":"hardBreak"},{"text":"To install it on a Debian/Ubuntu system, run the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install tesseract-ocr","type":"text"}]},{"type":"paragraph","content":[{"text":"Congrats ! Tesseract is now installed and ready to be used. However by default, Tesseract is only able to deal with english language, if you want to deal with other languages you will need to install the corresponding package (if available). The pattern of a language paclage is: tesseract-ocr-[country_code]","type":"text"},{"type":"hardBreak"},{"text":"The country code is based on three letters. For example the Tesseract package for the french language is: tesseract-ocr-fra","type":"text"},{"type":"hardBreak"},{"text":"So to install the french package you will need to run the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install tesseract-ocr-fra","type":"text"}]},{"type":"paragraph","content":[{"text":"You can find the list of available language packages for Tesseract on the web or here: ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/tesseract-ocr/tessdoc/blob/master/Data-Files-in-different-versions.md"}},{"text":" ","type":"text"},{"type":"hardBreak"},{"text":"You can also install all the available languages in a single command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install tesseract-ocr-all","type":"text"}]},{"type":"paragraph","content":[{"text":"At this step you can test if tesseract is correctly working with your desired languages by sending it a TIF image file containing text to OCR through the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"tesseract /path/to/my-test-image.tif /path/to/my-output","type":"text"}]},{"type":"paragraph","content":[{"text":"No need to add a file extension for the output file as Tesseract automatically adds “.txt”","type":"text"},{"type":"hardBreak"},{"text":"The “my-output.txt” file should contain the extracted text from your TIF file","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Configure the Tika Server","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"},{"text":"Once Tesseract is installed on the same machine than the Tika Server, it usually use it automatically to parse image files but in the Tika server provided with Datafari, we created a parameter to enable or disable the OCR parser and it is disabled by default. This parameter is named ‘DO_OCR’ and is located in the file /opt/datafari/tika-server/bin/set-tika-env.sh. You must set it to ‘true’ in order to perform OCR with the Tika Server: ","type":"text"}]},{"type":"codeBlock","content":[{"text":"export DO_OCR=true","type":"text"}]},{"type":"paragraph","content":[{"text":"Now the Tika server will use Tesseract but it will only use the english language to \"OCRize\" these files by default and you will need to configure the Tika Server to use different or several languages.","type":"text"},{"type":"hardBreak"},{"text":"To do so, the tika server provided with Datafari exposes the configuration files so they are easily accessible. Here are the locations of the two configuration files along with a quick description:","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"/opt/datafari/tika-server/conf/ocr/org/apache/tika/parser/ocr/TesseractOCRConfig.properties","type":"text","marks":[{"type":"strong"}]},{"text":": This is the configuration file of the main OCR parser which parses image files","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"/opt/datafari/tika-server/conf/ocr/org/apache/tika/parser/pdf/PDFParser.properties","type":"text","marks":[{"type":"strong"}]},{"text":": This is the configuration file of the PDF parser. To perform OCR process on images embbeded in a PDF file you need to tell the PDF parser of Tika to do so because it is disabled by default (for resources consumption and processing time reasons). ","type":"text"},{"type":"hardBreak"},{"text":"Note","type":"text","marks":[{"type":"strong"},{"type":"em"}]},{"text":": if you configure a Tika Server independently of Datafari on a different machine, we strongly recommand to copy the tika-server folder of Datafari in order to benefit from the simplified configuration options. Otherwise you will have to directly modify the two files listed above in the tika-server.jar file which is less convenient. In the jar file, the two files are located in ","type":"text"},{"text":"/ocr/org/apache/tika/parser/ocr/TesseractOCRConfig.properties ","type":"text","marks":[{"type":"strong"}]},{"text":"and ","type":"text"},{"text":"/ocr/org/apache/tika/parser/pdf/PDFParser.properties","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"},{"type":"hardBreak"},{"text":"First, edit the TesseractOCRConfig.properties file:","type":"text"},{"type":"hardBreak"},{"text":"In this file you can see a property named \"language\" which is set to \"eng+fra\". If you want to deal with documents in other languages, you will need to add the language code (which is the same three letters code of the package). Each specified language must be separated by a '+'.","type":"text"},{"type":"hardBreak"},{"text":"For example, if you want to deal with english, french and deutch documents, you will need to set the \"language\" property like this: ","type":"text"}]},{"type":"codeBlock","content":[{"text":"language=eng+fra+deu","type":"text"}]},{"type":"paragraph","content":[{"text":"In the 'TesseractOCRConfig.properties' you can also configure other things but it is not the purpose of this documentation to explain them: Google is you friend !","type":"text"},{"type":"emoji","attrs":{"id":"1f609","text":"😉","shortName":":wink:"}}]},{"type":"paragraph","content":[{"text":"Then edit the file PDFParser.properties:","type":"text"},{"type":"hardBreak"},{"text":"This file has an 'ocrStrategy' which is by default set to 'no_ocr'. You can set it with three other options:","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"auto ","type":"text","marks":[{"type":"strong"}]},{"text":": try to extract text, but run OCR if fewer than 10 characters were extracted of if there are more than 10 characters with unmapped Unicode values. This is the most efficient mode but you will loose OCR text on pages containing one or several images on which to perform OCR and a text of more than 10 characters ","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"ocr_only","type":"text","marks":[{"type":"strong"}]},{"text":" : don't bother extracting text, just run OCR on each page. So if you only have text in a pdf and no images it will be a huge amount of process time loss for nothing cause the OCR process will take time but will not retunr any text. Use this mode only if you are absolutely sure that you will only feed Tika with files containing only images to OCR process","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"ocr_and_text_extraction","type":"text","marks":[{"type":"strong"}]},{"text":" : Run both OCR and text extraction on each page. This is the mode that will ensure to retrieve everything (text + ocr processed text) but it is also the more expensive mode in terms of resources/processing time","type":"text"},{"type":"hardBreak"},{"text":"So for example if I want the PDF parser to return both \"OCRized\" text and \"natural\" text of PDF files, I will set the 'ocrStrategy' parameter like this:","type":"text"}]},{"type":"codeBlock","content":[{"text":"ocrStrategy ocr_and_text_extraction","type":"text"}]},{"type":"paragraph","content":[{"text":"NOTE: There is an “extractInlineImages” parameter in the PDFParser.properties file, it is used to change the OCR mode. There are two OCR modes:","type":"text"},{"type":"hardBreak"},{"text":"1/ ","type":"text"},{"text":"render each page and then run OCR on that rendered image","type":"text","marks":[{"type":"strong"}]},{"text":": This is the default mode. In this mode, each pdf page is rendered as a unique image and the OCR process is applied to this image. The advantage is that there is only one OCR process per page so it consumes less resources and is potentially faster than the other mode. The inconvenient is that if there are several images next to each other in the page, the semantic analysis of the OCR process may be wrong and/or have difficulties to process the page, resulting in bad accuracy and lack of performances ","type":"text"},{"type":"hardBreak"},{"text":"2/ ","type":"text"},{"text":"run OCR against inline images","type":"text","marks":[{"type":"strong"}]},{"text":": This is the alternative mode that is enabled be setting the “extractInlineImages” parameter to true. Instead of rendering each pdf page as a unique image, the parser will extract the images as they are in the page and OCR process each one of them. The advantage of this method is that the semantic analysis will be 100% correct in any case but the inconvenient is that if there are many images in a page, the performances will be much worse than the other mode","type":"text"},{"type":"hardBreak"},{"text":"The second mode “run OCR against inline images” MUST NOT BE ENABLED with Datafari when the option “extract archive content” is enabled on the Tika Server connector in a job. Otherwise Tika will process twice the PDF pages in case this mode is enabled, resulting in DRASTIC JOB PERFORMANCE LOSSES ! (at least x3 times slower)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"strong"}]},{"type":"hardBreak"},{"text":"And last but not least, as the OCR process can also take a very long time, you will also need to tell the Tika server not to kill its task too soon. By default a timeout of 120 seconds is set to a Tika task which means after 2 minutes, if the text extraction of a file is not finished, the task is killed. You can change the value of this timeout thanks to the ","type":"text"},{"text":"TIKA_SPAWN_TASK_TIMEOUT ","type":"text","marks":[{"type":"strong"}]},{"text":"parameter in the /opt/datafari/tika-server/bin/set-tika-env.sh file:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#timeouts are expressed in milliseconds\nexport TIKA_SPAWN_TASK_TIMEOUT=\"-taskTimeoutMillis 1200000\"","type":"text"}]},{"type":"paragraph","content":[{"text":"In our experience with OCR, many files can take over than 10 minutes to be OCR processed so we strongly recommend to set at least 20 minutes of task timeout !","type":"text"},{"type":"hardBreak"},{"text":"Once the above instructions are done, restart the Tika server. You can now test the OCR process with tika thanks to a PDF file containing images to OCR process with the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"curl -T /path/to/my-pdf-file.pdf http://localhost:9998/unpack/all --output /path/to/result-file.zip","type":"text"}]},{"type":"paragraph","content":[{"text":"Once processed, you should retrieve the extracted text in the “__TEXT__” file within the ‘result-file.zip’ file.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Enjoy !","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"},{"text":"Once you have completed the two previous sections, you can run your Tika Server and enjoy the power of the OCR !","type":"text"}]}]}]}]},{"type":"rule"},{"type":"expand","attrs":{"title":"Valid from 3.2 to 4.2"},"content":[{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Valid from 3.2 to 4.2","type":"text"}]},{"type":"paragraph","content":[{"text":"The documentation below is valid from Datafari v3.2 up to v4.2","type":"text"}]}]},{"type":"paragraph","content":[{"text":"By default a Tika server does not perform \"OCRization\" on incoming docs. To do so, you will need two things:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install Tesseract","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Configure the Tika Server","type":"text"}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"You may use the OCR Tika in two variations:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Direct use, as described here, and the OCR phase takes place at the indexing phase","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Indirect use, by going through an annotator (see ","type":"text"},{"text":"OCR Annotator Configuration - Enteprise Edition","type":"text","marks":[{"type":"link","attrs":{"href":"https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/116195560"}}]},{"text":"), and the OCR phase takes place asynchronously.","type":"text"}]}]}]}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install Tesseract","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"},{"text":"Tesseract is an open source application that \"OCRize\" documents. Tika uses this tool to retrieve the content of images.","type":"text"},{"type":"hardBreak"},{"text":"To install it on a Debian/Ubuntu system, run the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install tesseract-ocr","type":"text"}]},{"type":"paragraph","content":[{"text":"Congrats ! Tesseract is now installed and ready to be used. However by default, Tesseract is only able to deal with english language, if you want to deal with other languages you will need to install the corresponding package (if available). The pattern of a language paclage is: tesseract-ocr-[country_code]","type":"text"},{"type":"hardBreak"},{"text":"The country code is based on three letters. For example the Tesseract package for the french language is: tesseract-ocr-fra","type":"text"},{"type":"hardBreak"},{"text":"So to install the french package you will need to run the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install tesseract-ocr-fra","type":"text"}]},{"type":"paragraph","content":[{"text":"You can find the list of available language packages for Tesseract on the web or here: ","type":"text"},{"text":"https://packages.ubuntu.com/search?keywords=tesseract-ocr","type":"text","marks":[{"type":"link","attrs":{"href":"https://packages.ubuntu.com/search?keywords=tesseract-ocr"}}]},{"type":"hardBreak"},{"text":"You can also install all the available languages in a single command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install tesseract-ocr-all","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Configure the Tika Server","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"},{"text":"Once Tesseract is installed on the same machine than the Tika Server, this one will automatically use it to parse image files. But it will only use the english language to \"OCRize\" these files by default and you will need to configure the Tika Server to use different or several languages.","type":"text"},{"type":"hardBreak"},{"text":"To do so you will need to edit the jar file of Tika and then edit the following file: /org/apache/tika/parser/ocr/TesseractOCRConfig.properties","type":"text"},{"type":"hardBreak"},{"text":"In this file you can see a property named \"language\" which is set to \"eng\". If you want to deal with documents in other languages, you will need to add the language code (which is the same three letters code of the package). Each specified language must be separated by a '+'.","type":"text"},{"type":"hardBreak"},{"text":"For example, if you want to deal with english and french documents, you will need to set the \"language\" property like this:","type":"text"}]},{"type":"codeBlock","content":[{"text":"language=eng+fra","type":"text"}]},{"type":"paragraph","content":[{"text":"Make your choice and then save the file inside the jar","type":"text"},{"type":"hardBreak"},{"text":"In the 'TesseractOCRConfig.properties' you can also configure other things but it is not the purpose of this documentation to explain them: Google is you friend !","type":"text"},{"type":"emoji","attrs":{"id":"1f609","text":"😉","shortName":":wink:"}},{"type":"hardBreak"},{"text":"Then you need to configure the PDF parser of Tika, because it does not use Tesseract by default to treat PDF image files. Once again the configuration file is located inside the jar file of Tika: /org/apache/tika/parser/pdf/PDFParser.properties","type":"text"},{"type":"hardBreak"},{"text":"This file has an 'ocrStrategy' which is by default set to 'no_ocr'. You can set it with three other options:","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"auto ","type":"text","marks":[{"type":"strong"}]},{"text":": try to extract text, but run OCR if fewer than 10 characters were extracted of if there are more than 10 characters with unmapped Unicode values. This is the most efficient mode but you will loose OCR text on pages containing one or several images on which to perform OCR and a text of more than 10 characters ","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"ocr_only","type":"text","marks":[{"type":"strong"}]},{"text":" : don't bother extracting text, just run OCR on each page. So if you only have text in a pdf and no images it will be a huge amount of process time loss for nothing cause the OCR process will take time but will not retunr any text. Use this mode only if you are absolutely sure that you will only feed Tika with files containing only images to OCR process","type":"text"},{"type":"hardBreak"},{"text":"- ","type":"text"},{"text":"ocr_and_text_extraction","type":"text","marks":[{"type":"strong"}]},{"text":" : Run both OCR and text extraction on each page. This is the mode that will ensure to retrieve everything (text + ocr processed text) but it is also the more expensive mode in terms of resources/processing time","type":"text"},{"type":"hardBreak"},{"text":"So for example if I want the PDF parser to return both \"OCRized\" text and \"natural\" text of PDF files, I will set the 'ocrStrategy' parameter like this:","type":"text"}]},{"type":"codeBlock","content":[{"text":"ocrStrategy ocr_and_text_extraction","type":"text"}]},{"type":"paragraph","content":[{"text":"Set the wanted option to the 'ocrStrategy' parameter then save the file inside the jar","type":"text"},{"type":"hardBreak"},{"text":"NOTE: There is an “extractInlineImages” parameter in the PDFParser.properties file, it is used to change the OCR mode. There are two OCR modes:","type":"text"},{"type":"hardBreak"},{"text":"1/ ","type":"text"},{"text":"render each page and then run OCR on that rendered image","type":"text","marks":[{"type":"strong"}]},{"text":": This is the default mode. In this mode, each pdf page is rendered as a unique image and the OCR process is applied to this image. The advantage is that there is only one OCR process per page so it consumes less resources and is potentially faster than the other mode. The inconvenient is that if there are several images next to each other in the page, the semantic analysis of the OCR process may be wrong and/or have difficulties to process the page, resulting in bad accuracy and lack of performances ","type":"text"},{"type":"hardBreak"},{"text":"2/ ","type":"text"},{"text":"run OCR against inline images","type":"text","marks":[{"type":"strong"}]},{"text":": This is the alternative mode that is enabled be setting the “extractInlineImages” parameter to true. Instead of rendering each pdf page as a unique image, the parser will extract the images as they are in the page and OCR process each one of them. The advantage of this method is that the semantic analysis will be 100% correct in any case but the inconvenient is that if there are many images in a page, the performances will be much worse than the other mode","type":"text"},{"type":"hardBreak"},{"text":"The second mode “run OCR against inline images” MUST NOT BE ENABLED with Datafari when the option “extract archive content” is enabled on the Tika Server connector in a job. Otherwise Tika will process twice the PDF pages in case this mode is enabled, resulting in DRASTIC JOB PERFORMANCE LOSSES ! (at least x3 times slower)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"And last but not least, as the OCR process can also take a very long time, you will also need to tell the Tika server not to kill its task too soon. By default a timeout of 120 seconds is set to a Tika task which means after 2 minutes, if the text extraction of a file is not finished, the task is killed. You can change the value of this timeout thanks to the ","type":"text"},{"text":"TIKA_SPAWN_TASK_TIMEOUT ","type":"text","marks":[{"type":"strong"}]},{"text":"parameter in the /opt/datafari/tika-server/bin/set-tika-env.sh file:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#timeouts are expressed in milliseconds\nexport TIKA_SPAWN_TASK_TIMEOUT=\"-taskTimeoutMillis 1200000\"","type":"text"}]},{"type":"paragraph","content":[{"text":"In our experience with OCR, many files can take over than 10 minutes to be OCR processed so we strongly recommend to set at least 20 minutes of task timeout !","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Once you have completed these two tasks you can run your Tika Server and enjoy the power of the OCR !","type":"text"}]}]}],"version":1}

Browser not supported