Configuring Google Cloud Vision OCR appliance for DMS

This page describes the setup of a Google Cloud based OCR setup for use with the Extended DMS environment. DMS will issue commands so that documents to be OCR'ed are uploaded to Google Cloud, OCR is performed there, and the result is returned. The advantage is that the DMS appliance's CPU is not consumed by the OCR processing and that the OCR process takes advantage of the high accuracy and potential speed of the Google Cloud Vision technology.

Note that the Google Cloud service is not free of charge an requires an active Google Cloud account. See: https://cloud.google.com/vision/pricing

Also note that the performance of this OCR method is largely determined by your internet connection speed as all pdf documents must be uploaded to your Google Storage bucket. In addition to this, depending on setup, this OCR method may consume significant RAM if multiple parallel OCR processes are configured.

The below instructions describe the setup of the DMS host machine. Using this OCR method will not affect performance of the DMS service in a significant way; the scripts involved are only making calls to the Google Cloud API and relay documents/return information, however, they do not perform any processing of the documents.

Components and setup required:

1. Configure Google Cloud account: enable Google Vision API there and create Google Cloud Storage bucket for temp files created while processing
2. Configure Google Cloud SDK on DMS host machine: (i) see https://cloud.google.com/sdk/install and note that if you installed your DMS from the base OVA image provided by Patrix/Practice Insight, you operating system is CentOS. (ii) Further see https://cloud.google.com/sdk/docs/initializing and https://cloud.google.com/sdk/docs/authorizing.
3. Prepare service account key in JSON format for Google Cloud here: https://console.cloud.google.com/apis/credentials/serviceaccountkey

4. Create 2 permanent environment variables. For example, add the following 2 lines to /etc/environment:

   GCLOUD_OCR_BUCKET=<$gcloud_storage_bucket_name>
   GOOGLE_APPLICATION_CREDENTIALS=<$path_to_json>

   replace <$gcloud_storage_bucket_name> and <$path_to_json> with their resective values

5. Install Java Development Kit in the DMS VM by running

   yum install java-1.8.0-openjdk

6. Download pi-gcloud-ocr.jar (https://www.pace-ip.com/edms/downloads/components/pi-google-ocr.jar) and copy it to /opt/pi-gcloud-ocr.jar of the DMS VM host machine
7. Copy the below "pi-google-ocr" script to /usr/bin/ of the DMS VM host machine and make it executable (chmod +x). This script contains the commands to drive the OCR process for each file
8. Copy the below "piocr" script to /<storagepath>/nuxeo/scripts/ and make it executable (chmod +x). This script contains the commands to access pi-google-ocr script from inside the DMS
9. Create ssh key in DMS VM host machine using "ssh-keygen" command, copy public key (id_rsa.pub) and paste into <<adminuserhome>>/.ssh/authorized_keys file of DMS VM host machine (i.e. the same box)
10. DMS host needs /<storagepath>/nuxeo/data mounted to /var/lib/nuxeo/data and /<storagepath>/nuxeo/tmp mounted to /opt/nuxeo/server/tmp (see below section "Deploy Script changes")
11. Login from DMS VM host machine to itself using ssh. After successful login, move the DMS VM host machine's key files "id_rsa", "id_rsa.pub" and "known_hosts" from ~/.ssh/ to /<storagepath>/nuxeo/ssh/ 
12. Set permissions for the files in /<storagepath>/nuxeo/ssh/ folder to 1000:1000
13. Set key "ocr.engine.name" in PAT_DMS_SETTINGS table of Patricia db to "piocr"
14. Add below change commands to the "commands.conf" file of auto-deploy client specific repository (also found in ~/deploy/config/)
15. Start re-deploy of DMS using ~/deploy/deploy_script/deploy.sh command as outlined here: Auto-deploy script

Scripts:

#!/usr/bin/env bash
gsutil cp "$2" gs://${GCLOUD_OCR_BUCKET}
input_filename=$(basename $2)
output_filename=$(basename $1)
json=$(java -jar /opt/pi-google-ocr.jar gs://${GCLOUD_OCR_BUCKET}/${input_filename} gs://${GCLOUD_OCR_BUCKET}/${output_filename})
echo "$json"  > "$1"
gsutil rm "gs://${GCLOUD_OCR_BUCKET}/${input_filename}"
gsutil rm "gs://${GCLOUD_OCR_BUCKET}/${output_filename}*"

#!/bin/bash
echo "variables $1 $2"
touch ~/.ssh_config && ssh -F ~/.ssh_config <<adminuser>>@<<dms.host.name>> "pi-google-ocr $1 $2"

  add_volume "/<storage_path>/nuxeo/ssh" "/home/nuxeo/.ssh"
  add_volume "/<storage_path>/nuxeo/scripts/piocr" "/usr/local/bin/piocr"