Merge branch 'develop' into master

Author: mhucka

CHANGES.md

@@ -5,12 +5,15 @@ Version 1.3.0
 --------------
 
 * Handprint now requires Python version 3.6 or later.
+* Fixed issue [#19](https://github.com/caltechlibrary/handprint/issues/19), which caused Handprint to fail to produce any output images if both `-e` and `-G` were given.
+* Fixed warning about Matlplotlib GUIs and threading in `images.py`.
+* Fixed macOS Mojave compatibility (issue [#16](https://github.com/caltechlibrary/handprint/issues/16).
 * Updated `handprint/services/microsoft.py` to work with Azure API v. 3.0.
+* Updated Microsoft credentials code to allow the endpoint URI to be supplied.
 * Changed and expanded the possible exit codes returned by Handprint. (Please see the docs for more info.)
 * Changed Google interface to retrieve _only_ document text results instead of all possible results, for better efficiency.
 * Improved handling of `^C` interrupts from the command line.
 * Added signal catcher to drop Handprint into `pdb` upon receiving `SIGUSER`.
-* Fixed warning about Matlplotlib GUIs and threading in `images.py`.
 * Switched to the use of [Sidetrack](https://github.com/caltechlibrary/sidetrack) for debug logging.
 * Switched to the use of [Rich](https://github.com/willmcgugan/rich) for terminal output
 * Various internal code updates and refactoring.

README.md

@@ -14,9 +14,7 @@ The _**Hand**written **P**age **R**ecognit**i**o**n** **T**est_ program applies
 ❡ Log of recent changes
 -----------------------
 
-_Version 1.2.2_: This release fixes an inconsistency (compared to other Handprint output) in the way that Microsoft's text recognition service results are drawn on annotated images.  The copyright year in source files has also been updated.
-
-_Version 1.2.0_: This version fixes a bug in creating annotated results images, in which results from multiple services were overwritten on top of each other. It also fixes a bug with the Amazon interface that resulted in occasional random errors about `endpoint_resolver`. This version of Handprint also changes how output files are written; the new scheme uses the naming pattern `somefile.handprint.png` for the rescaled input image, `somefile.handprint-service.png` for the various service output results, and `somefile.handprint-all.png` for the summary grid image.  Handprint now also accepts PDF files as input.
+_Version 1.3.0_: This release brings a number of changes: (1) it now requires Python version 3.6 at minimum; (2) the Microsoft service interface now uses Azure API v3.0; (3) the Microsoft credentials scheme now allows you to change the endpoint URI; (4) the Google service interface now only gets the document text results instead of all possible results; (5) the possible program exit codes have changed slightly; and (6) interruption via <kbd>^C</kbd> should work better now.  Some bugs have been fixed and internals have been (hopefully) improved.
 
 The file [CHANGES](CHANGES.md) contains a more complete change log, and includes information about previous releases.
 
@@ -89,15 +87,18 @@ _SERVICENAME_ must be one of the service names printed by running `handprint -l`
 
 #### Microsoft
 
-Microsoft's approach to credentials in Azure involves the use of [subscription keys](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/vision-api-how-to-topics/howtosubscribe).  The format of the credentials file for Handprint just needs to contain a single field:
+Microsoft's approach to credentials in Azure involves the use of [subscription keys](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/vision-api-how-to-topics/howtosubscribe).  The format of the credentials file for Handprint needs to contain two fields:
 
 ```json
 {
- "subscription_key": "YOURKEYHERE"
+ "subscription_key": "YOURKEYHERE",
+ "endpoint": "https://ENDPOINT"
 }
 ```
 
-The value of "YOURKEYHERE" will be a string such as `"18de248475134eb49ae4a4e94b93461c"`.  To obtain a key, visit [https://portal.azure.com](https://portal.azure.com) and sign in using your account login.  (Note: you will need to turn off browser security plugins such as Ad&nbsp;Block and uMatrix if you have them, or else the site will not work.)  Once you are authenticated to the Azure portal, you can create credentials for using Azure's machine-learning services. Some notes about this can be found in the [Handprint project Wiki pages on GitHub](https://github.com/caltechlibrary/handprint/wiki/Getting-Microsoft-Azure-credentials).  Once you have obtained a key, use a text editor to create a JSON file in the simple format shown above, save that file somewhere on your computer (for the sake of this example, assume it is `myazurecredentials.json`), and use the command discussed above to make Handprint copy the credentials file:
+The value `"YOURKEYHERE"` will be a string such as `"18de248475134eb49ae4a4e94b93461c"`, and it will be associated with an endpoint URI such as `"https://westus.api.cognitive.microsoft.com"`.  To obtain a key and the corresponding endpoint URI, visit [https://portal.azure.com](https://portal.azure.com) and sign in using your account login.  (Note: you will need to turn off browser security plugins such as Ad&nbsp;Block and uMatrix if you have them, or else the site will not work.)  Once you are authenticated to the Azure portal, you can create credentials for using Azure's machine-learning services.  Some notes all about this can be found in the [Handprint project Wiki pages on GitHub](https://github.com/caltechlibrary/handprint/wiki/Getting-Microsoft-Azure-credentials).
+
+Once you have obtained both a key and an endpoint URI, use a text editor to create a JSON file in the simple format shown above, save that file somewhere on your computer (for the sake of this example, assume it is `myazurecredentials.json`), and use the command discussed above to make Handprint copy the credentials file:
 ```sh
 handprint -a microsoft myazurecredentials.json
 ```
@@ -322,7 +323,7 @@ Handprint produces color-coded diagnostic output as it runs, by default.  Howeve
 
 Handprint will send files to the different services in parallel, using a number of process threads equal to 1/2 of the number of cores on the computer it is running on.  (E.g., if your computer has 4 cores, it will by default use at most 2 threads.)  The `-t` option (`/t` on Windows) can be used to change this number.
 
-If given the `-@` argument (`/@` on Windows), this program will output a detailed trace of what it is doing. The debug trace will be sent to the given destination, which can be `-` to indicate console output, or a file path to send the output to a file.  Handprint will also install a signal handler that responds to signal `SIGUSR1`; if the signal is sent to the running process, it will drop Handprint into the `pdb` debugger.  _Note_: It's best to use `-t 1` when attempting to use a debugger because otherwise subthreads will continue running even if the main thread is interrupted.
+If given the `-@` argument (`/@` on Windows), this program will output a detailed trace of what it is doing.  The debug trace will be sent to the given destination, which can be `-` to indicate console output, or a file path to send the output to a file.  On non-Windows platforms, Handprint will also install a signal handler that responds to signal `SIGUSR1`; if the signal is sent to the running process, it will drop Handprint into the `pdb` debugger.  _Note_: It's best to use `-t 1` when attempting to use a debugger because otherwise subthreads will continue running even if the main thread is interrupted.
 
 If given the `-V` option (`/V` on Windows), this program will print the version and other information, and exit without doing anything else.
 
@@ -418,11 +419,13 @@ Handprint benefitted from feedback from several people, notably from Tommy Keswi
 
 Handprint makes use of numerous open-source packages, without which it would have been effectively impossible to develop Handprint with the resources we had.  I want to acknowledge this debt.  In alphabetical order, the packages are:
 
+* [aenum](https://pypi.org/project/aenum/) &ndash; advanced enumerations for Python
 * [appdirs](https://github.com/ActiveState/appdirs) &ndash; module for determining appropriate platform-specific directories
 * [boltons](https://github.com/mahmoud/boltons/) &ndash; package of miscellaneous Python utilities
 * [boto3](https://github.com/boto/boto3) &ndash; Amazon AWS SDK for Python
 * [colorama](https://github.com/tartley/colorama) &ndash; makes ANSI escape character sequences work under MS Windows terminals
 * [colored](https://gitlab.com/dslackw/colored) &ndash; library for color and formatting in terminal
+* [dateparser](https://pypi.org/project/dateparser/) &ndash; parse dates in almost any string format
 * [google-api-core, google-api-python-client, google-auth, google-auth-httplib2, google-cloud, google-cloud-vision, googleapis-common-protos, google_api_python_client](https://github.com/googleapis/google-cloud-python) &ndash; Google API libraries 
 * [halo](https://github.com/ManrajGrover/halo) &ndash; busy-spinners for Python command-line programs
 * [humanize](https://github.com/jmoiron/humanize) &ndash; make numbers more easily readable by humans

handprint/__main__.py

@@ -340,14 +340,19 @@ def main(add_creds = 'A', base_name = 'B', compare = False, no_color = False,
 
     prefix = '/' if sys.platform.startswith('win') else '-'
     hint = f'(Hint: use {prefix}h for help.)'
+    ui = UI('Handprint', 'HANDwritten Page RecognitIoN Test',
+            use_color = not no_color, be_quiet = quiet)
+    ui.start()
 
     # Preprocess arguments and handle early exits -----------------------------
 
     if debug != 'OUT':
         if __debug__: set_debug(True, debug, extra = '%(threadName)s')
         import faulthandler
         faulthandler.enable()
-        pdb_on_signal(signal.SIGUSR1)
+        if not sys.platform.startswith('win'):
+            # Even with a different signal, I can't get this to work on Win.
+            pdb_on_signal(signal.SIGUSR1)
 
     # Handle arguments that involve deliberate exits.
 
@@ -375,6 +380,7 @@ def main(add_creds = 'A', base_name = 'B', compare = False, no_color = False,
 
     # Do sanity checks on some other arguments.
 
+    services = services_list() if services == 'S' else services.lower().split(',')
     if services != 'S' and not all(s in services_list() for s in services):
         alert_fatal(f'"{services}" is not a known services. {hint}')
         exit(int(ExitCode.bad_arg))
@@ -393,20 +399,17 @@ def main(add_creds = 'A', base_name = 'B', compare = False, no_color = False,
     # Do the real work --------------------------------------------------------
 
     if __debug__: log('='*8 + f' started {timestamp()} ' + '='*8)
-    ui = body = exception = None
+    body = exception = None
     try:
-        ui = UI('Handprint', 'HANDwritten Page RecognitIoN Test',
-                use_color = not no_color, be_quiet = quiet)
-        ui.start()
         body = MainBody(files      = files,
                         from_file  = None if from_file == 'F' else from_file,
                         output_dir = None if output_dir == 'O' else output_dir,
                         add_creds  = None if add_creds == 'A' else add_creds,
                         base_name  = 'document' if base_name == 'B' else base_name,
                         make_grid  = not no_grid,
                         extended   = extended,
+                        services   = services,
                         threads    = max(1, cpu_count()//2 if threads == 'T' else int(threads)),
-                        services   = services_list() if services == 'S' else services.lower().split(','),
                         compare    = 'relaxed' if (compare and relaxed) else compare)
         body.run()
         exception = body.exception
@@ -423,19 +426,20 @@ def main(add_creds = 'A', base_name = 'B', compare = False, no_color = False,
 
     exit_code = ExitCode.success
     if exception:
-        if type(exception) == CannotProceed:
-            exit_code = exception.args[0]
-        elif type(exception) in [KeyboardInterrupt, UserCancelled]:
+        if exception[0] == CannotProceed:
+            exit_code = exception[1].args[0]
+        elif exception[0] in [KeyboardInterrupt, UserCancelled]:
             if __debug__: log(f'received {exception.__class__.__name__}')
+            warn('Interrupted.')
             exit_code = ExitCode.user_interrupt
         else:
+            msg = str(exception[1])
+            alert_fatal(f'Encountered error {exception[0].__name__}: {msg}')
+            exit_code = ExitCode.exception
             if __debug__:
                 from traceback import format_exception
-                msg = str(exception[1])
-                details = ''.join(format_exception(*exception))
-                alert_fatal(f'Error: {msg}')
+                details = ''.join(format_exception(exception))
                 logr(f'Exception: {msg}\n{details}')
-            exit_code = ExitCode.exception
     else:
         inform('Done.')
     if __debug__: log('_'*8 + f' stopped {timestamp()} ' + '_'*8)

handprint/credentials/microsoft_auth.py

@@ -26,6 +26,12 @@
 from .base import Credentials
 from .credentials_files import credentials_filename
 
+
+# Constants.
+# .............................................................................
+
+_DEFAULT_ENDPOINT = 'https://westus.api.cognitive.microsoft.com'
+
 
 # Main class.
 # .............................................................................
@@ -42,7 +48,15 @@ def __init__(self):
         try:
             with open(cfile, 'r') as file:
                 creds = json.load(file)
-                self.credentials = creds['subscription_key']
+                if 'endpoint' in creds:
+                    endpoint = creds['endpoint'].rstrip('/')
+                    if not endpoint.startswith('http'):
+                        endpoint = 'https://' + endpoint
+                else:
+                    if __debug__: log('endpoint not found; using default')
+                    endpoint = _DEFAULT_ENDPOINT
+                creds['endpoint'] = endpoint
+                self.credentials = creds
         except Exception as ex:
             raise AuthFailure(
                 'Unable to parse Microsoft exceptions file: {}'.format(str(ex)))

handprint/main_body.py

@@ -26,6 +26,7 @@
 from handprint import _OUTPUT_EXT, _OUTPUT_FORMAT
 from handprint.credentials import Credentials
 from handprint.exceptions import *
+from handprint.exit_codes import ExitCode
 from handprint.files import filename_extension, filename_basename
 from handprint.files import files_in_directory, filter_by_extensions
 from handprint.files import readable, writable, is_url
@@ -58,10 +59,6 @@ def __init__(self, **kwargs):
         self._manager = Manager(self.services, self.threads, self.output_dir,
                                 self.make_grid, self.compare, self.extended)
 
-
-    def run(self):
-        '''Run the main body.'''
-
         # On Windows, in Python 3.6+, ^C in a terminal window does not stop
         # execution (at least in my environment).  The following function
         # creates a closure with the worker object so that stop() can be called.
@@ -78,24 +75,23 @@ def ctrl_handler(event, *args):
             import win32api
             win32api.SetConsoleCtrlHandler(ctrl_handler, True)
 
+
+    def run(self):
+        '''Run the main body.'''
+
         if __debug__: log('running MainBody')
         try:
             self._do_preflight()
             self._do_main_work()
         except (KeyboardInterrupt, UserCancelled) as ex:
             # This is the place where we land when Handprint receives a ^C.
             if __debug__: log(f'got {type(ex).__name__}')
-            warn('Interrupted.')
             interrupt()
             self.stop()
-            self.exception = ex
-        except CannotProceed as ex:
-            if __debug__: log(f'got CannotProceed')
-            self.exception = (CannotProceed, ex)
+            self.exception = sys.exc_info()
         except Exception as ex:
             if __debug__: log(f'exception in main body: {str(ex)}')
             self.exception = sys.exc_info()
-            alert_fatal(f'Error occurred during execution:', details = str(ex))
         if __debug__: log('finished MainBody')
 
 

handprint/manager.py

@@ -149,6 +149,7 @@ def run_services(self, item, index, base_name):
             return
 
         # Send the file to the services and get Result tuples back.
+        self._senders = []
         if self._num_threads == 1:
             # For 1 thread, avoid thread pool to make debugging easier.
             results = [self._send(image, s) for s in services]
@@ -163,6 +164,9 @@ def run_services(self, item, index, base_name):
         # If a service failed for some reason (e.g., a network glitch), we
         # get no result back.  Remove empty results & go on with the rest.
         results = [x for x in results if x is not None]
+        if not results:
+            warn(f'Nothing to do for {item}')
+            return
 
         # Create grid file if requested.
         if self._make_grid:
@@ -202,8 +206,10 @@ def _get(self, item, base_name, index):
         if is_url(item):
             # First make sure the URL actually points to an image.
             if __debug__: log(f'testing if URL contains an image: {item}')
+            headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64)'}
             try:
-                response = urllib.request.urlopen(item)
+                request = urllib.request.Request(item, None, headers)
+                response = urllib.request.urlopen(request)
             except Exception as ex:
                 warn(f'Skipping URL due to error: {ex}')
                 return (None, None)
@@ -256,7 +262,6 @@ def _send(self, image, service):
         directory "dest_dir".
         '''
 
-        raise_for_interrupts()
         service_name = f'[{service.name_color()}]{service.name()}[/]'
         inform(f'Sending to {service_name} and waiting for response ...')
         last_time = timer()
@@ -272,21 +277,21 @@ def _send(self, image, service):
                 warn(f'Continuing {service_name}')
                 return self._send(image, service)
         if output.error:
-            alert(f'{service_name} failed: {output.error}')
+            # Sanitize the error string in case it contains '{' characters.
+            msg = output.error.replace('{', '{{{{').replace('}', '}}}}')
+            alert(f'{service_name} failed: {msg}')
             warn(f'No result from {service_name} for {relative(image.file)}')
             return None
-
-        raise_for_interrupts()
         inform(f'Got result from {service_name}.')
+        raise_for_interrupts()
+
+        inform(f'Creating annotated image for {service_name}.')
         file_name   = path.basename(image.file)
         base_path   = path.join(image.dest_dir, file_name)
-        annot_path  = None
+        annot_path  = self._renamed(base_path, str(service), 'png')
         report_path = None
-        if self._make_grid:
-            annot_path = self._renamed(base_path, str(service), 'png')
-            inform(f'Creating annotated image for {service_name}.')
-            with self._lock:
-                self._save(annotated_image(image.file, output.boxes, service), annot_path)
+        with self._lock:
+            self._save(annotated_image(image.file, output.boxes, service), annot_path)
         if self._extended_results:
             txt_file  = self._renamed(base_path, str(service), 'txt')
             json_file = self._renamed(base_path, str(service), 'json')

handprint/services/google.py

@@ -21,8 +21,6 @@
 import google
 from google.cloud import vision_v1p3beta1 as gv
 from google.api_core.exceptions import PermissionDenied
-from google.cloud.vision import enums
-from google.cloud.vision import types
 from google.protobuf.json_format import MessageToDict
 import json