Multiple registry endpoints

[Zarquan]

Update the registry client to handle multiple service endpoints and retry requests in they fail.

Needed to make use of backup registry services for SKAO.
https://jira.skatelescope.org/browse/SP-5555

[pdowler]

The required config file cadc-registry.properties currently has a single item for the base URL of the reg service to use:

ca.nrc.cadc.reg.client.RegistryClient.baseURL = https://{server-name}/reg

Background:

The RegistryClient makes a call to {baseURL}/resource-caps which is theoretically a canned query on an IVOA registry: select resource_id, access_url from ... where standard_id={VOSI-capabilities}. In practice the reg service is simply configured with that query result and there is no actual registry db behind it. So this is a registry service that only supports query by service identifier and defers other details to reading the capabilities document.

The results (of the canned query and reading capabilities documents) are cached locally because the expectation is that the same lookups happen a lot. The cache is text (dirs and files) and supports multiple registries already.

[pdowler]

To make operations more robust, one could deploy multiple reg services and configure all of them on the client side. The RegistryClient could cycle through and call each one until it succeeds. As mentioned above, the local caching can already handle this kind of usage.

In principle, every service in every node that requires cadc-registry.properties can/should use the exact same config file (intentional); there isn't any use for local customisation right now.

The subtle thing is knowing if the multiple instances are strictly mirrors or more generally just different registry services with possibly different content. I think this effects two things:

1. if RegistryClient thinks they are mirrors, it could return the first match; inconsistencies would show up as sometimes getting result A and sometimes B
2. if RegistryClient thinks they are mirrors, not-found could return after one successful query
3. if RegistryClient thinks there is different content, it could check multiple services and fail when the results are inconsistent, but then it could also find services that were only findable in one of the registries; higher latency on not-found isn't really that bad and with the caching it would not be noticeable.

Resolving inconsistencies would require more information (different canned query) so could be out-of-scope (but see below).

I think if this client was really using multiple IVOA RegTAP services (with their own in-sync or out-of-sync issues) then assuming different content is correct. This is how I would think about this.

So, the immediate thing would be to allow multiple baseURL(s) in the config and have an error handling loop around the canned query. That happens in RegistryClient.getAccessURL where it calls cachedCapSource.getContent(), lines 233-239.

---

aside: the config could be more specific and have a preferred (more trusted) baseURL to try first (eg using the current config property); then additional reg services could be configured to be alternatives with a different config property: then we would probably just ignore inconsistencies.

in principle, reg could be deployed at every node and services at the node could use the local reg and fallback to the global one, or fallback to any number of other mirrors... that kind of deployment (local) means that local operators are motivated to keep their own reg up and running and can fix it locally if it goes down. The ideas I saw about 1-2 in Europe, one in NA, and one in Asia seems more complicated than necessary... each node could decide to do it themselves and/or rely on a few others.

[pdowler]

aside: issue #182 nominally tackles the different registries have different content in another way, with the idea that people could run a local reg with their own services and configure it to make calls to other known/trusted registries. It does not address robustness (single-point-of-failure) per se, except that it's local and cacheable. It is related and potentially better than all clients (of reg) knowing about all registries.