Site-pipeline: Initial DetCal calculation needs to be more robust to missing .npy files.

[kmharrington]

We have an example of update_smurf_caldbs failing when one IV from an observation is missing: https://prefect.simonsobs.org/flow-runs/flow-run/9e78422f-43be-44c4-a590-cf4a63a437db

In obs_1762625109_satp3_1111111, oper_1762623919_satp3_1000000 is missing a .npy file. I have checked at this file does not exist at Level 2, so something failed in the IV taking/analysis on the smurf server. (Very rare error, we haven't seen this in awhile).

Right now, we're going to have to add obs_1762625109_satp3_1111111 to the failed_obsids.yaml. But longer-term we should also edit this code to continue to run for the other wafers/slots so that we don't lose this whole observation.

This area of the code is what needs to be fixed:

sotodlib/sotodlib/site_pipeline/update_smurf_caldbs.py

Line 263 in 5040d5e

ivas[dset] = load_smurf_npy_data(ctx, oid, 'iv')

[mmccrackan]

This same error is also now happening on the LAT for the following obs_ids:

obs_1763429876_lati1_111
obs_1763430130_lati1_111

The first failed Prefect flow is here:
https://prefect.simonsobs.org/flow-runs/flow-run/2a30750e-55b5-430e-a3e3-d937f23bb9a0

[mmccrackan]

This is happening on SATp2 now for obs_1766029460_satp2_1111111

https://prefect.simonsobs.org/flow-runs/flow-run/9914264d-e6a6-43a9-bd93-138f6295ca58

Added to failed_obsids.yaml.