Skip to content

Quality report refactor#1510

Open
mmccrackan wants to merge 16 commits intomasterfrom
260102_qr
Open

Quality report refactor#1510
mmccrackan wants to merge 16 commits intomasterfrom
260102_qr

Conversation

@mmccrackan
Copy link
Contributor

Reworks quality reporting code to improve h5 storage and add NEP plots.

@mmccrackan mmccrackan marked this pull request as draft January 4, 2026 22:33
keys = ['time', 'num_valid_dets', 'wafer.bandpass']#, '"wafer_slot"::tag', '"wafer.bandpass"::tag']
keys = ['time', 'num_valid_dets', 'bandpass']
if cfg.platform == "lat":
keys += ['array_net_T', 'det_net_T']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these nets or neps?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're now NEPs, though I guess since this is inconsistent in a few places we should just rename the Influx fields to something generic like noise.

@mmccrackan
Copy link
Contributor Author

mmccrackan commented Jan 21, 2026

Okay, I think I've incorporated discussions from the previous site-pipeline call into this. Compared to the version currently running on site, this PR implements the following:

  • Overhaul saving of the data so that things like arrays aren't saved as strings. Do not save the entire PWV array, just the values for each obs_id.
  • Exposes some of the main config options to the main functions so they can be configured in Prefect.
  • Longterm dataset now accumulates when monthly reports are run.
  • Add NEP vs PWV and NEP vs Time plots.
  • Add obsdb value vs time plot. Currently have boresight angle, hwp speed, and corotator angle. Will add scan params and PWV when those become available in the regular obsdb.
  • Added a coverage plot. Since this is slow, this is parallelized in the same way as preprocess_tod. It will take ~10 mins for the SATs and maybe 1-2 hours for the LAT on 4 processes. It saves a FITS file and a png file (the latter is what is plotted).
  • Rearranged the plots into separate rows so everything mostly lines up in time.
  • More stats at the top.
  • Added a "weekly/latest" and "monthly/latest" to the drop down box at the top.
  • Raises an error when any reports fail to prevent silent failures.

Example here: https://site.simonsobs.org/~mmccrackan/qr/satp3/weekly/20260107_20260114/report.html

@mmccrackan mmccrackan marked this pull request as ready for review January 21, 2026 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants