JP-4285: Cube Build updates - #10433
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #10433 +/- ##
==========================================
+ Coverage 86.59% 86.86% +0.26%
==========================================
Files 374 374
Lines 40471 40305 -166
==========================================
- Hits 35047 35010 -37
+ Misses 5424 5295 -129 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
I running the regression test to see if I need to fix any tests. |
|
@drlaw1558 One other thing while we are cleaning up the interface for cube build. Currently the parameter 'linear_wave' only works for NIRSpec data. This allows the user to switch between linear and non-linear wavelengths dimensions. The default value is True. For MIRI data, cube_build looks at the data that is to make the cube and decides if it should be linear or non-linear. In general, ifu cubes of a single band are linear and multi channel or multi band cubes can be non-linear. Cubes from spec2 are non-linear and cubes from spec3 are linear. But the user can run cube_build offline and create non-linear cube from spec3 associations by selecting the bands or channels for the cube. |
melanieclarke
left a comment
There was a problem hiding this comment.
This is a very nice simplification of the code!
Can you please add some unit tests for the expected filenames in various cases?
Also, I think we will need a breaking change note, since this removes parameters from the spec.
|
Currently the calwebb_spec2.py file sets output_type = 'multi' for MIRI MRS data and for NIRSpec it is set to 'band' |
|
@melanieclarke I wrote a unit test for testing suffix names but it uses from unittest.mock import MagicMock |
|
@melanieclarke https://github.com/spacetelescope/RegressionTests/actions/runs/27244907925 |
melanieclarke
left a comment
There was a problem hiding this comment.
Test failures are unrelated, from a bad CRDS context. I think everything looks good now, all comments addressed, so I will go ahead and get this merged. Thanks for all the hard work on this one!
See comment below:needs update to calspec2 parameter reference file.Update done.Resolves JP-4285
Closes #10362
This PR addresses how output filenames are created in cube_build. The 'pipeline' parameter was removed from cube along with the option to create 'single' output_type cubes. The 'single' type cubes had been used early in the mission in outlier detection to try and flag bad pixels. In the mode outlier detection called 'cube_build' to create a set of cube for each band. Each input file was mapped to the single band, all the single band s3d files where median combined and blotted back to the detector frame where bad pixels were flagged. The method of outlier detection for IFU data was replaced did not flag bad pixels well and was replaced soon after launch. The hooks to make this types of 'single' ifu cubes was left in cube_build. This pr removed the unnecessary code.
For calspec2 the internal 'step' routines create the default IFU cube names.
For calspec3 cube - cube build uses the asn name and information on the bands to create a string for the output name.
REMAINING ISSUE update to Parameter reference file.
In cube_build the 'output_type' has now been defaulted to 'band'. Only when running calspec2 on MIRI data do
we want the 'output_type' parameter = 'multi'
To run the calspec2 pipeline on miri data - the parameter reference file needs to be updated to
set 'output_type' = multi.
We could hard code it in the calwebb_spec2 code - but it is probably better to have a parameter reference file.
For miri the current calwebb_spec3 parameter reference file is set to
class: jwst.cube_build.cube_build_step.CubeBuildStep
name: cube_build
parameters: {output_type: band, skip: false}
If NIRSPec wants cubes other than 'band' from calspec3 then they should also create a parameter reference file
for calspec3.
Tasks
Build 12.0(use the latest build if not sure)no-changelog-entry-needed)changes/:echo "changed something" > changes/<PR#>.<changetype>.rst(see changelog readme for instructions)changes/<PR#>.breaking.rstnews fragmentdocs/pageRegression test passing: https://github.com/spacetelescope/RegressionTests/actions/runs/26462438602
- [ ] Do truth files need to be updated ("okified")?
- [ ] after the reviewer has approved these changes, run
okify_regteststo update the truth files