Initial upload of optimizers.

[astroRez]

Initial upload of MIRI-LRS, NIRSpec-PRISM, NIRSpec-G395H, and NIRISS-SOSS parametric optimizers.

[kevin218]

I've reviewed a few files and have made suggestions on how to economize the code (i.e. reduce duplication). I'm happy to chat more after you review my suggestions.

[kevin218]

Here are some additional suggestions based on my experiences in playing with the code:

• Move the optimizer demo files into their own folder within demo.
• Convert the .txt files to .ecf files and use the built-in ECF reader.
• Create a new folder (e.g., src/eureka/optimizer) and place files there.
• Create opt_meta.py and set all default values there. See s3_meta.py as an example for how to define global default values and instrument-specific default values.
• As a file naming convention, any place we have s#, use opt instead.
• I think we can delete spec_hw_selection and bg_hw_selection in favor of using default values.
• Make sure to define the functions of both manual and auto in the comments.
• In the ECF, allow users to specify a list of parameters that they wish to optimize. If set to None, the default will be to optimize all possible parameters.
• Should we start by optimizing bg_hw and spec_hw first?
• Try to create as many generic functions as possible to cut down on repeating code.
• Use Eureka!'s log capabilities instead of print statements. This will mean creating a log file in a folder that doesn't get deleted.
• When reporting the best parameter and fitness values, be sure to specify which parameter was being optimized.
• Move all plotting functions to plots_opt.py and follow the figure naming scheme (ask me for details).
• The content in optimization_results.txt should go into the log file.
• Instead of having if __name__ == "__main__": at the top of run_optimizer.py, put everything into a function (e.g., def parametric(eventlabel, ecf_path=ecf_path)) that can be called byrun_eureka.py using opt.parametric(eventlabel, ecf_path=ecf_path) or similar.
• I think you can remove if optimizer == "parametric" and __name__ == "__main__": for each step in run_optimizer.py.

[codecov]

Codecov Report

❌ Patch coverage is 50.30769% with 323 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.98%. Comparing base (1eed752) to head (0014414).
⚠️ Report is 45 commits behind head on main.

Files with missing lines	Patch %	Lines
src/eureka/optimizer/S1opt_optimizer.py	0.00%	141 Missing ⚠️
src/eureka/optimizer/S1opt_meta.py	0.00%	48 Missing ⚠️
src/eureka/optimizer/S3opt_meta.py	61.25%	31 Missing ⚠️
src/eureka/optimizer/objective_funcs.py	68.88%	28 Missing ⚠️
src/eureka/optimizer/optimizers.py	59.42%	28 Missing ⚠️
src/eureka/S3_data_reduction/niriss.py	15.38%	22 Missing ⚠️
src/eureka/optimizer/S3opt_optimizer.py	88.40%	16 Missing ⚠️
src/eureka/S3_data_reduction/s3_meta.py	73.33%	4 Missing ⚠️
src/eureka/S1_detector_processing/s1_meta.py	0.00%	2 Missing ⚠️
src/eureka/S3_data_reduction/bright2flux.py	50.00%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #757      +/-   ##
==========================================
- Coverage   60.18%   59.98%   -0.20%     
==========================================
  Files         107      114       +7     
  Lines       14084    14722     +638     
==========================================
+ Hits         8476     8831     +355     
- Misses       5608     5891     +283     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kevin218]

@taylorbell57 I've made significant changes to this PR, so I would appreciate your independent review.

[kevin218]

@taylorbell57 After running more extensive tests and tracking down some corner-case issues, I think this PR is finally ready for review.

[taylorbell57]

This is on my to-do list and I haven't forgotten it, FYI - I've just been swamped with research travel, a week of sick leave, and a week of personal vacation

[taylorbell57]

I haven't gotten into reviewing the major edits of this PR yet, but here's some preliminary feedback

[taylorbell57]

Here are a couple more comments on some basic aspects before I dive into the bigger parts of this PR on Thursday/Friday this week.

Please also note that there will need to be a bunch of additions to the docs to describe these new features and plots.

[taylorbell57]

meta.inputdir should be normalized to always end with an os.sep already. If that wasn't the case for you somewhere, we should fix that at the source to avoid having a tonne of os.sep scattered throughout the code

[kevin218]

inputdir is either user defined or as follows in s3_meta.py: self.inputdir = getattr(self, 'inputdir', 'Stage2'). The formatting is similar for other stages. While I support minimizing the use of os.sep, I feel this proposed change is outside the scope of this PR.

[taylorbell57]

inputdir is regularized within readECF here:

Eureka/src/eureka/lib/readECF.py

Lines 239 to 253 in 764a234

	self.inputdir_raw = self.inputdir
	self.outputdir_raw = self.outputdir
	# Join inputdir_raw and outputdir_raw to topdir for convenience
	# Use split to avoid issues from beginning
	self.inputdir = os.path.join(self.topdir,
	*self.inputdir.split(os.sep))
	self.outputdir = os.path.join(self.topdir,
	*self.outputdir.split(os.sep))
	# Make sure there's a trailing slash at the end of the paths
	if self.inputdir[-1] != os.sep:
	self.inputdir += os.sep
	if self.outputdir[-1] != os.sep:
	self.outputdir += os.sep

[taylorbell57]

How time consuming would it be to add support for running/skipping flat-field in Stage 2?

[kevin218]

Adding a Stage 2 optimization would be quite a bit of work for this PR, but we could explore that option in a subsequent PR.

[taylorbell57]

Can you open an Issue to look into that so we don't forget