Chansoo/ensemble jobhandler event wait

[chansoolee]

---

Pull Request Description

What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)

#2567

What are the significant changes in functionality due to this change request?

This change stabilizes EnsembleModel job completion when using the threaded parallel strategy. Previously, each ensemble evaluation thread repeatedly polled JobHandler.isThisJobFinished(), which acquired the shared JobHandler queue lock. With many concurrent ensemble evaluations, this polling could create enough lock contention to prevent the JobHandler polling loop from promptly moving completed jobs into the finished queue, producing hangs even after subprocesses had completed.

The JobHandler now creates a per-job threading.Event when a job is submitted, sets that event when the job is moved to the finished queue, and exposes getJobEvent() so EnsembleModel can wait on job completion without repeatedly acquiring the queue lock. The previous polling path remains as a fallback if no event is available.

This PR also improves failed sub-model handling in EnsembleModel. Failed evaluations now return no evaluation and allow the existing failed-run handling path to process the failure, instead of raising immediately and potentially leaving nested ensemble jobs unclaimed. Cleanup is limited to models that were actually submitted before the failed model, avoiding waits on downstream models that were never launched.

Additionally, MessageHandler.printWarnings() now clears stored warnings after printing so warnings are not reprinted later by subsequent error handling. A focused JobHandler unit test was added to verify event creation, event signaling on completion, and event cleanup after finished-job collection.

Tests performed

The following checks were run in /home/leec3/projects/raven_idaholab_issue2567:

• python -m py_compile ravenframework/JobHandler.py ravenframework/MessageHandler.py ravenframework/Models/EnsembleModel.py tests/framework/unit_tests/JobHandler/testJobHandlerEvents.py
• python -m py_compile ravenframework/CustomModes/PBSSimulationMode.py ravenframework/CustomModes/MPILegacySimulationMode.py
• ./run_tests --test-dir tests/framework/unit_tests/JobHandler
• ./run_tests --test-dir tests/framework/ensembleModelTests --re 'testEnsembleModelLinearThread|testEnsembleModelLinearExpectedFailure|testEnsembleModelWithCode' -j 3
• git diff --check

Cluster/qsub validation was also run through Slurm:

• Slurm job 1873408
• Command path: ./run_tests --raven --test-dir tests/cluster_tests/InternalParallel --re 'test_parallel_ensemble_linear' --run-types qsub -j 1
• Result: COMPLETED, exit code 0:0
• ROOK result: PASSED: 1, FAILED: 0
• Final run_tests result: Tested sets: RAVEN PluginAPI, ... tests passed!

---

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

• 1. Review all computer code.
• 2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
• 3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
• 4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass.
• 5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in XML block, the node <internalParallel> to True.
• 6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync.
• 7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done.
• 8. If an analytic test is changed/added is the the analytic documentation updated/added?
• 9. If any test used as a basis for documentation examples (currently found in raven/tests/framework/user_guide and raven/docs/workshop) have been changed, the associated documentation must be reviewed and assured the text matches the example.

[wangcj05]

@chansoolee I have few comments for you to consider.

[wangcj05]

changes are good.

[wangcj05]

tests are good also.