System dumps may not capture critical stack frame locals for a Java heap OutOfMemoryError #23984

Java -version output

JRE 1.8.0 Linux amd64-64-Bit Compressed References 20250625_97093 (JIT enabled, AOT enabled)
OpenJ9   - 5745bbce054
OMR      - e194079e7
IBM      - e7592ac

Summary of problem

In support case TS022088420, the system dump produced by a Java heap OutOfMemoryError caused significant confusion because it wasn't clear what was the proximate cause of the heap exhaustion. The support case details the evidence but we could deduce with very high confidence that the relevant stack frame locals driving the OOM were not in the system dump.

It appears this is due to a race condition inherent in the design of the dump agent system which is that the OutOfMemoryError system dump agent needs to acquire exclusive access to produce the dump.

While one thread was processing the system dump, the thread of interest had received its own thrown OutOfMemoryError exception which had already been caught and the thread was in the process of wrapping and re-throwing the exception all before the system dump agent could acquire exclusive access and produce the dump. Due to the unwinding of the stack of the suspect thread's OOM, the relevant stack frame locals that were the proximate cause of the OOM were already gone.

This was extra confusing because heap utilization was still very high because although the strong references were not in the dump, all the suspect objects still had paths through phantom references. This led to the discovery of issue #23866 under the assumption that there should be some strong reference path somewhere that was causing the OOM; however, that was not true, the strong references simply weren't in the dump. Sure enough, right after the dump, all the phantom references were cleared:

  <references type="phantom" candidates="1080807" cleared="1045740" enqueued="1045740" />

I remember a similar issue from many years ago, but this also makes me wonder how many other cases there have been where interesting stack frame locals are missing from the system dump due to this timing window.

Proposed solution

My first thought was that it would be ideal to take the system dump while the GC still has exclusive access; however, exploring the path down from openj9/runtime/gc_modron_startup/mgcalloc.cpp seems like it would be very complicated to work such code in and might even reach into OMR which would be difficult to bubble something up into the dump system. And then we'd want to dynamically create duplicate OOM dump agents that don't request exclusive access so there would just be a lot of complexity.

My next thought is to do the following to dramatically increase the likelihood of the system dump capturing the relevant thread locals:

1. Add a new handleOutOfMemoryError function to j9rasDumpFunctions which is called from mgcalloc.cpp right after the OOM is detected.
2. This function directly triggers OOM dump agents through an "early execution" mechanism.
3. The OOM is then thrown and will be handled again by the dump system but the agents will be skipped because they were "executed early".
4. The handleOutOfMemoryError method has a lock so that all threads throwing OOMs synchronize around the early dump creation.