JITServer: Compilation must be deallocated assertion error

[KavinSatheeskumar]

Failure link

https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/59817/

• test category, in Grinder equates to BUILD_LIST: openjdk
• test target name, in Grinder equates to TARGET: jdk_util_other_0
• OS/architecture, in Grinder equates to Jenkinsfile: s390_linux
• public build SHAs (i.e. java -version output), in Grinder provides JDK_VERSION/JDK_IMPL info and helps identify point at which regression introduced:
  OpenJ9 - e190a19
  OMR - 416a2d2
  JCL - 7df109a based on jdk-21.0.11+9

Optional info

• intermittent failure (yes|no): yes
• regression or new test: regression
• if regression, what are the last passing / first failing public SHAs (OpenJ9, OMR, JCL) : Unknown

Failure output (captured from console output)

[2026-04-28T01:07:20.640Z] VMState: 0x0005ff0b
[2026-04-28T01:07:20.640Z] 	Compilation must be deallocated and NULL by this point
[2026-04-28T01:07:20.640Z] compiling jdk/internal/org/objectweb/asm/ClassWriter.toByteArray()[B at level: cold
[2026-04-28T01:07:20.640Z] #0: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xdfe6c2) [0x3ffab8fe6c2]
[2026-04-28T01:07:20.640Z] #1: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xe0f298) [0x3ffab90f298]
[2026-04-28T01:07:20.640Z] #2: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c0e20) [0x3ffab3c0e20]
[2026-04-28T01:07:20.640Z] #3: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8c2056) [0x3ffab3c2056]
[2026-04-28T01:07:20.640Z] #4: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1bc946) [0x3ffaacbc946]
[2026-04-28T01:07:20.640Z] #5: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x268cda) [0x3ffaad68cda]
[2026-04-28T01:07:20.640Z] #6: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1bb55e) [0x3ffaacbb55e]
[2026-04-28T01:07:20.640Z] #7: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1bb9be) [0x3ffaacbb9be]
[2026-04-28T01:07:20.640Z] #8: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1bbaa4) [0x3ffaacbbaa4]
[2026-04-28T01:07:20.640Z] #9: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x349c0) [0x3ffb65349c0]
[2026-04-28T01:07:20.640Z] #10: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1bbec6) [0x3ffaacbbec6]
[2026-04-28T01:07:20.640Z] #11: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x9144) [0x3ffb6489144]
[2026-04-28T01:07:20.640Z] #12: /lib64/libc.so.6(+0x96fa6) [0x3ffb6e96fa6]
[2026-04-28T01:07:20.640Z] #13: /lib64/libc.so.6(+0x1102fe) [0x3ffb6f102fe]
[2026-04-28T01:07:20.640Z] 
[2026-04-28T01:07:20.640Z] Unhandled exception
[2026-04-28T01:07:20.640Z] Type=Unhandled trap vmState=0x0005ff0b
[2026-04-28T01:07:20.640Z] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
[2026-04-28T01:07:20.640Z] Handler1=000003FFB6653158 Handler2=000003FFB6533728
[2026-04-28T01:07:20.640Z] gpr0=0000000000000010 gpr1=000003FFFFFFFFE5 gpr2=0000000000000000 gpr3=00000000002DCD17
[2026-04-28T01:07:20.640Z] gpr4=0000000000000005 gpr5=0000000000000010 gpr6=000003FF98BF9F50 gpr7=0000000000000005
[2026-04-28T01:07:20.640Z] gpr8=000003FF00000402 gpr9=000003FF98BFA108 gpr10=000003FF98BFC840 gpr11=00000000002DCD17
[2026-04-28T01:07:20.640Z] gpr12=000003FFABD8BAC0 gpr13=000003FF98BFC840 gpr14=000003FFB6E98E1C gpr15=000003FF98BF9C48
[2026-04-28T01:07:20.640Z] psw=000003FFB6E98E26 mask=0705100180000000 fpc=00080000 bea=000003FFB6F622BC
[2026-04-28T01:07:20.640Z] fpr0=0000000000000003 (f: 3.000000, d: 1.482197e-323)
[2026-04-28T01:07:20.640Z] fpr1=000003ff98bf7608 (f: 2562684416.000000, d: 2.172068e-311)
[2026-04-28T01:07:20.640Z] fpr2=000003ffb6fccd08 (f: 3070020864.000000, d: 2.172318e-311)
[2026-04-28T01:07:20.640Z] fpr3=000003ff7c000c18 (f: 2080377856.000000, d: 2.171830e-311)
[2026-04-28T01:07:20.640Z] fpr4=000003ff98bf7710 (f: 2562684672.000000, d: 2.172068e-311)
[2026-04-28T01:07:20.640Z] fpr5=85440c44cedae307 (f: 3470451456.000000, d: -2.696395e-283)
[2026-04-28T01:07:20.640Z] fpr6=0000000000000064 (f: 100.000000, d: 4.940656e-322)
[2026-04-28T01:07:20.640Z] fpr7=3f80000000000000 (f: 0.000000, d: 7.812500e-03)
[2026-04-28T01:07:20.640Z] fpr8=000003ffaa10e978 (f: 2853234944.000000, d: 2.172211e-311)
[2026-04-28T01:07:20.640Z] fpr9=0000000000101000 (f: 1052672.000000, d: 5.200891e-318)
[2026-04-28T01:07:20.640Z] fpr10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2026-04-28T01:07:20.640Z] fpr11=000003ffe21f83ba (f: 3793716224.000000, d: 2.172676e-311)
[2026-04-28T01:07:20.640Z] fpr12=000003ff98bfc900 (f: 2562705664.000000, d: 2.172068e-311)
[2026-04-28T01:07:20.640Z] fpr13=000003ff44013cc8 (f: 1140931840.000000, d: 2.171365e-311)
[2026-04-28T01:07:20.640Z] fpr14=000003ff98afc000 (f: 2561654784.000000, d: 2.172067e-311)
[2026-04-28T01:07:20.640Z] fpr15=000003ff4400d038 (f: 1140903936.000000, d: 2.171365e-311)
[2026-04-28T01:07:20.640Z] Module=/lib64/libc.so.6
[2026-04-28T01:07:20.640Z] Module_base_address=000003FFB6E00000
[2026-04-28T01:07:20.640Z] 
[2026-04-28T01:07:20.640Z] Unhandled exception in signal handler. Protected function: writeCrashDataToConsole (0x0)
[2026-04-28T01:07:20.640Z] 
[2026-04-28T01:07:20.640Z] ----------- Stack Backtrace -----------
[2026-04-28T01:07:20.640Z] __pthread_kill_implementation+0xe6 (0x000003FFB6E98E26 [libc.so.6+0x98e26])
[2026-04-28T01:07:20.640Z] gsignal+0x20 (0x000003FFB6E49240 [libc.so.6+0x49240])
[2026-04-28T01:07:20.640Z] _ZN2TR4trapEv+0x52 (0x000003FFAB3C2002 [libj9jit29.so+0x8c2002])
[2026-04-28T01:07:20.640Z]  (0x000003FFAB3C205C [libj9jit29.so+0x8c205c])
[2026-04-28T01:07:20.640Z] _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x85e (0x000003FFAACBC946 [libj9jit29.so+0x1bc946])
[2026-04-28T01:07:20.640Z] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0xaca (0x000003FFAAD68CDA [libj9jit29.so+0x268cda])
[2026-04-28T01:07:20.640Z] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3ee (0x000003FFAACBB55E [libj9jit29.so+0x1bb55e])
[2026-04-28T01:07:20.640Z] _ZN2TR24CompilationInfoPerThread3runEv+0x5e (0x000003FFAACBB9BE [libj9jit29.so+0x1bb9be])
[2026-04-28T01:07:20.640Z] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FFAACBBAA4 [libj9jit29.so+0x1bbaa4])
[2026-04-28T01:07:20.640Z] omrsig_protect+0x268 (0x000003FFB65349C0 [libj9prt29.so+0x349c0])
[2026-04-28T01:07:20.640Z] _Z21compilationThreadProcPv+0x196 (0x000003FFAACBBEC6 [libj9jit29.so+0x1bbec6])
[2026-04-28T01:07:20.640Z] thread_wrapper+0x104 (0x000003FFB6489144 [libj9thr29.so+0x9144])
[2026-04-28T01:07:20.640Z] start_thread+0x146 (0x000003FFB6E96FA6 [libc.so.6+0x96fa6])
[2026-04-28T01:07:20.640Z]  (0x000003FFB6F102FE [libc.so.6+0x1102fe])
[2026-04-28T01:07:20.640Z] ---------------------------------------
[2026-04-28T01:07:20.640Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2026/04/27 18:07:09 - please wait.
[2026-04-28T01:07:20.640Z] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder/aqa-tests/TKG/core.20260427.180709.3001606.0001.dmp' in response to an event
[2026-04-28T01:07:20.640Z] JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder/aqa-tests/TKG/core.20260427.180709.3001606.0001.dmp
[2026-04-28T01:07:20.640Z] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Grinder/aqa-tests/TKG/javacore.20260427.180709.3001606.0002.txt' in response to an event
[2026-04-28T01:07:20.640Z] JVMDUMP012E Error in Java dump: /home/jenkins/workspace/Grinder/aqa-tests/TKG/javacore.20260427.180709.3001606.0002.txt
[2026-04-28T01:07:20.640Z] JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Grinder/aqa-tests/TKG/Snap.20260427.180709.3001606.0003.trc' in response to an event
[2026-04-28T01:07:20.640Z] JVMDUMP010I Snap dump written to /home/jenkins/workspace/Grinder/aqa-tests/TKG/Snap.20260427.180709.3001606.0003.trc
[2026-04-28T01:07:20.640Z] JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Grinder/aqa-tests/TKG/jitdump.20260427.180709.3001606.0004.dmp' in response to an event
[2026-04-28T01:07:20.640Z] JVMDUMP051I JIT dump occurred in 'JIT Compilation Thread-004' thread 0x0000000002999B00
[2026-04-28T01:07:20.640Z] JVMDUMP049I JIT dump notified all waiting threads of the current method to be compiled
[2026-04-28T01:07:20.640Z] JVMDUMP010I JIT dump written to /home/jenkins/workspace/Grinder/aqa-tests/TKG/jitdump.20260427.180709.3001606.0004.dmp
[2026-04-28T01:07:20.640Z] JVMDUMP056I Processed dump event "gpf", detail "" at 2026/04/27 18:07:14 (5.414 seconds).
[2026-04-28T01:15:24.083Z] Test results: passed: 556; excluded: 9; did not match keywords: 4; did not meet platform requirements: 2
[2026-04-28T01:15:45.194Z] Report written to /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17773378084853/jdk_util_other_0/report/html/report.html
[2026-04-28T01:15:45.194Z] Results written to /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17773378084853/jdk_util_other_0/work

[KavinSatheeskumar]

The assertion which tripped was added as apart of this pull request

https://github.com/eclipse-openj9/openj9/pull/11823

There was discussion that some error handling was omitted, so that it could be handled in another PR. I don't think that PR ever happened, leading to this issue

[KavinSatheeskumar]

After running some more Grinders, it seems like the error case discussed in the original PR is not the root cause of this issue. For some reason, the stream failures are coming from postCompilationTasks. There must be some change somewhere causing a method used in postCompilationTasks to use a JITServer method.

[KavinSatheeskumar]

I cannot 100% confirm weather this is the underlying cause, but this is the most likely candidate.

Upon testing with various Grinders, I was able to bound the source of the error to be between lines 7466 and 7552. Reading the code manually led me to this line.

openj9/runtime/compiler/control/CompilationThread.cpp

Line 7537 in 3fbeeff

= _vm->sharedCache()->getAllEnabledHints(method) & (TR_HintUpgrade | TR_HintHot | TR_HintScorching);

The getAllEnabledHints method eventually results in a call to front end query, which is overloaded on the JITServer. However, we are on the client, so I don't think that this would send a message. Not entirely sure why there is any communication going on at this stage.

I mistakenly thought the error was on the client and not the server. Now that we know it's on the server, this failure mode makes more sense. A grinder is being run to confirm this is the source of the error.

[KavinSatheeskumar]

Using the fix described in the PR, another Grinder was ran and no issues arose. This leads me to believe the explanation here was correct and is the only issue.
https://openj9-jenkins.osuosl.org/job/Grinder/5167/