Skip to content

JITServer: handle non-null compilation - #23846

Merged
mpirvu merged 1 commit into
eclipse-openj9:masterfrom
KavinSatheeskumar:jitserver-assert
Jun 20, 2026
Merged

JITServer: handle non-null compilation#23846
mpirvu merged 1 commit into
eclipse-openj9:masterfrom
KavinSatheeskumar:jitserver-assert

Conversation

@KavinSatheeskumar

@KavinSatheeskumar KavinSatheeskumar commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

When a JITServer stream failure is thrown, sometimes setting
the compilation object to NULL can be skiped to head straight
into exception handling. This causes an assert to be tripped.

In this particular case, there is a front-end call in
postCompilationTasks which can cause a stream error. Since it
isn't guarded by a global try-catch, it results in the given
assert being tripped.

Addresses this issue:
#23845

@mpirvu
mpirvu self-requested a review May 5, 2026 19:01
@mpirvu mpirvu self-assigned this May 5, 2026
@mpirvu mpirvu added the comp:jitserver Artifacts related to JIT-as-a-Service project label May 5, 2026
@github-project-automation github-project-automation Bot moved this to In progress in JIT as a Service May 5, 2026

@mpirvu mpirvu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't agree with the solution in this PR. The block of code that supposedly throws the StreamFailure exception you are trying to catch cannot be executed on the server. The block is protected by if (entry->isAotLoad() and AOT loads are only executed at the client. Moreover, the function that supposedly causes the StreamFailure has been redefined at JITServer:

    virtual uint16_t getAllEnabledHints(J9Method *method) override
    {
        TR_ASSERT_FATAL(false, "called");
        return 0;
    }

so we know it cannot be executed.
The high level pseudocode of the code is like this:

try {
  preCompilationTasks()
  compile()
  postCompilationTasks()
} catch {
   TR_ASSERT_FATAL(comp==NULL)
   postCompilationTasks()
}

It's possible that there is an exception thrown from postCompilationTasks(), but it's not caused by getAllEnabledHints. compilationEnd() is called from postCompilationTasks() and that's a big method. Maybe that one could throw?

@mpirvu

mpirvu commented May 13, 2026

Copy link
Copy Markdown
Contributor

compilationEnd() calls outOfProcessCompilationEnd() at the server and that function sends messages to the client: entry->_stream->finishCompilation(). It's quite possible that the StreamFailure exceptions come from here.

@mpirvu

mpirvu commented May 26, 2026

Copy link
Copy Markdown
Contributor

It's very likely that there is an exception being generated in postCompilationTasks(). The comments for that method say:

 * IMPORTANT: Because this method is called from both the try and catch blocks in
 * TR::CompilationInfoPerThreadBase::compile, an exception should never escape from this method.
 * Calls to this method could be guarded with a try/catch, but that defeats this
 * method's purpose - an escaped exception here would prevent necessary cleanup.

so we should not allow any exception to escape, but at the same time we cannot put a try/catch block around the entire method because we must execute all cleanup actions that postCompilationTasks() is supposed to do.
Looking at the code, most likely source of an exception is compilationEnd(). In there there are a few calls that I thought could throw exceptions:

  • stream->writeError()
  • stream->finishCompilation() coming from outOfProcessCompilationEnd()

However, the methods above already have try/catch blocks around them, so they are not the culprits.
The next step would be to identify precisely who throws an exception. We could proceed by elimination: we could place a try/catch block around compilationEnd() and other similar pieces of code and fire a fatal assert if an exception is detected.

@KavinSatheeskumar
KavinSatheeskumar force-pushed the jitserver-assert branch 2 times, most recently from a0abbf0 to d94b94d Compare June 11, 2026 18:03
Comment thread runtime/compiler/control/JITServerCompilationThread.cpp Outdated
Comment thread runtime/compiler/control/JITServerCompilationThread.cpp Outdated
Comment thread runtime/compiler/control/JITServerCompilationThread.cpp Outdated

@mpirvu mpirvu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mpirvu

mpirvu commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

jenkins test sanity plinuxjit,xlinuxjit,zlinuxjit,alinux64jit jdk21

@mpirvu

mpirvu commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

jenkins test sanity,extended plinuxjit,xlinuxjit,zlinuxjit,alinux64jit jdk21

@KavinSatheeskumar

KavinSatheeskumar commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

The following test suite had three errors
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_linux_jit_Personal/27/
jdk_security2_0

[2026-06-15T07:00:53.969Z] ACTION: main -- Error. Program `/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_x86-64_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 963249ms).

jdk_collections_0

[2026-06-15T07:37:19.872Z] ACTION: main -- Error. Program `/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_x86-64_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 961619ms).

jdk_stream_0

[2026-06-15T05:48:55.441Z] ACTION: testng -- Error. Program `/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_x86-64_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 968449ms).

@KavinSatheeskumar

Copy link
Copy Markdown
Contributor Author

The following test suite had one error
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_s390x_linux_jit_Personal/29/
jdk_lang_1

[2026-06-15T11:48:25.906Z] Unhandled exception
[2026-06-15T11:48:25.906Z] Type=Segmentation error vmState=0x00000000
[2026-06-15T11:48:25.906Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=000000a9 Signal_Code=00000001
[2026-06-15T11:48:25.906Z] Handler1=000003FF9FA54288 Handler2=000003FF9F933870 InaccessibleAddress=0000000400000000 Sending_Process=00000004
[2026-06-15T11:48:25.906Z] gpr0=0000000000000000 gpr1=000003FFA02CA900 gpr2=000003FF8D3AF500 gpr3=0000000000000008
[2026-06-15T11:48:25.906Z] gpr4=000003FFA02DCC20 gpr5=000003FFA02CA900 gpr6=000003FFA4673340 gpr7=000003FFA037E9E8
[2026-06-15T11:48:25.906Z] gpr8=0000000400000000 gpr9=000003FF254DE6C9 gpr10=00000000020E0001 gpr11=000003FF8D3AF500
[2026-06-15T11:48:25.906Z] gpr12=000003FF8D5FEFA8 gpr13=000003FF9FCAF990 gpr14=000003FF9FB43E34 gpr15=000003FFA4672D20
[2026-06-15T11:48:25.906Z] psw=000003FF9FB38A0A mask=0705100180000000 fpc=0008fe00 bea=000003FF9FB3F4E0
[2026-06-15T11:48:25.906Z] fpr0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2026-06-15T11:48:25.906Z] fpr1=3f80000000000000 (f: 0.000000, d: 7.812500e-03)
[2026-06-15T11:48:25.906Z] fpr2=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2026-06-15T11:48:25.906Z] fpr3=3f947ae140000000 (f: 1073741824.000000, d: 2.000000e-02)
[2026-06-15T11:48:25.906Z] fpr4=0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
[2026-06-15T11:48:25.906Z] fpr5=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2026-06-15T11:48:25.906Z] fpr6=4118db6400000000 (f: 0.000000, d: 4.072570e+05)
[2026-06-15T11:48:25.906Z] fpr7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2026-06-15T11:48:25.906Z] fpr8=000654496694d4d3 (f: 1721029888.000000, d: 8.801899e-309)
[2026-06-15T11:48:25.906Z] fpr9=000002aa13a6c990 (f: 329697664.000000, d: 1.447364e-311)
[2026-06-15T11:48:25.906Z] fpr10=000000003ed3da70 (f: 1054071424.000000, d: 5.207805e-315)
[2026-06-15T11:48:25.906Z] fpr11=0000000000000001 (f: 1.000000, d: 4.940656e-324)
[2026-06-15T11:48:25.906Z] fpr12=000002aa13a6c9a0 (f: 329697696.000000, d: 1.447364e-311)
[2026-06-15T11:48:25.906Z] fpr13=000003ff6401ca08 (f: 1677838848.000000, d: 2.171631e-311)
[2026-06-15T11:48:25.906Z] fpr14=0000000000d10178 (f: 13697400.000000, d: 6.767415e-317)
[2026-06-15T11:48:25.906Z] fpr15=000003ff64038c38 (f: 1677954048.000000, d: 2.171631e-311)
[2026-06-15T11:48:25.906Z] Module=/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9vm29.so
[2026-06-15T11:48:25.906Z] Module_base_address=000003FF9FA00000
[2026-06-15T11:48:25.906Z] Target=2_90_20260615_78 (Linux 3.10.0-1160.119.1.el7.s390x)
[2026-06-15T11:48:25.906Z] CPU=s390x (4 logical CPUs) (0x1ec1b1000 RAM)
[2026-06-15T11:48:25.906Z] ----------- Stack Backtrace -----------
[2026-06-15T11:48:25.906Z] _ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread+0x1afda (0x000003FF9FB38A0A [libj9vm29.so+0x138a0a])
[2026-06-15T11:48:25.906Z] bytecodeLoopFull+0xee (0x000003FF9FB1D97E [libj9vm29.so+0x11d97e])
[2026-06-15T11:48:25.906Z] c_cInterpreter+0x64 (0x000003FF9FBD861C [libj9vm29.so+0x1d861c])
[2026-06-15T11:48:25.906Z] ---------------------------------------
[2026-06-15T11:48:25.906Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2026/06/15 07:48:10 - please wait.
[2026-06-15T11:48:25.906Z] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/output_1781523485321/jdk_lang_1/work/scratch/2/core.20260615.074810.44775.0001.dmp' in response to an event
[2026-06-15T11:48:25.906Z] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/output_1781523485321/jdk_lang_1/work/scratch/2/core.20260615.074810.44775.0001.dmp
[2026-06-15T11:48:25.906Z] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/output_1781523485321/jdk_lang_1/work/scratch/2/javacore.20260615.074810.44775.0002.txt' in response to an event
[2026-06-15T11:48:25.906Z] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/output_1781523485321/jdk_lang_1/work/scratch/2/javacore.20260615.074810.44775.0002.txt
[2026-06-15T11:48:25.906Z] JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/output_1781523485321/jdk_lang_1/work/scratch/2/Snap.20260615.074810.44775.0003.trc' in response to an event
[2026-06-15T11:48:25.906Z] JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/output_1781523485321/jdk_lang_1/work/scratch/2/Snap.20260615.074810.44775.0003.trc
[2026-06-15T11:48:25.907Z] JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_s390x_linux_jit_Personal_testList_1/aqa-tests/TKG/output_1781523485321/jdk_lang_1/work/scratch/2/jitdump.20260615.074810.44775.0004.dmp' in response to an event
[2026-06-15T11:48:25.907Z] JVMDUMP051I JIT dump occurred in 'main' thread 0x000003FFA00C3700
[2026-06-15T11:48:25.907Z] JVMDUMP056I Processed dump event "gpf", detail "" at 2026/06/15 07:48:18 (7.468 seconds).

@KavinSatheeskumar

Copy link
Copy Markdown
Contributor Author

The following test suite had 3 errors
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_aarch64_linux_jit_Personal/24/
jdk_collections_0

[2026-06-15T07:58:35.459Z] TEST RESULT: Error. Program `/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_aarch64_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java' timed out (timeout set to 1440000ms, elapsed time including timeout handling was 1440619ms).

jdk_stream_0

[2026-06-15T08:24:28.132Z] config java.util.stream.LoggingTestCase.before(): success [0ms]
[2026-06-15T08:24:28.132Z] test org.openjdk.tests.java.util.stream.WhileOpStatefulTest.testCountDropWithCount(): failure [29762ms]
[2026-06-15T08:24:28.132Z] java.lang.RuntimeException
[2026-06-15T08:24:28.132Z] 	at org.openjdk.tests.java.util.stream.WhileOpStatefulTest.lambda$testWhileMulti$62(WhileOpStatefulTest.java:259)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.ReferencePipeline$15$1.accept(ReferencePipeline.java:540)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.LinkedList$LLSpliterator.tryAdvance(LinkedList.java:1263)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.WhileOps$UnorderedWhileSpliterator$OfRef$Dropping.tryAdvance(WhileOps.java:795)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:332)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:522)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:512)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:239)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.stream.LongPipeline.reduce(LongPipeline.java:498)
[2026-06-15T08:24:28.132Z] 	at org.openjdk.tests.java.util.stream.WhileOpStatefulTest.lambda$testCountDropWithCount$34(WhileOpStatefulTest.java:155)
[2026-06-15T08:24:28.132Z] 	at org.openjdk.tests.java.util.stream.WhileOpStatefulTest.lambda$testWhileMulti$72(WhileOpStatefulTest.java:279)
[2026-06-15T08:24:28.132Z] 	at org.openjdk.tests.java.util.stream.WhileOpStatefulTest.testWhileMulti(WhileOpStatefulTest.java:295)
[2026-06-15T08:24:28.132Z] 	at org.openjdk.tests.java.util.stream.WhileOpStatefulTest.testDropWhileMulti(WhileOpStatefulTest.java:245)
[2026-06-15T08:24:28.132Z] 	at org.openjdk.tests.java.util.stream.WhileOpStatefulTest.testCountDropWithCount(WhileOpStatefulTest.java:151)
[2026-06-15T08:24:28.132Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2026-06-15T08:24:28.132Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:586)
[2026-06-15T08:24:28.132Z] 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
[2026-06-15T08:24:28.132Z] 	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
[2026-06-15T08:24:28.132Z] 	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
[2026-06-15T08:24:28.132Z] 	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
[2026-06-15T08:24:28.132Z] 	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
[2026-06-15T08:24:28.132Z] 	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
[2026-06-15T08:24:28.132Z] 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
[2026-06-15T08:24:28.132Z] 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
[2026-06-15T08:24:28.132Z] 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
[2026-06-15T08:24:28.132Z] 	at org.testng.TestRunner.privateRun(TestRunner.java:764)
[2026-06-15T08:24:28.132Z] 	at org.testng.TestRunner.run(TestRunner.java:585)
[2026-06-15T08:24:28.132Z] 	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
[2026-06-15T08:24:28.132Z] 	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
[2026-06-15T08:24:28.133Z] 	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
[2026-06-15T08:24:28.133Z] 	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
[2026-06-15T08:24:28.133Z] 	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
[2026-06-15T08:24:28.133Z] 	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
[2026-06-15T08:24:28.133Z] 	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
[2026-06-15T08:24:28.133Z] 	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
[2026-06-15T08:24:28.133Z] 	at org.testng.TestNG.runSuites(TestNG.java:1069)
[2026-06-15T08:24:28.133Z] 	at org.testng.TestNG.run(TestNG.java:1037)
[2026-06-15T08:24:28.133Z] 	at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:107)
[2026-06-15T08:24:28.133Z] 	at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:63)
[2026-06-15T08:24:28.133Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2026-06-15T08:24:28.133Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:586)
[2026-06-15T08:24:28.133Z] 	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
[2026-06-15T08:24:28.133Z] 	at java.base/java.lang.Thread.run(Thread.java:1600)

jdk_security2_0

[2026-06-15T04:02:52.279Z] ACTION: main -- Error. Program `/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_aarch64_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/bin/java' timed out (timeout set to 1440000ms, elapsed time including timeout handling was 1440894ms).

@KavinSatheeskumar

Copy link
Copy Markdown
Contributor Author

It appears that the build failures in this test suite all have the same common cause.
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_aarch64_linux_jit_Personal/39/

[2026-06-15T04:49:39.940Z] TESTING:
[2026-06-15T14:39:01.893Z] Cancelling nested steps due to timeout

@mpirvu

mpirvu commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

At least one of the aborted tests, extended.functional on xlinux, had multiple tests that experienced a timeout. The timeouts generate coredumps which fill the disk space.

02:05:35  ***[TEST INFO 2026/06/15 03:05:35] ProcessKiller detected a timeout after 10000 milliseconds!***
..............
02:05:36  GDB OUT Thread 6 (Thread 0x7fd55af96700 (LWP 29502)):
02:05:36  GDB OUT #0  0x00007fd8293ff75d in read () from /lib64/libpthread.so.0
02:05:36  GDB OUT No symbol table info available.
02:05:36  GDB OUT #1  0x00007fd811c2a921 in JITServer::CommunicationStream::readMessage(JITServer::Message&) () from /home/jenkins/workspace/Test_openjdk21_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
02:05:36  GDB OUT No symbol table info available.
02:05:36  GDB OUT #2  0x00007fd81186ba5b in handleServerMessage(JITServer::ClientStream*, TR_J9VM*, JITServer::MessageType&) () from /home/jenkins/workspace/Test_openjdk21_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
..............
02:12:23   [ERR] JVMDUMP039I Processing dump event "abort", detail "" at 2026/06/15 03:12:16 - please wait.
02:12:23   [ERR] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk21_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_17814979031868/testSCCMLTests5_3/core.20260615.031216.29496.0001.dmp' in response to an event
02:12:23   [ERR] JVMDUMP030W Cannot write dump to file /home/jenkins/workspace/Test_openjdk21_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_17814979031868/testSCCMLTests5_3/core.20260615.031216.29496.0001.dmp: No space left on device
02:12:23   [ERR] JVMDUMP030W Cannot write dump to file /home/jenkins/workspace/Test_openjdk21_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_17814979031868/testSCCMLTests5_3/javacore.20260615.031216.29496.0002.txt: No space left on device
02:12:23   [ERR] JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk21_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_17814979031868/testSCCMLTests5_3/Snap.20260615.031216.29496.0003.trc' in response to an event
02:12:23   [ERR] JVMDUMP030W Cannot write dump to file /home/jenkins/workspace/Test_openjdk21_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_17814979031868/testSCCMLTests5_3/Snap.20260615.031216.29496.0003.trc: No space left on device
02:12:23   [ERR] JVMDUMP030W Cannot write dump to file /home/jenkins/workspace/Test_openjdk21_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_17814979031868/testSCCMtee: write error

@mpirvu

mpirvu commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

jenkins test sanity.functional xlinux,xlinuxjit jdk21

@KavinSatheeskumar

Copy link
Copy Markdown
Contributor Author

Related, this Grinder I ran to test the issue has completed 75 out of 100 iterations of the original test case without any errors being thrown
https://openj9-jenkins.osuosl.org/job/Grinder/5227/
In most of our testing, this would have been enough to trip the error. When it is complete, this is evidence that this fix has addressed the original issue

@KavinSatheeskumar
KavinSatheeskumar force-pushed the jitserver-assert branch 3 times, most recently from 0aa2bad to 2cc5681 Compare June 17, 2026 14:11
@mpirvu
mpirvu removed the request for review from dsouzai June 17, 2026 18:50
@KavinSatheeskumar

Copy link
Copy Markdown
Contributor Author

Grinder with 100 iterations:
https://openj9-jenkins.osuosl.org/job/Grinder/5234/

Comment thread runtime/compiler/control/CompilationThread.cpp Outdated
@KavinSatheeskumar
KavinSatheeskumar force-pushed the jitserver-assert branch 3 times, most recently from 9cf62db to 23c9130 Compare June 17, 2026 20:51
@KavinSatheeskumar

Copy link
Copy Markdown
Contributor Author

New Grinder with the updated implementation
https://openj9-jenkins.osuosl.org/job/Grinder/5235/

TRIGGER_J9HOOK_JIT_COMPILING_END(_jitConfig->hookInterface, vmThread, method);

if (_compiler && _methodBeingCompiled->isOutOfProcessCompReq()) {
// If JITServer is enabled, this will send messages

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested re-phrasing:

            // In JITServer mode preemtively call _compiler->retainedMethods() to
            // create the retainedMethod set if it doesn't exist. If this operation
            // is not done now, it will be done later during:
            // postCompilationTasks
            //   - compilationEnd
            //     - outOfProcessCompilationEnd
            //       - computeDataForCHTableCommit
            //         - getDataForClient
            // The server will send a message to the client, which could cause an
			// exception and exceptions are not allowed to be triggerred from postCompilationTasks()

bool useServerOffsets = aotCacheStore && clientData->useServerOffsets(entry->_stream);
bool useServerOffsets = aotCacheStore;
try {
// in the case useServerOffsets triggers an message

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If using full sentences, the sentence must start with a capital letter.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested re-phrasing:
Guard useServerOffsets() with a try block to prevent exceptions from
interrupting the flow of postCompilationTasks() at a higher level.

Comment thread runtime/compiler/control/JITServerCompilationThread.cpp Outdated
When a JITServer stream failure is thrown, sometimes setting
the compilation object to NULL can be skipped to head straight
into exception handling. This causes an assert to be tripped.

In this particular case, there is a front-end call in
outOfProcessCompilationEnd (within computeDataForCHTableCommit)
which can cause a stream error. Since it isn't guarded
by a global try-catch, it results in the given
assert being tripped.

Addresses this issue:
eclipse-openj9#23845
@mpirvu

mpirvu commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

jenkins test sanity plinuxjit,xlinuxjit,zlinuxjit,alinux64jit jdk25

@mpirvu

mpirvu commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

xlinux had one failure: for testJITServer_0

16:19:22  Failed to properly start client, it terminated prematurely with exit value: 1
16:19:22  Dumping the contents of log file: /home/jenkins/workspace/Test_openjdk25_j9_sanity.functional_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_17819000753873/testJITServer_0/testServer.client.out
16:19:22  ////////////////////////////////////////////////////////////////////////
16:19:22  //// [IncludeExcludeTestAnnotationTransformer] [INFO] EXCLUDE_FILE environment variable: /home/jenkins/workspace/Test_openjdk25_j9_sanity.functional_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/../TestConfig/resources/excludes/latest_exclude_25.txt,/home/jenkins/workspace/Test_openjdk25_j9_sanity.functional_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/../TestConfig/resources/excludes/feature_ojdkmh_exclude.txt
16:19:22  //// [IncludeExcludeTestAnnotationTransformer] [INFO] Processing exclude file: /home/jenkins/workspace/Test_openjdk25_j9_sanity.functional_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/../TestConfig/resources/excludes/latest_exclude_25.txt
16:19:22  //// [IncludeExcludeTestAnnotationTransformer] [INFO] Processing exclude file: /home/jenkins/workspace/Test_openjdk25_j9_sanity.functional_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/../TestConfig/resources/excludes/feature_ojdkmh_exclude.txt
16:19:22  //// ...
16:19:22  //// ... TestNG 6.14.2 by Cédric Beust (cedric@beust.com)
16:19:22  //// ...
16:19:22  //// 
16:19:22  //// Exception in thread "main" java.lang.NoClassDefFoundError: org.testng.internal.Invoker$SameClassNamePredicate
16:19:22  //// 	at org.testng.internal.Invoker.<clinit>(Invoker.java:82)
16:19:22  //// 	at org.testng.TestRunner.init(TestRunner.java:231)
16:19:22  //// 	at org.testng.TestRunner.<init>(TestRunner.java:167)
16:19:22  //// 	at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:663)
16:19:22  //// 	at org.testng.SuiteRunner.init(SuiteRunner.java:260)
16:19:22  //// 	at org.testng.SuiteRunner.<init>(SuiteRunner.java:198)
16:19:22  //// 	at org.testng.TestNG.createSuiteRunner(TestNG.java:1295)
16:19:22  //// 	at org.testng.TestNG.createSuiteRunners(TestNG.java:1273)
16:19:22  //// 	at org.testng.TestNG.runSuitesLocally(TestNG.java:1128)
16:19:22  //// 	at org.testng.TestNG.runSuites(TestNG.java:1049)
16:19:22  //// 	at org.testng.TestNG.run(TestNG.java:1017)
16:19:22  //// 	at org.testng.TestNG.privateMain(TestNG.java:1354)
16:19:22  //// 	at org.testng.TestNG.main(TestNG.java:1323)
16:19:22  //// Caused by: java.lang.ClassNotFoundException: org.testng.internal.Invoker$SameClassNamePredicate
16:19:22  //// 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:754)
16:19:22  //// 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:1055)
16:19:22  //// 	... 13 more
16:19:22  ////////////////////////////////////////////////////////////////////////

10x grinder passed: https://openj9-jenkins.osuosl.org/job/Grinder/5237/

@mpirvu

mpirvu commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Since all tests passed (the xlinux one was infra related), this PR is ready to be merged.

@mpirvu
mpirvu merged commit 752e3bf into eclipse-openj9:master Jun 20, 2026
14 of 16 checks passed
@github-project-automation github-project-automation Bot moved this from In progress to Done in JIT as a Service Jun 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:jitserver Artifacts related to JIT-as-a-Service project

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants