GlassFish sometimes doesn't stop because of a non-daemon thread started from fighterfish

[tsyamamoto]

loadgc.zip

GlassFish sometimes doesn't stop if Full GC doesn't occur.

E:\tmp\glassfish5\glassfish\bin>asadmin stop-local-instance --force=false instance
Waiting for the instance to stop .....................
Timed out (60 seconds) waiting for the domain to stop.
Command stop-local-instance failed.

Environment Details

• GlassFish Version (and build number): 4.1, 5.1
• JDK version: 8
• OS: Windows, Linux
• Database: none

---

Problem Description

I investigated the reason.
A thread started by fighterfish isn't a daemon thread,
and ThreadPoolExecutor.shutdown isn't called.

Therefore, a non-daemon thread sometimes leaves
when I'm trying to stop process gracefully.

Steps to reproduce

E:\tmp\glassfish5\glassfish\bin>asadmin start-domain
Waiting for domain1 to start .............
Successfully started the domain : domain1
domain  Location: E:\tmp\glassfish5\glassfish\domains\domain1
Log File: E:\tmp\glassfish5\glassfish\domains\domain1\logs\server.log
Admin Port: 4848
Command start-domain executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin create-cluster cluster
Command create-cluster executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin create-local-instance --cluster cluster instance
Rendezvoused with DAS on localhost:4848.
Port Assignments for server instance instance:
OSGI_SHELL_TELNET_PORT=26666
JAVA_DEBUGGER_PORT=29009
JMS_PROVIDER_PORT=27676
HTTP_LISTENER_PORT=28080
IIOP_SSL_LISTENER_PORT=23820
ASADMIN_LISTENER_PORT=24848
IIOP_SSL_MUTUALAUTH_PORT=23920
JMX_SYSTEM_CONNECTOR_PORT=28686
HTTP_SSL_LISTENER_PORT=28181
IIOP_LISTENER_PORT=23700
Command create-local-instance executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin create-jvm-options --target cluster -Xmx2048m
Created 1 option(s)
The configuration already has maximum heap size specified: -Xmx512m. Verify the java configuration by doing list-jvm-options.
Command create-jvm-options executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin create-jvm-options --target cluster -Xms2048m
Created 1 option(s)
It appears that the initial heap size specified: 2048 MB is larger than the maximum heap size in the configuration: 512 MB. JVM might not start. Ensure that this is valid, by doing doing list-jvm-options.
Command create-jvm-options executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin create-jvm-options --target cluster -XX\:MaxMetaspaceSize=512m
Created 1 option(s)
Command create-jvm-options executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin create-jvm-options --target cluster -XX\:MetaspaceSize=512m
Created 1 option(s)
Command create-jvm-options executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin create-jvm-options --target cluster -XX\:NewSize=512m
Created 1 option(s)
Command create-jvm-options executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin create-jvm-options --target cluster -XX\:MaxNewSize=512m
Created 1 option(s)
Command create-jvm-options executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin delete-jvm-options --target cluster -Xmx512m
Deleted 1 option(s)
Command delete-jvm-options executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin delete-jvm-options --target cluster -XX\:MaxPermSize=192m
Deleted 1 option(s)
Command delete-jvm-options executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin set cluster.transaction-service.automatic-recovery=false
cluster.transaction-service.automatic-recovery=false
Command set executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin deploy --target cluster c:\work\loadgc.war
Application deployed with name loadgc.
Command deploy executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin set cluster.gms-enabled=false
cluster.gms-enabled=false
Command set executed successfully.

(TO avoid strange ssl failure)
E:\tmp\glassfish5\glassfish\bin>asadmin set cluster.admin-service.jmx-connector.system.security-enabled=false
cluster.admin-service.jmx-connector.system.security-enabled=false
Command set executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin set cluster.network-config.network-listeners.network-listener.admin-listener.protocol=admin-listener
cluster.network-config.network-listeners.network-listener.admin-listener.protocol=admin-listener
Command set executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin start-local-instance instance
Waiting for instance to start ...................................................
Successfully started the instance: instance
instance Location: E:\tmp\glassfish5\glassfish\nodes\localhost-domain1\instance
Log File: E:\tmp\glassfish5\glassfish\nodes\localhost-domain1\instance\logs\server.log
Admin Port: 24848
Command start-local-instance executed successfully.

E:\tmp\glassfish5\glassfish\bin>asadmin stop-local-instance --force=false instance
Waiting for the instance to stop .....................
Timed out (60 seconds) waiting for the domain to stop.
Command stop-local-instance failed.

Target source code

glassfish-fighterfish\module\osgi-javaee-base\src\main\java\org\glassfish\osgijavaeebase\ExtenderManager.java

A thread generated by *1 doesn't stop unless a finalization occurs.

    private class GlassFishServerTracker extends ServiceTracker {

        /**
         * Create a new instance.
         * @param ctx the bundle context
         */
        @SuppressWarnings("unchecked")
        GlassFishServerTracker(final BundleContext ctx) {
            super(ctx, GlassFish.class.getName(), null);
        }

        @Override
        @SuppressWarnings("unchecked")
        public Object addingService(final ServiceReference reference) {
            LOGGER.logp(Level.FINE, "ExtenderManager$GlassFishServerTracker",
                    "addingService", "GlassFish has been created");
            final GlassFish gf = GlassFish.class.cast(context
                    .getService(reference));
            ExecutorService executorService = Executors
                    .newSingleThreadExecutor(); *1 
            return executorService.submit(new Runnable() {
                @Override
                public void run() {
                    try {
                        // Poll for GlassFish to start. GlassFish service might
                        // have been registered by
                        // GlassFishRuntime.newGlassFish() and hence might not
                        // be ready to use.
                        // This is the case for GlassFish < 4.0
                        GlassFish.Status status;
                        while ((status = gf.getStatus())
                                != GlassFish.Status.STARTED) {
                            if (status == GlassFish.Status.DISPOSED) {
                                return;
                            }
                            try {
                                Thread.sleep(1000);
                            } catch (InterruptedException e) {
                                return;
                            }
                        }
                        // start extenders first before registering for events,
                        // otherwise we can deadlock
                        // if startExtender() is in progress and glassfish sends
                        // PREPARE_SHUTDOWN event.
                        startExtenders();
                        events = gf.getService(Events.class);
                        listener = new EventListener() {
                            @Override
                            public void event(final Event event) {
                                if (EventTypes.PREPARE_SHUTDOWN
                                        .equals(event.type())) {
                                    stopExtenders();
                                }
                            }
                        };
                        events.register(listener);
                    } catch (GlassFishException e) {
                        // TODO(Sahoo): Proper Exception Handling
                        throw new RuntimeException(e);
                    }
                }
            });
        }