Gradle daemon becomes unusable - resurrected

gradle-3567

(xtracoder) #1

I’m resurrecting that ancient thread

I’ve got mentioned problem reproducible in both 2.14.1 and 3.1. The root cause, as far as I understand, is unhanded exception in the socked connection in the target daemon. That connection is terminated abruptly and client retries with new daemon until number of gradle daemons reaches max limit of 100.

Here is stack trace I see in daemon’s log (the root cause - class, which is not available in daemon’s classloader, is present in the uploaded serialized data)

17:32:20.252 [WARN] [org.gradle.launcher.daemon.server.DefaultIncomingConnectionHandler] Unable to receive command from client socket connection from /127.0.0.1:29478 to /127.0.0.1:29479. Discarding connection.
org.gradle.internal.remote.internal.MessageIOException: Could not read message from '/127.0.0.1:29479'.
    at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:85)
    at org.gradle.launcher.daemon.server.SynchronizedDispatchConnection.receive(SynchronizedDispatchConnection.java:68)
    at org.gradle.launcher.daemon.server.DefaultDaemonConnection$1.run(DefaultDaemonConnection.java:63)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
    at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: <<-- UNSUPPORTED CLASS NAME HERE, subclass of java.io.File  -->>
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:628)
    at org.gradle.internal.io.ClassLoaderObjectInputStream.resolveClass(ClassLoaderObjectInputStream.java:40)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at org.gradle.internal.serialize.DefaultSerializer.read(DefaultSerializer.java:45)
    at org.gradle.internal.serialize.DefaultSerializerRegistry$TaggedTypeSerializer.read(DefaultSerializerRegistry.java:110)
    at org.gradle.internal.serialize.Serializers$StatefulSerializerAdapter$1.read(Serializers.java:36)
    at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:78)
    ... 7 more
17:32:20.253 [DEBUG] [org.gradle.launcher.daemon.server.SynchronizedDispatchConnection] thread 13: stopping connection
17:32:20.255 [DEBUG] [org.gradle.launcher.daemon.server.SynchronizedDispatchConnection] thread 13: stopping connection

Gradle daemon filling memory
(xtracoder) #2

I’ve found code on client which spawn 100 daemons

gradle-3.1\src\launcher\org\gradle\launcher\daemon\client\DaemonClient.java

public Object execute(BuildAction action, BuildRequestContext requestContext, BuildActionParameters parameters, ServiceRegistry contextServices) {
    Object buildId = idGenerator.generateId();
    List<DaemonInitialConnectException> accumulatedExceptions = Lists.newArrayList();

    int saneNumberOfAttempts = 100; //is it sane enough?

    for (int i = 1; i < saneNumberOfAttempts; i++) {
        final DaemonClientConnection connection = connector.connect(compatibilitySpec);
        try {
            Build build = new Build(buildId, connection.getDaemon().getToken(), action, requestContext.getClient(), requestContext.getBuildTimeClock().getStartTime(), parameters);
            return executeBuild(build, connection, requestContext.getCancellationToken(), requestContext.getEventConsumer());
        } catch (DaemonInitialConnectException e) {
            // this exception means that we want to try again.
            LOGGER.debug("{}, Trying a different daemon...", e.getMessage());
            accumulatedExceptions.add(e);
        } finally {
            connection.stop();
        }
    }

    throw new NoUsableDaemonFoundException("Unable to find a usable idle daemon. I have connected to "
            + saneNumberOfAttempts + " different daemons but I could not use any of them to run the build. BuildActionParameters were "
            + parameters + ".", accumulatedExceptions);
}

but problem definitely needs to be fixed on server, somewhere around this method
gradle-3.1\src\messaging\org\gradle\internal\remote\internal\inet\SocketConnection.java

public T receive() throws MessageIOException {
    try {
        return objectReader.read();
    } catch (EOFException e) {
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug("Discarding EOFException: {}", e.toString());
        }
        return null;
    } catch (Exception e) {
        throw new MessageIOException(String.format("Could not read message from '%s'.", remoteAddress), e);
    }
}

(Lari Hotari) #3

Hi @xtracoder, Thanks for the detailed problem report.

Could you provide some more details about the class that wasn’t found?

Where is the class defined and how is it used?

One way to get around the problem would be to add this method to the subclass of java.io.File:

    Object writeReplace() throws ObjectStreamException {
        return new File(getAbsolutePath());
    }

(docs for writeReplace)

Do you have the ability to add this method to the subclass of java.io.File?

-Lari


(xtracoder) #4

That class is defined in my client library and occasionally was passed into GradleConnector.forProjectDirectory() (probably it is better to through IllegalArgumentException there if file.getClass() != File.class).

I’ve already replaced it with raw java.io.File, but investigation of the problem took rather long time - current Gradle’s failure in this edge case takes rather long time and is not self-descriptive.

I’ll also try ‘writeReplace()’. I think it should work, need to decide which way is more correct.


(Lari Hotari) #5

@xtracoder , Thanks for the additional information. I agree that it would be better to handle this on Gradle’s side. Could you summarize your use case? It will be easier for me to create an issue to Jira once there is a clear use case.


(xtracoder) #6

I think I’ve already provided all “relevant” information, more details will not add something useful. But since you are asking, then … there is actually no secret.

I’m using Gradle from NetBeans, it has some bottlenecks in file operations - after replacing usage of java.io.File with subclass which optimizes access to file attributes, it sneaked into external world - Gradle plugin in this cases (his author is originator of original question). Because its class is in Netbeans internals - it can’t be de-serialized in Gradle’s deamon.


(Lari Hotari) #7

@xtracoder , Thanks for your help. I’ve created GRADLE-3567 for the issue.