When is detached configuration guaranteed to be resolved?

Help/Discuss
1

Hi,

I was doing some performance optimizations including using detached configurations instead of non-detached ones.

It looks like resolving detached configuration by calling resolve() method is not blocking and does not guarantee all files are resolved after the method calls finishes.

Used test code:

private static final Map<String, Configuration> artifactGAVToDetachedConfigurationMapper = new HashMap<>()

private void addExtractApolloNativeArtifactExtClosure(Project project) {
    project.ext[APOLLO_EXTRACT_NATIVE_ARTIFACT_EXT_CLOSURE_NAME] = { args ->
        ...
        String artifactGAV = "${args.group}:${args.name}:${args.version}${classifier}@zip"
        def artifactDependency = project.dependencies.create(artifactGAV)
        def detachedArtifactConfiguration = null

        if (artifactGAVToDetachedConfigurationMapper.containsKey(artifactGAV))
            detachedArtifactConfiguration = artifactGAVToDetachedConfigurationMapper.get(artifactGAV)
        else {
            project.logger.info "Creating detached configuration to download artifact '$artifactGAV'."
            detachedArtifactConfiguration = project.rootProject.configurations.detachedConfiguration(artifactDependency)

            artifactGAVToDetachedConfigurationMapper.put(artifactGAV, detachedArtifactConfiguration)
        }

        project.logger.info "Resolving detached configuration resolving artifact '$artifactGAV'."
        def configurationFiles = detachedArtifactConfiguration.resolve()
        if (configurationFiles.isEmpty())
            throw new IllegalStateException("Resolved detached configuration for artifact '$artifactGAV' is empty!")

        project.copy {
            from configurationFiles.collect {
                project.zipTree(it).matching({...})
            }
        }
    }
}

The thrown exception:

Caused by: org.gradle.api.InvalidUserDataException: Cannot expand ZIP '/home/martin/.gradle/caches/modules-2/files-2.1/apollosoft.native.libs/gfxexport/16.07.04/76a11597ee36c0bee0c3255f801e72aa177e87c3/gfxexport-16.07.04-linux-32-release.zip' as it does not exist.

Which is correct as the file does not exist yet. So this looks like resolving detached configuration does not block till the artifact is resolved. Tested on Gradle 4.0-rc-2.

Can you please comment/verify?

2

The method is blocking as it returns Set<File>. I’m not sure whats going wrong here tbh. do you maybe delete this file somewhere in between?

3

Well, no…

The code is essentially resolving the configuration and right after that iterating over the resolved files.

4

I think I found out the problem. I was using custom workflow as I needed to always verify all dependencies are downloaded correctly.

It looks like Gradle Daemon is not always correctly refreshing the existence of files in gradle caches folder (~/.gradle/caches).

To reproduce:

  1. ./gradlew --stop

  2. rm -rf ~/.gradle/caches/

  3. ./gradlew --refresh-dependencies --stacktrace --recompile-scripts clean :project:subproject:task | tee out.txt

    • => this resolves correctly

  4. rm -rf ~/.gradle/caches/

  5. ./gradlew --refresh-dependencies --stacktrace --recompile-scripts clean :project:subproject:task | tee out.txt

    • => this fails on
      Caused by: org.gradle.api.InvalidUserDataException: Cannot expand ZIP '/home/martin/.gradle/caches/modules-2/files-2.1/apollosoft.native.libs/gfxexport/16.07.04/76a11597ee36c0bee0c3255f801e72aa177e87c3/gfxexport-16.07.04-linux-32-release.zip' as it does not exist.

  6. the previous command will always fail with the same error

PID STATUS INFO
27963 IDLE 4.0-rc-2
20507 STOPPED (stop command received)
21983 STOPPED (stop command received)
23276 STOPPED (stop command received)
27322 STOPPED (stop command received)


To recover from this broken state:

1. surprisingly adding `--rerun-tasks` resolves everything correctly
2. stopping Gradle wrapper also works `./gradlew --stop`

I was not able to reproduce it on smaller single project build