Gradle appears to be deleting build directories between tasks

Help/Discuss
1

Hi folks,

I’m converting a maven-based build to a gradle build. The original POM copied some static configuration files and a configuration files (filtered to substitute variables) to a build directory, and then ran sphinx-build to generate the project documentation.

I’ve created a replacement build.gradle which does the same thing. A generateSphinxConfiguration task generates the configuration; a copySphinxStaticResources task copies the static files (current an empty directory) and a generateDocumentation task depends on these, and then runs the build.

This does work, but fails right at the end. See the scan build log. I added MISSING/EXISTS/CREATE log messages to try to understand the order of execution and at what point the directory was present or missing.

It looks like the _static directory is deleted part way through the build so when the sphinx-build program is executed, it ultimately fails due to it being missing. But the logging I added clearly shows it being present during the early stages of the build.

If I comment out the copy{} block, I get this instead: scan log

Here the copy task doesn’t create the directory; it’s done manually by a createSphinxStaticDirectory task. But. That directory which the logging confirms is created, is deleted as soon as the next task runs. What is responsible for this deletion?

I’m very new to gradle, and so I may be simply misunderstanding something to do with the dependencies or execution order. I don’t get why this behaviour is apparently occurring, and would be very grateful for any advice on this point.

Thanks all,
Roger

2

Sharing output directories or subdirectories thereof is a bad practice in Gradle. I can easily reduce your build script to this that eliminates the problem if I am correct.

import org.apache.tools.ant.filters.ReplaceTokens

apply plugin: 'distribution'

description = 'OME imaging metadata model documentation'

ext {
    sphinxBuildDir = project.file("build/sphinx")
    sphinxBuilder = 'html'
    sphinxStaticBuildDir = project.file("build/sphinx/_static")
    model_version = "${modelSchemaVersion}"
}

task generateSphinxConfiguration(type: Copy) {
    into sphinxBuildDir

    into('.') {
        from project.projectDir
        include 'conf.py'

        filter(ReplaceTokens, tokens: [
            'sphinx_srcdir': "${project.projectDir}".toString(),
            'sphinx_builddir': "${sphinxBuildDir}".toString(),
            'model_version': "${modelSchemaVersion}".toString()
        ])
    }

    into('_static') {
        from '_static'
    }
}

task generateDocumentation(type: Exec) {
    dependsOn generateSphinxConfiguration
    executable 'sphinx-build'
    args '-D', "release=${project.version}",
         '-D', "version=${majorVersion}.${minorVersion}",
         '-c', "${sphinxBuildDir}",
         '-d', "${sphinxBuildDir}/cache",
         '-b', "${sphinxBuilder}",
         '-W', "${project.projectDir}",
         "${sphinxBuildDir}/${sphinxBuilder}"
    workingDir project.projectDir
}

distributions {
    main {
        contents {
            from {
                "${sphinxBuildDir}/${sphinxBuilder}"
            }
        }
    }
}

tasks.withType(Tar){
    compression = Compression.GZIP
    extension = 'tar.gz'
}

distZip.dependsOn(generateDocumentation)
distTar.dependsOn(generateDocumentation)

publishing {
    publications {
        docs(MavenPublication) {
            artifact distZip
            artifact distTar
        }
    }
}

What this does is use the power of the Copy task and nested CopySourceSpec of Gradle.

The first instance of into with no arguments other than the directory defines the overall top level destinations.
The second into is a nested CopySourceSpec using the same directory as the top level but adding a source directory, an include filter and a file token filter.
The third into CopySourceSpec copies in the nested _static destination from the projectDir/_static location. from will always assume relative paths are rooted in projectDir.

The rest of your script is unchanged aside from a small change in the task dependency since there is only one task to do Copy work now.

You problem is caused probably by Gradle parallel execution and since the two competing Copy tasks are sharing the output directory and the createSphinxStaticDirectory did not declare its intent to also use the same output directory some race condition from the overlapping directory operations causes this problem for you. With this there are no output overlaps and you get what you want.

3

Thanks, consolidating the Copy tasks seems to be better, however I’m still running into failures. (Did you run git clean -dxf between runs?) It’s failing for me with a clean tree even with your changes. I needed to change the task to:

task generateSphinxConfiguration(type: Copy) {
    into sphinxBuildDir

    into('.') {
        from project.projectDir
        include 'conf.py'

        filter(ReplaceTokens, tokens: [
            'sphinx_srcdir': "${project.projectDir}".toString(),
            'sphinx_builddir': "${sphinxBuildDir}".toString(),
            'model_version': "${modelSchemaVersion}".toString()
        ])
    }

    into('.') {
        from project.projectDir
        include '_static/**'
        includeEmptyDirs = true
    }
}

(last into changed) for it to copy the empty directory correctly. If I put some content there, your solution worked fine (but it needs to cope with content being present or absent).

Thanks,
Roger