Native Binaries: parallel builds with visualcpp usually fail

I’m finding that when gradle builds projects using the visualcpp toolchain, it does not appear to be making very good parallel execution choices. For example, it seems to try to run compilation of each cpp source file it finds in parallel. This has disastrous consequences when producing pdb files, or when different translation units refer to the same header file. In these cases, you’ll get two worker threads asking the compiler to access or modify the same physical file (asm’s, pdb’s, etc), which crashes the build.

It also does not seem to honor dependency relationships between components, and so runs build activities on all components simultaneously even if, for example, an executable component has a library dependency on another component in the build script. Rather than waiting for build activity on the dependency to complete, it appears to just start building everything all at once.

To date, I’ve only been able to work around these problems by handicapping the compiler with options like /FS, and launching the build script with --max-workers=1.

Has anyone else found these same problems? Is there are workaround? Are there some parallel build controls that I’m missing?

Are you using the /Fd option as well? I had a similar issue with the Visual Studio compilation threads accessing the same database file. I resolved it with the following:

if (toolChain in VisualCpp) {
    cppCompiler.args(
        '/FS',   // serialize writes to PDB (VS>=2013)
        "/Fd${it.component.baseName}-${it.name}.pdb", // give each component its own database file
        '/Zi')   // program database
}

Yes - that’s the same workaround I’ve had to use, but I was hoping gradle would be detecting inter-component dependencies and 1) only process source files in parallel if they belonged to different components (but compile serially within a component) and 2) delay build activity on dependent components until build activity on their dependencies is finished.

@Kevin2: I think I’ve been operating under the mistaken assumption that gradlew build would ensure that components are built in dependency order. Instead, I’m finding that components get built in alphabetical order, with no regard at all to dependencies.

I was wrong in stating that all source files for all components are compiled simultaneously. It appears that only source files for the component being built by the task in process at the time are compiled. But, the ordering of tasks is not what I was expecting.

If I have two components: zLib, a library, and test, an executable that links in zLib, and I run gradlew build, it will try to build the test component first, because it comes first alphabetically among the available tasks. Of course it can’t successfully do this because zLib has to be built first.

This seems pretty unfortunate to me, because now it seems that I can’t simply run gradlew build and expect it to work. I also have to know what the dependency relationships are, and I have to specify all of the tasks for all the binaries I want in dependency order on the command line, like so: gradlew zLib<platform><variant>StaticLibrary test<platform><variant>Executable.

This seems completely impractical to me - am I missing something obvious?

For reference, my build script is set up like so (lots of compiler/linker options and so on omitted for clarity):

model {
    platforms {
        windows_x86 {
            architecture 'x86'
            operatingSystem 'windows'
        }
        linux_x64 {
            architecture 'x86_64'
            operatingSystem 'linux'
        }
    }

    buildTypes {
        debug
        release
    }

    repositories {
        libs(PrebuiltLibraries) {
            spdlog {
                headers.srcDir "../spdlog/include"
            }
            bandit { headers.srcDir '../bandit'}
            // others...
        }
    }

    components {
        zLib(NativeLibrarySpec) {
            targetPlatform 'windows_x86'
            targetPlatform 'linux_x64'
            sources {
                baseName(zlib)
                binaries.withType(SharedLibraryBinarySpec) { buildable = false }
                cpp.lib library: 'spdlog', linkage: 'api'
                // other dependencies
            }
        }
        test(NativeExecutableSpec) {
            targetPlatform 'windows_x86'
            targetPlatform 'linux_x64'
            sources {
                baseName 'test'
                cpp.lib library: 'zLib', linkage: 'static'
                cpp.lib library: 'bandit', linkage: 'api'
                cpp.lib library: 'spdlog', linkage: 'api'
            }
        }
    }
}

Whoops - I completely misunderstood the task model. Because I was getting compilation errors and stalling out there, I didn’t realize the obvious organization of link tasks following all of the compilation tasks avoids any ordering problems with compilation.

As Emily LaTella would say, never mind!