Sharing directory created by subproject task with task in another subproject

Help/Discuss
1

There are a bunch of examples and bits of documentation related to parts of this, but I seem to be missing the bigger picture, so I’m posting here.

I have a build consisting of a number of subprojects, where subprojects build on each other’s outputs. My problem is with one subproject that produces a shared library (.so/.dylib) and a set of dynamically created C++ sources, and the other subproject which needs those artifacts to produce a Python wheel.

Here’s what I’ve been trying:

cpp/build.gradle.kts

val sharedLibFiles by configurations.consumable("sharedLibFiles")
val cApiSourceDir by configurations.consumable("cApiSourceDir")


// A custom task which builds the shared library files and lists them in the task outputs
val sharedLib = tasks.register<SharedLibTask>("shared-lib") {
  outputDir = project.layout.buildDirectory.dir("c")
}

// A custom task which has an @OutputDirectory named outputDir
val cApiSetup = tasks.register<CApiSetupTask>("api-setup")

sharedLibFiles.outgoing.artifacts(sharedLibFiles.map { it.outputs.files }) { builtBy(sharedLib) }
cApiSourceDir.outgoing.artifact(cApiSetup.map { it.outputDir })

python/build.gradle.kts

val sharedLibsBucket by configurations.dependencyScope("sharedLibFilesBucket")
val sharedLibs by configurations.resolvable("sharedLibFiles") {
    extendsFrom(sharedLibsBucket)
}

val cApiSourceDirBucket by configurations.dependencyScope("cApiSourceDirBucket")
val cApiSourceDir by configurations.resolvable("cApiSourceDir") {
    extendsFrom(cApiSourceDirBucket)
}

dependencies {
    sharedLibsBucket(project(mapOf("path" to ":cpp",
        "configuration" to "sharedLibFiles")))
    cApiSourceDirBucket(project(mapOf("path" to ":cpp",
        "configuration" to "cApiSourceDir")))
}

// custom task that does the python build
val buildWheel = project.tasks.register<BuildWheel>("build_wheel") {
    cApiSourceDir = cApiSourceDir.map { it.artifacts().files.singleFile }
    sharedLibFiles = sharedLibs.map { it.artifacts().files }
    outputDir = layout.buildDirectory.dir("wheel")
}

The aim is to provide the path to the output directory to the cApiSourceDir property of the BuildWheel task, which is a DirectoryProperty, and the files from the sharedLibs configuration to the sharedLibFiles property of the BuildWheel task.

At the moment I can’t even get the python.gradle.kts to compile - there are errors about being not being able to reassign a val, type mismatches, and missing operator =. I usually see these things when there’s type mismatches which mean the Gradle magic property DSL stuff isn’t able to be invoked because the type signatures aren’t matching.

One of the other parts of the project generates various Java jars and things that get consumed by other bits of the real CPP project, which is what I’ve based the above on, but I’m basically cargo culting it and I can’t figure out what I ought to do from the other documentation.

Does anyone have a suggestion, or an example you could point me to? Thanks!

2

Your thought about type mismatch was exactly right.

As you use val sharedLibs by configurations.resolvable... and val cApiSourceDir by configurations.resolvable..., those variables are of type ResolvableConfiguration.
Those are Iterable<File>.
So the .map you call is not Provider.map but Iterable.map which means you are right away resolving those configurations at configuration time and then map on the resolved files, which fails as File does not have an artifacts() method, and as a follow-up error you get the “cannot reassign val” as the type does not match as it cannot even be inferred:

image
image743×85 15.9 KB

If you would not use by but = and artifacts not artifacts(), you would indeed get Providers out:

image
image941×87 22.6 KB

But even that might not be what you want eventually. :man_shrugging:

3

That was super helpful, thank you.

I managed to get everything linked together well enough to compile and for tasks to run. Now the problem is that using the consumable configurations doesn’t result in the creating tasks being triggered.

Once again, I’m dangerously close to pure copy-pasta cargo-cult land with this stuff, so I’ve created the most trivial demonstration of this I can and put it on GitHub: GitHub - fidothe/subproject-artifact-dependencies: Demonstrating / hopefully exploring and fixing problems with Gradle subproject artefact sharing.

The consuming task is getting the correct paths from the ResourceConfiguration, but the creating task is not being invoked to actually create them.

It feels like the dependency source information is being somehow lost, but I’ve been really belt-and-braces about it: I would have thought that this would have been sufficient by itself:

val generator = tasks.register("...")...

val srcFiles by configurations.consumable("srcFiles")
srcFiles.outgoing.artifacts(generator.map { it.outputs.files })

But I’ve got the builtBy declaration in there as well, just in case, and it’s still not working:

srcFiles.outgoing.artifacts(generator.map { it.outputs.files }) { builtBy(generator) }

I must be missing something obvious, but it’s not obvious to me, as is always the way…

4

The problem is not the producer-side.
You discard the dependency on the consumer side.

By setting src = project.provider<File> { srcFiles.first() } you set src to a newly created provider without the necessary task dependency information, what you do inside the lambda is unimportant for that.

Similarly if you call .flatMap on a Provider instance you also loose the task dependency information, unless the provider you return in the lambda has the same information anyway.

If you call .map on a provider, the task dependency information is preserved.

If you use .elements on a Configuration you get a Provider<FileSystemLocation> that also preserves the task dependency information.

So in your case you could for example do src = srcFiles.elements.map { it.first().asFile } and the task dependency is preserved.

5

Oh, .elements is the magic invocation! That’s brilliant, thank you.

(I’ve updated my sample public repo to use this, hopefully that may be useful to other baffled folks like me in the future.)

1 Like