Dependent tasks are not build using artifact Transforms

Help/Discuss
1

Hello

So in the past I roughly used this syntax to add tasks dependcies as declared by the configuration to the task, if automatic wiring does not work:

val useRuntTime by tasks.registering(Sync::class){
    val used = configurations.runtimeClasspath
    from(used.map { it.files.filter { it.extension == ".jar" } })  //just a dummy to break automatic wiring
    dependsOn(used) // add dependency explicitly because the filter... 
    into("build/runTime")
 }

and this will automatically build every project on the runtimeClasspath

However now I am working with artifact transform and that makes me questioning my understanding of gradle…

So now roughly the same thing but now i am using an artifact transform…

val lazyConfig = configurations.resolvable("lazyConfig"){
    extendsFrom(configurations.implementation.get())
    attributes {
        attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, "transformed-jar")
    }
}


abstract class MyTransform : TransformAction<TransformParameters.None> {
    @get:InputArtifact
    abstract val inputArtifact: Provider<FileSystemLocation>

    override fun transform(outputs: TransformOutputs) {
       println("this is very expensive.." + inputArtifact.get() )
        Thread.sleep(1000)
        println("done")
        val asFile = inputArtifact.get().asFile
        if (asFile.isFile) {
            outputs.file(asFile)
        }
    }
}

val useTransform by tasks.registering(Sync::class){
    val used = lazyConfig
    from(used.map { it.files.filter { it.extension == ".jar" } })
    dependsOn(used)
    into("build/transform")
}

And in this example dependend projects are no longer build. However if I I am able to use this syntax:

val useTransform by tasks.registering(Sync::class){
    from(lazyConfig)
    into("build/transform")
}

then project dependencies are build again.

So I am currently a bit unsure: Have I been using a problematic syntax the entire time, is this a bug with artifacts transormations?

2

Your code at least is questionable, but maybe depends on the concrete use-case.

Practically any explicit dependsOn that does not have a lifecycle task on the left-hand side is a code smell and usually hinting at you doing something not the proper way like not wiring task outputs to task inputs or similar.

Your first example should for example probably be something like

from(configurations.runtimeClasspath) {
    include("**/*.jar")
}

or if the filtering by name was an unlucky example, maybe

from(
    configurations
        .runtimeClasspath
        .flatMap { it.elements }
        .map { it.filter { it.asFile.extension == ".jar" } }
)

And your transform example should probably not be an own resolvable configuration, but an artifact view on the runtimeClasspath configuration, but just guessing here your use-case.

Anyhow, asfair even being bad practice, using dependsOn should work properly.
Maybe it would help if you can knit an MCVE.
If you are going to report a bug about it you will most probably need one anyway.

3

minig3.zip (5.0 MB)
Attached a reproducer → executing useTransformBroken will not build dependencies.

Thanks for your help. I had no clue that going that extra mile using getElements “fixes” gradles dependcy tracking.

Practically any explicit dependsOn that does not have a lifecycle task on the left-hand side is a code smell and usually hinting at you doing something not the proper way like not wiring task outputs to task inputs or similar.

Theoretically I would agree. In practice I see often that especially for people that are new to gradle it can be fairly confusing if u need to adapt something in the from block sometimes the dependencies completly break and sometimes they dont. But maybe we are just too used to how ant works

  from(lazyConfig) // auto dependcies work
  from(lazyConfig.map { it.map { zipTree(it) } }) // auto dependencies dont work
    from(lazyConfig.flatMap { it.elements }.map { it.map { zipTree(it)}}) //auto dependencie work but the cc is broken

    val ops = serviceOf<ArchiveOperations>()
    from(lazyConfig.flatMap { it.elements }.map { it.map { ops.zipTree(it)}}) // and that is the correct way of doing stuff i guess
4

and that is the correct way of doing stuff i guess

Actually not, because you use private API. :slight_smile:
serviceOf is not part of the public API and shouldn’t be used.
If you are at a point where you think about using serviceOf, it is almost always quite about time to create a custom task class instead of doing things ad-hoc.
Then you can simply get the service you need @Injected.

A way to get hold of these services with public API would be to define an interface that gets the service injected and then use objects.newInstance to create that interface, then get the service from it.

Besides that, for what you showed an artifact transform seems to be the better way actually.
The documentation example for them even is exactly an unzip transform.

5

Intresting,

I would love to agree with you on that, the problem we run into is that as soon as you implement your own task class gradle starts aligning the classpaths of our plugin/tasks and as soon as anything changs in any plugin all tasks will become outdated due to classpath changes. This doesnt happen with core tasks. So currently there is just inherently a strong motivation to solve everything by configuring gradle core tasks whenever possible…

And well yes it would be possible to use transformations, but transformations pack their own weird sideffects because classpath tracking, intellij sync, Artifacts transforms for projects are never run in parallel · Issue #30867 · gradle/gradle · GitHub …

Back to the topic: I guess its not quite clear what the semantics of dependending on a configuration with transforms actually are. I shall raise an issue and mark this solved?

6

I would love to agree with you on that

Then do, nothing hinders you :slight_smile:

the problem we run into is that as soon as you implement your own task class gradle starts aligning the classpaths of our plugin/tasks and as soon as anything changs in any plugin all tasks will become outdated due to classpath changes.

Well, yes and no.
If you use on plugin in one build script and another plugin in another build script and put them in different plugin projects, they do not disturb each other.
And actually it is not really different with built-in tasks, if they change (because you change Gradle version) also everything is outdated.
Do you really constantly change your build logic classes so that this point would be relevant?

And well yes it would be possible to use transformations, but transformations pack their own weird sideffects because classpath tracking, intellij sync

I don’t remember weird sideeffects with transforms so far. Mind elaborating a bit which you have seen? I have no idea what you refer to with your catchphrases.

Artifacts transforms for projects are never run in parallel · Issue #30867 · gradle/gradle · GitHub …

Wasn’t aware of that, but depending on situation you could maybe work around this by eradicating the need to do transform, by already providing the necessary variants (or secondary variants) directly in the projects.

Back to the topic: I guess its not quite clear what the semantics of dependending on a configuration with transforms actually are.

The semantics are clear.
As I said, even being bad practice, the explicit dependsOn should work.

I shall raise an issue and mark this solved?

No necessary, the dependsOn works fine.
I had a look at your MCVE.
Even if it lacks the “C” and “V” as you did not include the useTransformBroken task, I copied it from this thread.

If you comment out the from line, the dependsOn triggers the task dependencies just fine.
What you hit is a bug that I right now not find the issue for.
The problem is, that with the .map you do there, when determining input files for the task (iirc) the artifact transforms are already run, which you can for example easily see when you run with -m and your transform output comes.
At this time you are still in configuration phase so the tasks are not executed even though the transforms run.

If then later the dependsOn is evaluated it reaches the transform (which has the actual task dependency), the transform knows it is already executed, so just continues without checking its dependencies, because it is already executed, so all must be fine. Due to this the task dependency is lost.

But even if the task dependency would not have been lost, it would even be worse in the current situation, because then the transform would still run first and run on absent or stale files and later on the task dependency would be executed, but the result not used as the result of the already executed artifact transform would be used. (Actually, in your MCVE this would not be the case, but only because it is an identity transform and so the input file is directly used as output file, but if the transform would have done anything it would be the case).

7

Ah, now I found it.
This is the issue you are hitting: Resolving a transformed configuration at configuration time uses non-existing or stale artifacts and breaks later tasks · Issue #19707 · gradle/gradle · GitHub

8

And this would have saved you by complaining about the premature artifact transform: Deprecate premature artifact transformation of not-yet-built artifacts (#27372) by Vampire · Pull Request #29840 · gradle/gradle · GitHub

9

Yes. 300% yes. Well maybe in some far future the pace will slow down, but we still have so many things undone or done highly questionable in the past that need cleanup soon because we didnt know better…
So the thing that happens constantly is something like the senior architect absolutely demands that all jar files contain a buildstamp. Fantastic. Well so we need to modify the jar task… does mean we need to recompile our java-code? no because thats on the gradle classpath. Does that mean we have to have recompile everything else which doesnt even end up in any jar? Yes. Could be addressed if we strictly put everything which gets compiled by our own tasks in synthetic empty projects, but that just feels odd as well.
Yes gradle version will also cause a full rebuild but there isnt a new gradle release on a daily basis…

So a few things that do not make me hyped about transformations

  1. Sometimes transformations are executed during intellij sync. I havent looked deep into this, but when it happens it will obviously slow down the sync dramatically. This very much might be related to 29840
  2. Transformations are also subject to the classpath tracking thing, and you cant “hijack” gradle core transformations like you can with tasks to convince gradle that you are not trying to do anything funny
  3. In almost all situations I had the api that filetree provides seems like a natural fit to what was going on. Like there is a tarball and we just need the dir win64/ but not the .template files and then it also needs to be flatten for whatever reason so its something like
	project.tarTree("sdfsd").matching{
		include("win64/**")
		exclude("**.template")
	}.files

Could be reimplemented as a custom transform with parameters for includes and excludes and flatten, but why reinvent the wheel? Like if I had this part of the build under control there wouldnt be a tarball anyway, just give me these files…

Thanks for analyzing the issue here, your explaenation makes sense.

Well since we are here anyway…

from(runtimeClasspath) // does establish an dependcy

from(runtimeClasspath.map{it.files}) // doesnt establish a dependency
from(runtimeClasspath.flatMap{it.elements}) // establish a dependency
I know that that happens but I dont know why mapping the configuration does not create a dependency, but flatmapping its elements does. what rules apply here?

10

runtimeClasspath is a Provider<Configuration> that itself does not have any task dependencies, the Configuration or the things that are within it have task dependencies.

.map on a Provider preserves task dependencies of that provider, but as I said, the Provider in question does not have the task dependencies.

So by doing the map on Provider<Configuration> you get a Provider<Set<File>> where now neither the Provider nor the thing it provides has any task dependency and so you lost the information.

FileCollection.elements (Configuration is-a FileCollection) is exactly the bridge from a FileCollection to a Provider that preserves the task dependencies of the FileCollection.

Provider.flatMap looses the task dependencies on the provider you call it, returning the Provider you flatMap to.

So by calling .flatMap{it.elements} on the Provider<Configuration> you replace it by the Provider<Set<FileSystemLocation>> that is returned from FileCollection.elements that has the task dependencies and thus preserves the task dependencies.