Gradle 9: Collecting artifacts from multiple projects after variants have been resolved

Hello.

I’m trying to collect Android Lint reports from multiple gradle modules and then aggregate them in the root module. And I’m trying to do this the project-isolation-compatible way, so no direct references between projects. From what I could read, the best way to do this was using configurations. Here is what I came up:

First, I would create a configuration in the every module:

project.configurations.create("reportAggregatingConfiguration")

Then, in producer modules, I would configure all AndroidLint tasks to push artifacts into the created configuration:

tasks.withType(AndroidLintTask::class.java).configureEach { lintTask ->
   artifacts.add("reportAggregatingConfiguration", lintTask.sarifReportOutputFile) { artifact ->
         artifact.builtBy(lintTask)
      }
}

In the root module, I would first automatically add a dependency to all subprojects (since I don’t want to specify every single module manually):

subprojects {
   dependencies.add(
      "reportAggregatingConfiguration",
      dependencies.project(
         mapOf(
            "path" to it.isolated.path,
            "configuration" to "reportAggregatingConfiguration"
         )
      )
   )
}

Then, finally, I would create a task in the root module that receives all of the files added to that configuration :

project.tasks.register("reportMerge", SarifMergeTask::class.java) { task ->
   task.input.from(configurations.getByName("reportAggregatingConfiguration").incoming.artifactView {
      it.isLenient = true
   }.files)
}

Above worked fine until Gradle 9, but now it crashes with the Cannot mutate the artifacts of configuration ‘reportAggregatingConfiguration’ after the configuration was consumed as a variant. After a configuration has been observed, it should not be modified. error.

From what I can see, the issue is that AndroidLint tasks get created and configured after all variants are resolved and after my configuration has been consumed, freezing it.

How could this be resolved? Is there an alternate path that I can take?

Thanks.

From what I could read, the best way to do this was using configurations.

Actually using variants.
This of course includes using outgoing configurations on the producing side.
But on the consuming side directly saying “give me configuration X” is discouraged by the Gradle folks and instead attribute- / variant-aware resolution should be used.

You can for example have a look at the test report aggregation and jacoco report aggregation plugins that use exactly that tactic to do exactly what you want, creating an aggregated report over all projects.

builtBy(lintTask)

This should not be necessary, assuming sarifReportOutputFile is a Property that is marked with @OutputFile, because those inherit the task dependency automatically.

Then, in producer modules, I would configure all AndroidLint tasks to push artifacts into the created configuration

You actually don’t do what you say you do.
At least not unless task-configuration avoidance is broken for those tasks.
By using configureEach, you adhere to task-configuration avoidance, that means this block is only executed if the task is actually realized, usually because it is going to be executed for some reason.
But if the task is not going to be executed, it will not be realized and thus the artifact will not be added to the configuration.

I’m not sure whether there is a good way to configure an unknown amount of tasks of a specific type as artifacts without breaking task-configuration avoidance for those tasks. There might not be, then you could for example use all instead of configureEach which will cause those tasks to always be configured no-matter what. Maybe it works to define an intermediary lifecycle task that depends on all those lint tasks by type and keep the configuration you have, except for also having the builtBy set to the lifecycle task or something like that, no idea never tried something like that yet.

subprojects { ... }

That is configuring the subprojects dependencies, so besides breaking IP and adding project coupling, you also just add self-references. You probably wanted to do subprojects.forEach { ... } which iterates over the subprojects instead of configuring them.

it.isLenient = true

This is most probably not what you want.
lenient on an artifact view means, that you ignore each and every problem, including download problems, resolution problems, … not just ignore if a dependency does not have the variant.
The “ignore if dependency does not have this variante” leniency is the default for artifact views and can also not be disabled.

What you probably want to do is to add a componentFilter { it is ProjectComponentIdentifier } to your artifact view to only consider dependencies on projects in the current build and completely ignore external dependencies.

but now it crashes with

That’s probably because of the mentioned subprojects { ... } instead of subprojects.forEach { ... } I guess.

You can for example have a look at the test report aggregation and jacoco report aggregation plugins

Thanks, I looked into it and changed the code.

This should not be necessary, assuming sarifReportOutputFile is a Property

You are right, this was an artifact of something I was testind out. Removed.

You actually don’t do what you say you do.

Actually, this is exactly what we want, I think I did not phrase it well enough.

What we want is to only configure lint tasks that are used. So if I run ./gradlew :app:lintDebug reportMerge, I only want the debug variant of the lint task in the app module to get configured. I don’t want every single other lint task in the project to get configured. This should especially help with the configuration on demand enabled. Of course, the downside of this is that I must always pair reportMerge run with the corresponding lint calls, but I consider it as a lesser of two evils.

You probably wanted to do subprojects.forEach { ... }

Thanks, changed. I thought that subprojects {} was just an alias for the subprojects.forEach {}. Is there a documentation explaining the difference?

The “ignore if dependency does not have this variante” leniency is the default for artifact views and can also not be disabled.

Hm, this does not appear to be working for me. The issue are the modules that do not expose any artifacts, but are still included in the subprojects.foreach { (due to IP, I cannot really check and filter modules here). And when I attempt to run the project, it crashes out with the No matching variant of project :module-without-lint was found..

Here is my code: GitHub - matejdro/issues at gradle-artifact-aggregation. If you run ./gradlew lintDebug reportMerge, it finishes normally, but if isLenient is removed, it fails. But maybe I screwed up something with the variant configuration.

That’s probably because of the mentioned subprojects { ... } instead of subprojects.forEach { ... } I guess.

Changing this does not appear to fix the issue.

Here is my code: GitHub - matejdro/issues at gradle-artifact-aggregation. f you run ./gradlew lintDebug lintRelease reportMerge, it finishes normally, but if gradle version in the gradle-wrapper.properties is bumped to the 9.0.0, it fails.

Is there a documentation explaining the difference?

I don’t know, the only place coming to mind in the current docs is the warning not to use the former: Sharing Build Logic using buildSrc>.
subprojects { ... } is a DSL Block that cross-configures other projects which is highly discouraged bad practice.
subprojects.forEach { ... } is just iterating over the subprojects collection and as long as you only access unmodifiable information like the path or name of the projects that is fine. Getting other information from them is not good either and almost as bad as the cross-project configuration.

Hm, this does not appear to be working for me. The issue are the modules that do not expose any artifacts, but are still included in the subprojects.foreach { (due to IP, I cannot really check and filter modules here). And when I attempt to run the project, it crashes out with the No matching variant of project :module-without-lint was found. .

Even without IP you must not do any such check in the subprojects.forEach { ... }. As mentioned above, that would be almost as bad as cross-project configuration.

It does work, as I described it.
The artifact view ignores dependencies that do not have the variant.
But you don’t come that far with such ill dependencies, because the configuration first has to match a valid variant and after that the artifact view can use artifact transforms and / or variant reselection to select a different variant and there dependencies that cannot be satisfied are ignored.

Again, if you use lenient mode of the artifact view, you ignore practically everything that might go wrong like a catch (Throwable) { } and that is almost never really what you should do. It would be better if you would get your dependencies fixed by either making sure all projects provide such a variant, or by only depending on those projects that do provide that variant.

Changing this does not appear to fix the issue.

Well, the problem is that you ignore deprecation warnings.
You really should not do that, especially before doing a major version upgrade.
If you run with 8.13 it tells you exactly where the problem is and that it will fail with 9.0.0.

In the configureEach for the lint tasks you try to add artifacts after configuration took part in resolution.
But the main question is, why it already took part in resolution, because this should usually be avoided at configuration time.

I’m not 100% sure, but I think it is because of the “bad intentions” you have. :smiley:
The tasks.withType(AndroidLintTask::class.java).configureEach action is only evaluated when those tasks are going to be executed (if configuration avoidance is not broken).
The :app:sarifReportElements configuration is marked as already consumed while the task dependencies for the root project were determined.
Then when configuring the lint task for the subproject it is too late to add an artifact which would also add the task dependencies necessary for normal usage.

I have no idea how you can achieve what you intend to, never tried something like that.

Is there no good way to filter and only get modules that provide the variant? First option is not really good, because it would require me to add extra clutter code to the non-android modules. And the second option would require me to manually declare every single valid module in the project (and then maintain that list over time), which would be a big maintenance hassle.

Well, the problem is that you ignore deprecation warnings.

Guilty as charged. That’s why I want to do this the right way this time :slight_smile:

But the main question is, why it already took part in resolution, because this should usually be avoided at configuration time.

Could this be a bug in the AGP?

bad intentions

Which intentions are bad here?

Essentially we want to do the same thing the Jacoco plugin does. The issue is that, unlike Jacoco plugin, we cannot edit the original Android Lint tasks to add the support, so we need to do it from “the outside”.

Is there no good way to filter and only get modules that provide the variant?

None I’m aware of besides having a list somewhere.
You should not reach into the project model of another project to check anything, whether you use IP or not.
You can of course use the isLenient as long as you are aware that it will ignore each end every error, no matter what it is (almost).
Or as I said, you take care each of the dependencies is resolvable by the configuration your view is based on whether it provides the variant or not. You could have a convention plugin that ensures this that you apply to all projects, so the clutter would just be one additional line if you don’t have a convention plugin applied to all projects already anyway.

Could this be a bug in the AGP?

I don’t think so, I described what I think is the reason.
But you never know, AGP is always special and I don’t do Android development.

Which intentions are bad here?

The “only aggregate what got executed anyway”, that is conceptually not really idiomatic. :slight_smile:

Essentially we want to do the same thing the Jacoco plugin does.

No, the JaCoCo plugin does not conditionally only aggregate what was executed in the same build anyway. With the JaCoCo report aggregation plugin if you say “give me the aggregated report” it will execute the tests in the dependency projects that produce the to-be-aggregated JaCoCo result files and then aggregates them.

Okay, I think I got it. Thanks a lot for your help!

1 Like

In case anyone stumbles upon this thread and has a similar issue:

Our final solution was to create an intermediate merge task. This task would get created immediately (before variant resolution). All lint tasks would put all their outputs into a ConfigurationFileCollection that would be the input of the intermediate task. Then, intermediate task would first do a intermediate merge of all report files for this module into a single module-wide report file. This final report file would then get exposed as an artifact to the root module.

1 Like