Duplicate resources in class path when running build task

[Skater901]

Hi all,

I’m having a problem with Gradle detecting duplicate resources in my classpath when running the build task. I’m hoping you can explain to me why, and what I can do to fix it.

Firstly, the project I’m dealing with can be found here: GitHub - Skater901/wc3connect-notification-bot: A bot for notifying when a game is hosted on w3cconnect

There are four problems that occur when running ./gradlew clean build.

The first problem is:

* What went wrong:
Execution failed for task ':processResources'.
> Entry META-INF/services/ch.qos.logback.classic.spi.Configurator is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/8.5/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

This can be fixed by commenting out the following line in the main build.gradle.kts file: wc3connect-notification-bot/build.gradle.kts at main · Skater901/wc3connect-notification-bot · GitHub
resources { srcDir("src/main/resources") }

After commenting that line out, the second problem is:

* What went wrong:
Execution failed for task ':processTestResources'.
> Entry fixtures/wc3connect/games.json is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/8.5/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

This can be fixed by commenting out the following line in the main build.gradle.kts file: wc3connect-notification-bot/build.gradle.kts at main · Skater901/wc3connect-notification-bot · GitHub
resources { srcDir("src/test/resources") }

After commenting that line out, the third problem is:

* What went wrong:
Execution failed for task ':discord-module:processResources'.
> Entry META-INF/services/au.com.skater901.wc3connect.NotificationModule is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/8.5/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

This can be fixed by commenting out the following line in the discord-module/build.gradle.kts file: wc3connect-notification-bot/discord-module/build.gradle.kts at main · Skater901/wc3connect-notification-bot · GitHub
resources { srcDir("src/main/resources") }

Finally, after commenting out that line, the fourth problem is:

> Task :test

JdbiChannelNotificationDAOITCase > initializationError FAILED
    liquibase.exception.CommandExecutionException at CommandScope.java:258
        Caused by: liquibase.exception.ChangeLogParseException at XMLChangeLogSAXParser.java:128
            Caused by: java.io.IOException at ResourceAccessor.java:255

It’s not really clear from the console output, but the error is Liquibase complaining that it detected two migrations.xml files in the class path; one in build/resources/main, and one in build/libs/wc3connect-notification-bot-1.0-SNAPSHOT.jar. This can be fixed by commenting out the following line in the main build.gradle.kts file: wc3connect-notification-bot/build.gradle.kts at main · Skater901/wc3connect-notification-bot · GitHub
runtimeOnly(project(":discord-module")) // TODO make run depend on compile

Now, the first three problems are just things I’m curious about; I can fix these problems simply by removing the mentioned lines from the Gradle build files. The fourth one is the real problem; I need the discord-module to be a runtime dependency of the main project, but having it as a runtime dependency is stopping my database integration tests from running.

So my questions are:
1: Why does explicitly specifying the resources folder cause Gradle to detect my resource files multiple times?
2: Why does adding the discord-module as a runtime dependency of the main project cause the migrations.xml file to show up twice in the classpath?

Thanks!

[Vampire (Björn Kautler)]

There are four problems that occur when running ./gradlew clean build.

The first problem is, that you do run “clean build”, unless this was only for demonstration purposes.
Always using “clean” to get consistent and reliable results is something you need with inherently broken and bad Maven. With Gradle this is just a big waste of time and hindering Gradle in doing what it is exceptionally good at, avoiding unnecessary work.

But that aside.

The first problem is:

-

the third problem is:

src/main/resources is already the conventional resources directory and you add it a second time, thus duplicating all resource files.
Actually the whole two sourceSets block just try to configure what is already the default.
Remove them completely and these problems are gone.

1: Why does explicitly specifying the resources folder cause Gradle to detect my resource files multiple times?

Because a + a = 2 * a
If you really want to set the defaults again needlessly, use setSrcDirs to overwrite what was previously there instead of adding to it.
But really, don’t do that, just delete the whole sourceSets block.

2: Why does adding the discord-module as a runtime dependency of the main project cause the migrations.xml file to show up twice in the classpath?

I guess that is because you use the intended unit test suite for integ tests.
Because the test test suite is specially set up in a way that it does not need to build a jar to run the tests but simply use the compile result and process resources result in directories.
By then having a dependency on the discord module which has a dependency back on the root project now the root project is again included as jar and you get the duplicate resources.

I’m not 100% sure, but this might as well be a bug in Gradle that you maybe should report.

To mitigate the problem there are multiple ways.
You could for example make the discord module use compileOnly instead of implementation for the root project, then the jar is not introduced additionally.
Or you could refrain from using the unit test test suite for integ tests and instead create a dedicated test suite for integ tests, because there you do not have the special setup and both dependencies, the one on the main source set from the tests and the one through the discord module both bring in the jar and that only once overall.
Or you could break that strange cycle that root depends on discord and discord depends on root, that sounds wrong architecturally anyways.

[Skater901]

@Vampire

The first problem is, that you do run “clean build”, unless this was only for demonstration purposes.
Always using “clean” to get consistent and reliable results is something you need with inherently broken and bad Maven. With Gradle this is just a big waste of time and hindering Gradle in doing what it is exceptionally good at, avoiding unnecessary work.

I mean… sometimes I find things still don’t work, even with Gradle. It’s just habit I guess, doing IT 101; “Have you tried turning it off and on again?”

@Vampire

src/main/resources is already the conventional resources directory and you add it a second time, thus duplicating all resource files.
Actually the whole two sourceSets block just try to configure what is already the default.
Remove them completely and these problems are gone.

@Vampire

Because a + a = 2 * a
If you really want to set the defaults again needlessly, use setSrcDirs to overwrite what was previously there instead of adding to it.
But really, don’t do that, just delete the whole sourceSets block.

I’m kind of surprised… I thought Gradle would de-duplicate source sets. But yeah, I don’t need to specify the defaults; this is my first time doing a multi-project setup so I’m still learning what I have to do.

@Vampire

You could for example make the discord module use compileOnly instead of implementation for the root project, then the jar is not introduced additionally.

Good idea, thanks, I’ll keep that in mind.

@Vampire

Or you could refrain from using the unit test test suite for integ tests and instead create a dedicated test suite for integ tests, because there you do not have the special setup and both dependencies, the one on the main source set from the tests and the one through the discord module both bring in the jar and that only once overall.

I’ll have to look into this. I’ve never looked at configuring different types of tests in Gradle before, and I’m not even sure what the difference is. If my test setup is incorrect, it’s definitely worth trying to fix that.

@Vampire

Or you could break that strange cycle that root depends on discord and discord depends on root, that sounds wrong architecturally anyways.

Hey if you have any ideas for this I’m all ears. I can’t see any other way to do it, though.

Hopefully you can understand from my code base what I’m trying to do? Basically I want to be able to make multiple modules, and each module can interface with a specific protocol. But then I want those modules to be automatically detected and registered in the main app. The only way I can see to do that, though, is with the setup I have currently; the modules require the main app as a dependency so that they can implement the interfaces, but the modules have to be in the runtime classpath of the main app so that the service loader can load them.

Thanks for your time answering my questions!

[Vampire (Björn Kautler)]

I’m kind of surprised… I thought Gradle would de-duplicate source sets.

Feel free to post a feature request for it.
I have no idea whether this is intentional or an oversight by the Gradle folks.

Hey if you have any ideas for this I’m all ears. I can’t see any other way to do it, though.

Sorry, I’m not going to analyze your architecture for you.
But there are usually always ways to avoid dependency cycles.
For example pull the things that are needed by discord module and root project into a common project, then make the discord module and the root project depend on common.
Et voilá, cycle broken.