Composite: Copy files from one subproject to another

Help/Discuss
1

Hi,

I’m trying to refactor a monolith into modules using a composite project. Right now this is in 1 repo, but eventually I want to split them (just not yet).

I have some files that need to be shared across several subprojects just for testing purposes. I came across TestFixtures, which seem to be just what I need. For classes and anything that can be read from the classpath directly this works great. But some files need to be accessible from a relative path on the filesystem.

I have created a small sample project GitHub - schaarda/MoveFilesSample: Gradle-Example to move files between composite builds which (hopefully) shows what I want to achieve and what I have so far.

ComponentA contains a file under src/testFixtures/resources/tomcat.p12. I want component B to copy this file to it’s build-Folder.

Using this task

val a by tasks.registering(Task::class) {
        dependsOn("clean","assemble")
        doLast{
            var x = configurations["testCompileClasspath"].copyRecursive{it.group == "org.example" && it.name=="ComponentA"}
            resources.text.fromArchiveEntry(x,"tomcat.p12")
                .asFile().copyTo(file("build/testFolder/tomcat.p12"))
        }

I can do this if the files are in src/main.

With TestFixtures I now have two problems:

  1. The filtered configuration has 2 archives: the ComponentA.jar and ComponentA-test-fixtures.jar. The method fromArchiveEntry can’t handle this, so how can I filter further so that only the test-fixtures.jar remains?

  2. If the files are in src/main, the corresponding jar from ComponentA get’s built first, so there is a jar to extract. When in testFixtures, the jar is not build (not even if I dependOn(“testFixturesJar”)). Gradle seems to link directly to the project of ComponentA. Things on the classpath are found by ComponentB, but there is no .jar in the build/libs-Folder of A, so the fromArchiveEntry runs into a FileNotFound.

I’dont want to use a relative path across multiple projects as eventually I want to build them completely on their own just with the corresponding dependencies.

Bonus challenge: How can I do this with multiple files in 1 folder?

Any ideas on how to solve this? Is it really this hard to get 1 file from the projects classpath in Gradle or are there better ways?

2

I don’t think using copyRecursive and then filtering is too much idiomatic.
I would simply declare an additional custom configuration.
On that configuration declare the one (or multiple) dependencies where you want to pull files out.
This configuration can also be declared non-transitive to not get the transitive dependencies that you are not interested in here (like the main jar of component A).

For the unpacking, I would probably use an artifact transform.
This can then also handle multiple files or multiple such dependencies easily.

3

I tried with filtering because I need that dependency as test-fixture anyway. But using an additional configuration should not be a problem.

But does this solve the problem about test-fixtures having 2 archives or should I put these files in a completely separate module? And there not as test-fixtures but as src/main?

Never heard about artifact transform, so I’ll have to dig into that. Thanks for the pointer.

4

As I said, the test fixtures variant has a dependency on the main variant.
So why you get both is because the dependency and as you filter by coordinates only but not by variant you get both artifacts.
And by using a non-transitive dependency or configuration, you would not get it.

Btw. if you want to avoid declaring the dependency twice, you could e.g. make testImplementation extendsFrom your custom configuration and then only declare it on your custom configuration.

5

Ok, using my own configuration works, so I get only the test-fixture dependeny.

Unfortunately the problem number 2 from my initial question remains:

Building the test-fixtures.jar in ComponentA is not triggered from B. So this only works, if the jar already exists before starting the build. Should Gradle not have an implicit dependency on the testFixtureJar-Task because I require it from B?

I can get the FooClass() from A in B (so the dependency itself is found), but there is no jar-File I can extract because the task testFixturesJar is not run in A.

6

You don’t declare it as input for the task, so there is no implicit task dependency.
Actually any explicit dependsOn is a code smell (unless there is a lifecycle task on the left-hand side) and should be avoided.

But if you use an artifact transform, situation will change anyway.
In the end you should have an implicit dependency, yes.

7

I’ve tried to wrap my head around the artifact tranform, but I can’t make heads or tails.

Do I understand correctly that I have to define a custom attribute, set this to false on all artifacts. Then require it on true for my configuration and write a Transformer that extracts the archive setting the attribute to true. And of course register it?

That’s an awful lot to just extract a file from an archive.

I tried to implement this (in a rough way) into my MoveFiles-Sample-Project but still have the same problem as before:

If I clean componentA so that the test-fixtures.jar is removed, componentB resolves the configuration without errors, but the jar-File is (of course) not there to be extracted. And why is the outputs-Folder of this transformation in ComponentA although the transformation happens in ComponentB?

8

That’s an awful lot to just extract a file from an archive.

It is, but it has other pros, like you don’t need to do it on each build invocation but it is only done once, you can handle multiple dependencies at once, you can handle multiple files easily, …

It was just a recommendation of how to do it properly.

Of course if it is only about extracting one file from one dependency, you can simply use a zipTree in combination with a Sync task or sync { ... } method call to just do it.
That’s also what I would use inside the transform implementation actually: I m trying to figure out why when declaring configurations a gradle-community #community-support

Btw. setting transitive to true on testImplementation should not be necessary.
Neither this setting, nor attributes should be inherited through extendsFrom, just the declared (and inherited) dependencies.

That it is not working as expected in your MCVE is because of the configuration resolving at configuration time. That is bug Resolving a transformed configuration at configuration time uses non-existing or stale artifacts and breaks later tasks · Issue #19707 · gradle/gradle · GitHub.

If you remove that part and instead properly use the configuration e.g. with

val b by tasks.registering(Sync::class) {
    from(etcConfiguration)
    into(temporaryDir)
}

it works as expected.