Defining subproject dependencies for particular artifacts

Help/Discuss
1

I’m trying to refactor a complex build into subprojects. I’m not sure I completely understand the documentation, so I thought I’d see if someone who perhaps understands it better than I could provide some advice.

I have project “alpha” that builds an application. If you just let compileJava run, it builds an application that includes all of the sources from src/main/java and is suitable for debugging. There are also build targets for three separate jars, A, B, and C that (through a variety of interesting mechanisms) are built from different sets of sources that have been transformed and copied to build/src/A, build/src/B, etc.

Now I have a second project “beta” that is built using build artifacts from “alpha”. If I specify that “beta”
has a dependency on “alpha”:

dependencies {
  implementation project(':alpha')
}

I can see that the libs/java.jar build artifact from “alpha” is added to the classpath. That’s fine and makes sense. Or at least, I think I see why that works from the documentation.

But I need to have build targets in “beta” that depend specifically on A.jar (or B or C), not just on the sort of unifed whole.

How should I approach this?

2

To me it is a bit unclear what the situation is, can you maybe knit an MCVE?
If you depend on alpha by default you get an alpha.jar as dependency with the classes from the main source set.
If you for example in alpha have three source sets A, B, and C and some configuration that combines them into the main jar and configures the name to be java.jar, then that is what you get.

But from your abstract description it is hard - at least to me - to identify what the situation is concretely.

3

To make the problem explicit, the sources under /src/main/java in alpha are combined to produce the enterprise version alpha-EE, the professional version, alpha-PE, and the home version, alpha-HE. These are different jar files. There are different combinations of classes for the three versions with some…interesting preprocessing done. So the build process is to make alpha-EE transforms the sources under src/main/java in different ways to build/src/ee/java and then that’s compiled to make the alpha-EE.jar. Similar processes are used to make the PE and HE jars. Is this an absolutely ideal way to structure a Java project? Maybe not, but there are some things I am not in a position to change!

I want to know how to make a task in beta depend on the alpha-EE artifact in particular, not the general libs/java.jar artifact that is the combination of all of them. (That does compile, and it is used for some testing, but it’s not a release artifact.)

4

I should add that making :alpha:eeJar a dependency of a task in beta and then adding project(':alpha').buildDir/eej/classes to the classpath of the task explicitly seems to work. But reading the documentation about dependencies between subprojects makes me feel like that’s not a recommended approach.

5

That’s not just not recommended, that is highly unsafe and should definitely not be done. :slight_smile:

But when you have read that warning, you were already at the right page that describes how to share build outputs between projects.

So you would create three (assuming you need all accessible) outgoing configurations and define the artifacts on them, then you can depend on those configurations from beta.

Or for the “non-simplified” way, you would declare feature variants for the three artifacts and then request the according capability you need.

An alternative could be to have three subprojects or sisterprojects of alpha which build the respective jar and either get the sources of alpha as input and then do their respective transformations and processing, producing the respective jar, or you do the transformations in alpha and get the transformed sources in those new projects.

Actually, the source files of alpha are already available as an outgoing configuration if you are on a recent enough Gradle version, you can check with ./gradlew alpha:outgoingVariants. There should then be a mainSourceElements configuration with the default capability. You can get this by depending on the named configuration, or by matching by attributes accordingly, to get the sources in the new projects and then transform them.

6

I have read that page more than once and failed to grok it. I shall try again. Or perhaps I’ll construct a trivial MCVE and let someone point and laugh at where I’m being an idiot. Won’t be the first time!

7

Yeah, that part is not particularly easy to get if you are new to it.
I threw together a quick example of the two variants I mentioned.
The concrete details of course heavily depend on the actual needs as there are quite some nuances, but you should be able to at least get the general idea.
For example in both variants beta compiles against untransformed alpha but then uses the transformed variants for the additional run tasks.
The build script of beta is probably more complex than necessary, as it cares about all four variants, in reality you might maybe just need one of the variants, of have an own beta for each variant and thus do not need all the custom configurations and so on.
Those are some of the “nuances” I meant. :slight_smile:
alpha-beta.zip (101.3 KB)

8

Thank you. A lot. A concrete example really does help. It’s still a little hard to see what the tradeoffs are between the variants. I had just about concluded that I wanted the “advanced” path, variant2 I think, so maybe I’ll try some experiments with that.

Thank you again! (But I’m sure I’ll be back with more questions.)

I haven’t taken the plunge of converting from Groovy to Kotlin. That’s one more hurdle for the whole team if I do.

1 Like
9

I am struggling rather harder than I expected attempting to translate from Kotlin back into Groovy. I remain seriously tempted to start using Kotlin, but in the meantime, could I trouble you to point me to where I might learn how to translate this into the Groovy syntax of Gradle?

val alphaSourcesBucket by configurations.dependencyScope("alphaSourcesBucket")

dependencies {
    alphaSourcesBucket(project(":alpha", "mainSourceElements"))
}

val alphaSources = configurations.resolvable("alphaSources") {
    extendsFrom(alphaSourcesBucket)
}

val transformSources by tasks.registering(Sync::class) {
    from(alphaSources) {
        filteringCharset = UTF_8.name()
        filter {
            it.replace("\"Alpha Generic\"", "\"Alpha HE\"")
        }
    }
    destinationDir = layout.buildDirectory.dir("transformed-sources").get().asFile
}
10

I’ll just quickly translate it for you if that’s ok for you. :slight_smile:

configurations {
    dependencyScope('alphaSourcesBucket')
}

dependencies {
    alphaSourcesBucket(project(path: ':alpha', configuration: 'mainSourceElements'))
}

def alphaSources = configurations.resolvable('alphaSources') {
    extendsFrom(configurations.alphaSourcesBucket)
}

tasks.register('transformSources', Sync) {
    from(alphaSources) {
        filteringCharset = UTF_8.name()
        filter {
            it.replace('"Alpha Generic"', '"Alpha HE"')
        }
    }
    destinationDir = layout.buildDirectory.dir('transformed-sources').get().asFile
}

sourceSets.main {
    java {
        srcDir(transformSources)
    }
}
11

Thank you. There are definitely some things in there I don’t recognize. I’ll poke at it while I read my new Kotlin book.

Thank you!

12

Feel free to ask if you need some more explanation :slight_smile:

13

My transformSources task isn’t a simple Sync. It runs the sources through a preprocessor.

I start with

def allInputs = files(alphaSources).asFileTree.matching { .include..., .exclude..., ... }.files
inputs.files allInputs
outputs.files fileTree(dir: "${buildDir}/src/hej")

Then I fuss a bit with copying some files around, and then I run the preprocessor:

  doLast {
    javaexec {
      classpath = configurations.preprocessor
      mainClass = "com.igormaznitsa.jcp.JcpPreprocessor"
      args "/c",
        "/i:${buildDir}/filtered/hej", "/o:${buildDir}/src/hej",
        "/p:EE=false", "/p:PE=false"
    }
  }

That seems to work, but downstream tasks object to the sourceSet because it doesn’t contain (a single?) directory. The compileJava task fails with Source directory '/path/to/first/file.java' is not a directory.

I thought I might try to set output.file file("${buildDir}/src/hej") but that’s also an error because it’s a directory.

14

Yeah, you define files as source directory of course. :smiley:
I think what you want is outputs.dir(layout.buildDirectory.dir('src/hey')).

If you would really need the files as ouptuts there, you probably would need on the consumer side a Sync task that copies the files to some directory and then use that sync task as srcDir.

But I think the outputs.dir is more appropriate.

15

#facepalm. Thank you. I did know that.

1 Like
16

I think I’ve worked out the HE/PE/EE projects that are derived from preprocessed Java source files. I like the fact that the standard tasks, like compileJava work correctly in those projects. That feels like I’m doing it better this time :slightly_smiling_face:

I’ve looked at the “beta” example and I can see how

configurations {
  transpiler.extendsFrom(implementation)
  dependencyScope('eeRuntimeOnly')
}

dependencies {
  implementation project(':java')
  eeRuntimeOnly project(":java:ee")
}

gives me access to the ee.jar build from the :java:ee project. New wrinkle: in project gamma, I want access to the source files from :java:ee (the ones that were preprocessed for that target).

Is there a principled way to get access to the outputs of the preprocess task in another project?

17

You should never try to “get the output of a task in another project” directly.
This is unsafe and what is warned about in the warning on Sharing outputs between projects.
(I know you did not try, or assume you didn’t but asked first, just want to make sure you don’t start to try :slight_smile: )

The mentioned page also shows how you properly and safely share things from one project to another project.

Actually, if you want all the sources that :java:ee has (in case it is not only the preprocessed sources but also “hard-coded” ones) you are lucky, as these are already setup properly and you just need to get it by using according attributes. Those are already published as the JaCoCo report aggregation plugin needs the sources from the other projects to create the aggregated JaCoCo report properly.

So you would either use a resolvable configuration, or an artifact view with variant reselection.
And you would request the attribute “category” to be “verification” and “verification type” to be “main-sources”.

You can call gw :java:ee:outgoingVariants to see all the currently defined outgoing variants of the project and will there find the mainSourceElements variant that contains all the sources of that module.

If you do not want all sources of ee, but only the output of the preprocessing, you would need to publish an additional variant accordingly where only that tasks output is published.

18

Okay. I tried reading the “sharing outputs” chapter several more times. I attempted to create an eeSources configuration in the :java:ee subproject to identify the sources. That eventually lead to a clash since it was the same as main-sources. Okay. I guess I don’t need that one.

Now over in the gamma configuration side, I’m trying to work out how to construct a configuration that refers to the main-sources output of the :java:ee project. (I am, in this case anyway, content to get all the sources.)

The sharing outputs chapter uses an example where an “instrumented-jars” name is added as a LIBRARY_ELEMENTS_ATTRIBUTE. Do I need to add some sort of custom attribute so that I can refer to that name on the consumer side? Or am I being clueless again?

19

It turns out that this works, I wonder if this is the right thing? In gamma:

configurations {
  dependencyScope('eeSourcesBucket')
}

dependencies {
  eeSourcesBucket(project(path: ':java:ee', configuration: 'mainSourceElements'))
}

def eeSources = configurations.resolvable('javaSources') {
    extendsFrom(configurations.eeSourcesBucket)
}

tasks.register("helloWorld") {
  doLast {
    println("Hello, world")
    files(eeSources).each { path ->
      println(path)
    }
  }
}

The helloWorld task prints the three directories. Is it safe to traverse them with fileTree to find the actual sources for subsequent processing?

20

That’s one option, yes.
Well, at least almost.
You miss to declare eeSources as input for helloWorld.
This way you for example miss the dependency to the task transforming the sources.
From alpha-beta example an added gamma with three variants of how to do it (sorry it is in Kotlin again, but I think you get the idea):

plugins {
    java
}

val eeSourcesBucket by configurations.dependencyScope("eeSourcesBucket")
val eeSourcesBucket2 by configurations.dependencyScope("eeSourcesBucket2")

repositories {
    mavenCentral()
}

dependencies {
    implementation("commons-io:commons-io:+")
    implementation(project(":alpha:ee"))
    eeSourcesBucket(project(":alpha:ee"))
    eeSourcesBucket2(project(path = ":alpha:ee", configuration = "mainSourceElements"))
}

val eeSources by configurations.resolvable("eeSources") {
    extendsFrom(eeSourcesBucket)
    attributes {
        attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.VERIFICATION))
        attribute(Bundling.BUNDLING_ATTRIBUTE, objects.named(Bundling.EXTERNAL))
        attribute(VerificationType.VERIFICATION_TYPE_ATTRIBUTE, objects.named(VerificationType.MAIN_SOURCES))
    }
}

val eeSources2 by configurations.resolvable("eeSources2") {
    extendsFrom(eeSourcesBucket2)
}

val copyEeSources by tasks.registering(Sync::class) {
    from(eeSources)
    into(temporaryDir)
}

val copyEeSources2 by tasks.registering(Sync::class) {
    from(eeSources2)
    into(temporaryDir)
}

val copyEeSources3 by tasks.registering(Sync::class) {
    from(configurations.runtimeClasspath.get().incoming.artifactView {
        withVariantReselection()
        attributes {
            attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.VERIFICATION))
            attribute(Bundling.BUNDLING_ATTRIBUTE, objects.named(Bundling.EXTERNAL))
            attribute(VerificationType.VERIFICATION_TYPE_ATTRIBUTE, objects.named(VerificationType.MAIN_SOURCES))
        }
    }.files)
    into(temporaryDir)
}

val copyAllSources by tasks.registering {
    dependsOn(tasks.withType<Sync>())
}
  1. eeSourcesBucket / eeSources / copyEeSources uses a dedicated configuration and attributes
  2. eeSourcesBucket2 / eeSources2 / copyEeSources2 uses a dedicated configuration and directly requests the configuration, basically what you also did in your last post
  3. copyEeSources3 uses an artifact view with attributes to get the sources for all dependencies that are also in runtimeClasspath and have any