Download source classifiers for all compile dependencies

Help/Discuss
1

Hi all,

I’m working on a Java project that uses Gradle as the build tool.

I’m using Emacs + jdtls (LSP), not Idea or Eclipse, for semantic information such as completion at point.

I would like to download all the sources for my dependencies, including transitive dependencies. The only advice I can see from searching this forum is several years old (predating vscode usage etc) and suggests enabling sources in the idea or eclipse plugins, but I’m not using an IDE.

In addition, when I tried the idea/eclipse plugins to do this it puts the files into a gradle cache directory, not the maven/ivy directory where the jar files go. I am sharing my maven/ivy repos with other build tools and would like to reuse the sources that have previously been downloaded (it’s also where my tooling expects to find them!)

2

In addition, when I tried the idea/eclipse plugins to do this it puts the files into a gradle cache directory, not the maven/ivy directory where the jar files go.

Not sure what you mean.
Also the “normal” jars are going to the Gradle cache directory and are used directly from there where possible.
So not sure what you mean by “not the maven/ivy directory where the jar files go”.

If you want to get the sources, you should use an artifact view on the compileClasspath configuration where you select the sources variant by attributes. This would give you most sources of most dependencies.

Where it does not work is:

  • dependency does not publish sources
  • dependencies that publish Gradle Module Metadata but do not publish the proper sources variant but just the sources jar as additional manual artifact

The second point is a publishing bug.
For dependencies without Gradle Module Metadata the sources variant is created on-the-fly, but if a dependency publishes Gradle Module Metadata it is expected that it declares the existing variants explicitly, so if it does not but has the sources jar published as “unrelated” additional artifact this is a publishing but and you will miss the sources from there.

Besides those two mentioned cases, such an artifact view should give you all the source jars that you can then copy to wherever you need them for your tooling.

This is also how for example IntelliJ resolves sources for Gradle projects nowadays by injecting an init script to a build execution that resolves the sources that way.

3

Thanks @Vampire .

I have never edited a gradle file, so I’m a bit lost on how I would implement such “an artifact view”, is there some documentation on how I could achieve that.

With regards to the location of the jar files, that is perhaps a tangential issue. I think I’d be happy getting hold of the files without needing eclipse or idea plugins in the first instance, and I can look into exactly where things go in a follow up. I have a lot of jars and sources already downloaded from scala/sbt builds in ~/.cache/coursier and I’d be happy to avoid double downloading everything when changing build tool. But maybe that’s a pipe dream.

4

For example in Kotlin DSL:

val collectSources by tasks.registering(Sync::class) {
    from(
        configurations.compileClasspath.map {
            it.incoming.artifactView {
                withVariantReselection()
                attributes {
                    attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.DOCUMENTATION))
                    attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType.SOURCES))
                    attribute(Bundling.BUNDLING_ATTRIBUTE, objects.named(Bundling.EXTERNAL))
                }
            }.files
        }
    )
    into(layout.buildDirectory.dir("compile-dependency-sources"))
}
5

Converting into regular gradle, I’m putting this in my ~/.gradle/init.gradle

gradle.afterProject { p ->
    def cc = p.configurations.findByName('compileClasspath')
    if (cc != null) {
        p.tasks.register('collectSources', Sync) {
            from(
                cc.incoming.artifactView {
                    withVariantReselection()
                    attributes {
                        attribute Category.CATEGORY_ATTRIBUTE, p.objects.named(Category, Category.DOCUMENTATION)
                        attribute DocsType.DOCS_TYPE_ATTRIBUTE, p.objects.named(DocsType, DocsType.SOURCES)
                        attribute Bundling.BUNDLING_ATTRIBUTE, p.objects.named(Bundling, Bundling.EXTERNAL)
                    }
                }.files
            )
            into p.layout.buildDirectory.dir('compile-dependency-sources')
        }
    }
}

and it puts these files into a <subproject>/build/compile-dependency-sources directory, great!

Is there a way to have these instead put alongside the jars in ~/.gradle/caches? That is where an LSP may expect to find them.

6

oh, wait a minute, it is putting them in the caches directory. Albeit with a uuid folder substructure. I don’t need the final compile-dependency-sources copy of it all though, so I guess I just drop the from and into

7

Converting into regular gradle

That’s not “regular gradle”, it is Groovy DSL.
“regular” I would say is Kotlin DSL as it is the default one since quite some time.

Also, by using findByName you introduce ordering constraints and additionally destroy laziness.

And also it only works for those projects where the Gradle version is new enough to support the APIs you use but will fail for all builds that are older.

If you are fine with all those problems and drawbacks it might be ok.

and it puts these files into a <subproject>/build/compile-dependency-sources directory, great!

If you execute the collectSources task, yes.

Is there a way to have these instead put alongside the jars in ~/.gradle/caches? That is where an LSP may expect to find them.

No.
Well, you can surely program it, but that will pollute the Gradle cache and you really, really should not do that.

oh, wait a minute, it is putting them in the caches directory.

Of course, different projects or working trees could need them. That is where Gradle caches the downloaded files and uses it from if possible. That they are also landing in build/compile-dependency-sources is because you have a Sync task that copies them over from their cache location to that directory.

Albeit with a uuid folder substructure.

No, checksum.

I don’t need the final compile-dependency-sources copy of it all though, so I guess I just drop the from and into

If you do that, nothing will happen, as you have a Sync task that has no inputs and thus will not do any work, neither will the sources jars be downloaded to the cache.

You can probably have some task that just has them as inputs like

        p.tasks.register('collectSources') {
            inputs.files(
                cc.incoming.artifactView {
                    withVariantReselection()
                    attributes {
                        attribute Category.CATEGORY_ATTRIBUTE, p.objects.named(Category, Category.DOCUMENTATION)
                        attribute DocsType.DOCS_TYPE_ATTRIBUTE, p.objects.named(DocsType, DocsType.SOURCES)
                        attribute Bundling.BUNDLING_ATTRIBUTE, p.objects.named(Bundling, Bundling.EXTERNAL)
                    }
                }.files
            )
            outputs.upToDateWhen { true }
            doLast { }
        }

which should trigger the files to be downloaded to the cache I think if having them there is enough for you.
(the last two lines need to stay, I think. Without the doLast line the task might be skipped for not having work without checking the inputs, without the outputs line the task can never be up-to-date.

8

great, thanks!

For completeness, this is what I ended up adding to my init file, and re-naming the task to align with the sbt name for this

gradle.afterProject { p ->
    def cc = p.configurations.findByName('compileClasspath')
    if (cc != null) {
        p.tasks.register('updateClassifiers') {
            inputs.files(
                cc.incoming.artifactView {
                    withVariantReselection()
                    attributes {
//                        attribute Category.CATEGORY_ATTRIBUTE, p.objects.named(Category, Category.DOCUMENTATION)
                        attribute DocsType.DOCS_TYPE_ATTRIBUTE, p.objects.named(DocsType, DocsType.SOURCES)
                        attribute Bundling.BUNDLING_ATTRIBUTE, p.objects.named(Bundling, Bundling.EXTERNAL)
                    }
                }.files
            )
            outputs.upToDateWhen { true }
            doLast { }
        }
    }
}

Now I have all the sources available.

I will probably open another question to ask if it is possible to share caches between gradle and maven / ivy / coursier (sbt).

9

Don’t comment the category, it is one of the most important attributes.

I will probably open another question to ask if it is possible to share caches between gradle and maven / ivy / coursier (sbt).

Probably not.

10

Oh I thought this was disabling javadocs. I’ll put it back in.

If I wanted to download javadocs too how would I adapt this for that purpose?

11

docs type is javadoc for javadocs

1 Like