Limitations of Gradle source dependencies

[ppalaga (Peter Palaga)]

Hi, I am the author of srcdeps, a source dependencies implementation primarily for Maven [1] but I also created rather a hacky PoC for Gradle [2]. Having seen your announcement [3] I am experimenting with your implementations and try to figure our where its limitations are. Here are my questions:

(1) There is no way to depend on a specific commit sha1, right? Are there any plans to add this?

(2) Is there a way to depend on a Maven/Ant/sbt project?

(3) Knowing that source dependencies use composite builds under the hood, I am trying to figure out if the use of the artifacts produced by source dependencies is somehow limited. E.g. if I can include the dependency jar into a war. Or if I can get the jar and unzip it. I tried something like the following:

// settings.gradle
rootProject.name = 'gradle-source-dependencies-demo'
sourceControl {
    gitRepository("file:///home/ppalaga/orgs/srcdeps/demos/181011-hello-gradle/.git") {
        producesModule("org.srcdeps.hello:hello-gradle")
    }
}

// build.gradle
repositories {
    mavenLocal()
    mavenCentral()
}

apply plugin: 'java'
apply plugin: 'maven'

dependencies {
    implementation('org.srcdeps.hello:hello-gradle') {
        version {
            branch = 'master'
        }
    }
    testCompile 'junit:junit:4.12'
}

sourceCompatibility = 1.8
group = 'org.srcdeps.gradle.plugin.quickstarts.srcdeps-gradle-dep-gradle-git-revision-quickstart'
version = '0.0.1-SNAPSHOT'

def unzippedDir = "$buildDir/dependencies/unzippedDir"

task unzip(type: Copy) {
    configurations.implementation.asFileTree.each {
        from(zipTree(it))
    }
    into unzippedDir
}

build.dependsOn unzip

But I got this error:

...
Caused by: java.lang.IllegalStateException: Resolving configuration 'implementation' directly is not allowed
        at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.assertResolvingAllowed(DefaultConfiguration.java:1060)
        at org.gradle.api.internal.artifacts.configurations.DefaultConfiguration.access$1500(DefaultConfiguration.java:123)

Is there a way to get the jar somehow? Or is this not possible due to how composite build works?

Thanks,

Peter

[1] https://github.com/srcdeps/srcdeps-maven
[2] https://github.com/srcdeps/srcdeps-gradle-plugin
[3] https://blog.gradle.org/introducing-source-dependencies

[ppalaga (Peter Palaga)]

(4) Is there a way to define in the dependent project which gradle tasks should (or should not) be invoked to build the dependency?

[Daniel_L (Daniel Lacasse)]

Thanks @ppalaga for those great question and your contribution to the community with a solution for source dependencies. Here are the answer to your questions:

(1) There is no way to depend on a specific commit sha1, right? Are there any plans to add this?

At the moment, this is not possible. I created an issue to track this feature. It should be quite easy to add as a feature. What requires more thinking is how such feature will look in the user-facing DSL.

(2) Is there a way to depend on a Maven/Ant/sbt project?

You can add dependencies to any project Maven/Ant/sbt/etc. However, you need to provide a Settings plugin that bridges the project to Gradle. This sample shows how it can be achieved in the native world. The same applies to the JVM world.

(3) Knowing that source dependencies use composite builds under the hood, I am trying to figure out if the use of the artifacts produced by source dependencies is somehow limited. E.g. if I can include the dependency jar into a war. Or if I can get the jar and unzip it.

The implementation configuration cannot be resolved. For this purpose, you need to use either compileClasspath or runtimeClasspath depending on your intention. See this section of the user guide. The code would become:

task unzip(type: Copy) {
    from({ configurations.compileClasspath.asFileTree.collect {zipTree(it) } })
    into unzippedDir
}

(4) Is there a way to define in the dependent project which gradle tasks should (or should not) be invoked to build the dependency?

The Gradle tasks invoked to build the dependency are the one used when declaring the “exported” artifact. In the case of a Java project, the jar task and any of its dependencies will be executed. Any included build features should also work as expected.

Don’t hesitate to ask more questions,

Daniel

[ppalaga (Peter Palaga)]

Thanks for your answer, @Daniel_L.

(1) Great!

(2) Interesting, will try to find some time to experiment.

(3) Thanks, compileClasspath works.

(4) Is there a way to define in the dependent project which gradle tasks should (or should not) be invoked to build the dependency?

The Gradle tasks invoked to build the dependency are the one used when declaring the “exported” artifact.

What is “exported” artifact. Is it something the dependent project has to declare explicitly?

In the case of a Java project, the jar task and any of its dependencies will be executed. Any included build features should also work as expected.

So if jar depends on running a long test suite, there is no way to avoid running the tests when building the source dependency?

Thanks again,

Peter

[PrashamTrivedi]

Not sure if it considered as a limitation or a question, but I would like to point it out.

How can I pass credentials to a private Git repository when defining it in source control?

As far as I have seen the docs, I can’t find anything which mentions how can I pass credentials to gitRepository url.

My Usecase: I am trying to build a dependency from a repository which is not Open source, but My user has rights to read that repository.

[sterling (Sterling Greene)]

@ppalaga

So if jar depends on running a long test suite, there is no way to avoid running the tests when building the source dependency?

This is correct, but builds wouldn’t normally do this and I would recommend people to not do this. Regular multi-project builds behave in the same way.

@PrashamTrivedi

How can I pass credentials to a private Git repository when defining it in source control?

At the moment, this isn’t really configurable. How do you access the private repo now? If it’s with SSH, I think this may work out of the box.

[PrashamTrivedi]

At the moment, this isn’t really configurable. How do you access the private repo now? If it’s with SSH, I think this may work out of the box.

At this point I am using HTTPs login. I will try SSH. But as a library developer I can’t force everyone to go on SSH route.

Later on I have discovered another Use case for Using Source Dependency.

I can use source dependency as a mechanism for distributing my private libraries.

Do you think this is a valid use case for that? Is it ok if I file an issue to allow passing credentials to private repository?

[ppalaga (Peter Palaga)]

as a library developer I can’t force everyone to go on SSH route.

You cannot, but I think it is still still the best way to do that both for you and for the users or your lib.

I can use source dependency as a mechanism for distributing my private libraries. Do you think this is a valid use case for that?

It can work, but note that source dependencies work well only inside a Gradle build. They do not work well when there is a need to release the dependent project into a public Maven repository such as Maven Central. I hapen to have a picture for that. In the picture your user’s project is dependent-a and your lib is dependency-2.3.4-srcdeps. If dependent-a is released to a Maven repo, the dependency-2.3.4-srcdeps is not resolvable for any dependent-b that depends on your user’s project is dependent-a.

Source of the picture: srcdeps

[mkulak (Misha)]

At the moment, this isn’t really configurable. How do you access the private repo now? If it’s with SSH, I think this may work out of the box.

It doesn’t. If I do this:

sourceControl {
    gitRepository("git@my.gitserver:user/project.git") {
        producesModule("org.example:project")
    }
}

then it fails with:
org.eclipse.jgit.errors.NoRemoteRepositoryException: file:///Users/myuser/work/git@my.gitserver:user/project.git: not found.

[sterling (Sterling Greene)]

You have to translate this into a ssh:// URL: https://github.com/gradle/gradle-native/issues/839

This is a known thing we need to fix.

[mkulak (Misha)]

@sterling thanks for suggestion.
I changed it to gitRepository("ssh://git@my.gitserver/user/project.git") and it still doesn’t work. But now I have different exception:

Caused by: com.jcraft.jsch.JSchException: USERAUTH fail
        at com.jcraft.jsch.UserAuthPublicKey.start(UserAuthPublicKey.java:119)
        at com.jcraft.jsch.Session.connect(Session.java:470)

At the same time my git clone works perfectly. How can I tell jgit to get key from ~/.ssh/id_rsa?

[tasomaniac (Said Tahsin Dane)]

@mkulak I created an issue on this at Gradle repo https://github.com/gradle/gradle/issues/8245

I am at the same step as you were in October. I have no idea how I can provide so that it would use my local id_rsa. Did you find a solution?

[stewartbryson]

From the blog post: “Now, when Gradle needs to find a version of the “utilities” library, it will look for a matching tag in the Git repository.”

I’ve always created tags that look like v1.1.0 (notice the ‘v’). Am I right to assume that for repositories that work with source dependencies, I would need to start tagging instead with 1.1.0?

[tasomaniac (Said Tahsin Dane)]

I understand the tag becomes the version. So you may not have to do that. I would try with

implementation 'com.mygroup:library:v1.1.0

[ArtemYemelin]

Hi all,

Gradle downloads external repo into the folder like
“.gradle\vcs-1\EXTERNAL_REPO_3gn1livbki1bt2w83rtyfejww”
which is not in the sub-project folder but in the root.
For the future plugin development it would be great that the dependent module will be copied into “source-dependency” folder) under “build” folder of the sub-project (or it could be configured) and instead of commit hash in the folder name a tag name will be used.
In that case it would have been convenient to configure and to add it to project’s “sourceSets” config.

Or maybe there is a way to configure it somehow event now?

[ddimitrov (Dimitar Dimitrov)]

Ideally, the download location should be in the Gradle user home, right next to the artifact cache and cached wrapper distributions. It should also be subject to cleanup policies.

Also, may be worth making it clear that source dependencies must not be edited manually (this allows to share them). If one needs to edit the external code they can check it out in their preferred location and set up a composite build.

[ArtemYemelin]

Sounds good, the only problem for now is that IDEs do not support it properly.
Gradle builds the Jar with all dependencies inside and it runs properly but within an editor all code from source dependency repo is in red

[aanno (Thomas Pasch)]

Interesting feature. I wonder how it could archive ‘right’ dependency resolution as it has been implemented based on composite builds.

I tried the resolution directly on composite builds, but run into problems.

[wdb (ob)]

Do you have an update how we can get SSH to work with private repository for source dependencies?

[gladed (Glade Diviney)]

I’m using:

sourceControl {
    gitRepository('ssh://git@{GIT_HOST]/[ORG]/[REPO].git') {
        producesModule('[GROUP]:[NAME]')
    }
}

Also be sure there’s an entry in your ~/.ssh/config corresponding to a public key you’ve uploaded to the Git host:

Host [HOST]
  IdentityFile ~/.ssh/[PRIVATE_KEY_FILE]
  HostName [HOST]
  User git