Gradle and git dependencies


(Marcel Overdijk) #1

Hi,

I wonder if Gradle could support git dependencies like Bundler (Ruby) is also supporting, e.g.:

gem "nokogiri", :git => "git://github.com/tenderlove/nokogiri.git", :branch => "1.4"
  1. Is this something Gradle would consider supporting in the future (or does it already)?

  2. Does anybody have any pointers how to start implementing this myself?

Use case for this would be to define dependency to e.g. javascript/css/image resources.

Something like e.g.:

dependencies {
     compile git: "git://github.com/someproject/someproject.git", branch: "1.4"
}

Regards,

Marcel


(Marcel Overdijk) #2

Bump, any suggestions are welcome…


(Peter Niederwieser) #3

Source (control) dependencies aren’t currently supported, but have been discussed and might be supported in the future. To implement this yourself, you could write a ‘git’ method (‘dependencies { compile git(…) } })’ that returns a lazy ‘FileCollection’.


(Marcel Overdijk) #4

Thanks Peter.

Good to hear that this feature is being discussed. I’m currently also looking into https://github.com/ajoberstar/gradle-git which provides some convienient tasks for cloning GitHub repositories etc.


(Marcel Overdijk) #5

I’m experimenting with a custom

class GitResolvingDependency extends AbstractDependency implements SelfResolvingDependency {

which contains the resolve methods as declared in SelfResolvingDependency.

However I wonder when these methods are called as the code inside does not seem to be called during dependency resolving. Not when executing gradle dependencies, but also not in a custom task poking with configurations.compile.incoming and similar methods.

For reference I’m just posting my code:

dependencies {
    compile files("a.jar", "b.jar")
    compile git(url: "git://github.com/test/test", branch: "master")
}
  Dependency git(Map<String, ?> notation) {
    return new GitResolvingDependency(url: notation.url, branch: notation.branch)
}
  task doit << {
    println ">> configurations.compile.artifacts"
    configurations.compile.artifacts.each {
        println it.properties
    }
    println ">> configurations.compile.dependencies"
    configurations.compile.dependencies.each {
        println it.properties
    }
    println ">> configurations.compile.incoming.files"
    configurations.compile.incoming.files.each {
        println it.properties
    }
    println ">> configurations.compile.resolvedConfiguration.resolvedArtifacts"
    configurations.compile.resolvedConfiguration.resolvedArtifacts.each {
        println it.properties
    }
}
  class GitResolvingDependency extends AbstractDependency implements SelfResolvingDependency {
      String url
    String branch
    String tag
    String ref
      public boolean contentEquals(Dependency dependency) {
        if (!(dependency instanceof GitResolvingDependency)) {
            return false;
        }
        GitResolvingDependency that = (GitResolvingDependency) dependency;
        return this.url == that.url && this.branch == that.branch && this.tag == that.tag && this.ref == that.ref
    }
      public Dependency copy() {
        return new GitResolvingDependency(url: this.url, branch: this.branch, tag: this.tag, ref: this.ref);
    }
      public String getGroup() {
        return null
    }
      public String getName() {
        return "unspecified"
    }
      public String getVersion() {
        return null
    }
          public TaskDependency getBuildDependencies() {
        // TODO clone, build, do whatever needs to be done
        return new DefaultTaskDependency()
    }
      public Set<File> resolve() {
        println "MARCEL >> resolve called"
        return resolve(true)
    }
      public Set<File> resolve(boolean transitive) {
        println "MARCEL >> resolve called"
        return project.files("c.jar").files
    }
}

What I basically need to do is clone the git repo, build the code (C Makefile), package it up to temp folder, use this packaged up distribution as part of dependency. Am I right these tasks should be configured in getBuildDependencies() ?

I’m trying to understand the Gradle internals and looking for advise.

Any help is appreciated, Marcel


#6

Unfortunately, implementing ‘SelfResolvingDependency’ does not work as advertised in the docs: this part of the API is a relic of the very early iterations of Gradle and has not functioned in this way since v0.9. We should probably update the docs and deprecate ‘SelfResolvingDependency’ to reflect the current status.

If you want to continue with your experiment, you’ll probably want to implement ‘org.gradle.api.internal.artifacts.ResolvableDependency’, by overriding the ‘resolve(DependencyResolveContext)’ method on ‘AbstractDependency’.

I’m not sure that this will work, but that method is certainly called when the dependency is resolved. You’re delving pretty deep into internal classes in doing this; you need to be prepared for classes to change without notice between versions.

Longer term we plan to make it easier to provide your own ‘repository’ implementations to support your own protocols like this. But for now, there are not really any convenient extension points.

To be honest, you’re probably better off trying for Peter’s suggestion of returning a FileCollection from your ‘git’ method.


(Marcel Overdijk) #7

Thanks for helping out Daz.

I understand using the internal classes is at my own risk, which I would like to avoid if possible.

I only wonder how I can use a FileCollection to achieve what I want.

E.g.

dependencies {
    compile git(url: "some git url", branch: "some branch")
}
  def git(Map<String, ?> notation) {
    // notation would contains something like url (required) and branch|tag|ref (optional)
    FileCollection files = // to what should the file collection point to?
    return files
}

I’m open for any suggestion.

To explain the rationale behind this is that we are creating a PoC to use Gradle in a huge C/C++ environment. Main goal for Gradle is to the resolve dependencies between projects; the building itself is done using makefiles. Most dependencies will be retrieved from a repository manager like Artifactory (along with their ivy.xml for transitive dependencies) but some modules do not have published modules (for various reasons) and therefor we would like to retrieve the sources directly from a git repository, just like Bnndler (Ruby) supports.

Thanks, Marcel


#8

Check out the documentation for Project.files. Note that a Closure or Callable passed to the method will be lazily evaluated.

dependencies {
    compile git(url: "some git url", branch: "some branch")
}
def git(Map<String, ?> notation) {
    // notation would contains something like url (required) and branch|tag|ref (optional)
    return project.files({
         // This closure will be evaluated every time the file collection is resolved.
         // Return the artifact files required
    })
}

(Marcel Overdijk) #9

But I guess you assume the my project already contains the files from git repo?

As part of the resolving I would like to download the sources from git repo.


#10

Nope, you could do that inside the closure, calling out to whatever methods you like. You’d obviously need to check if the files were already downloaded in this invocation to avoid calling ‘git’ multiple times.

The fact that the closure is lazily evaluated is what’s important here. The code won’t be called until the dependencies are resolved. So you can do any work required at that time.


(Marcel Overdijk) #11

OK thanks, that suits well for the moment. I ended up indeed using the closure and cloning my repo from there.


(Marcel Overdijk) #12

Hi @Daz,

Would it be possible to provide this custom git method via a plugin/extension? I found out that I can call extension method like myplugin.git(…) But I’m wondering if there is a way to remove the ‘myplugin’ part? Only git(…) looks much better.


(edgurgel+gradle) #13

Where’s the code? There are more people interested :slight_smile:


(Marcel Overdijk) #14

@Daz,

I experienced using the “files” approach but this is not sufficient unfortunately.

I managed to clone a git repo etc. but basically I also need to include transitive dependencies of my “source repository dependency”.

I moved forward creating a GitDependency by extending AbstractDependency (which implements ResolvableDependency as you mentioned).

Now I was thinking the following to do inside the resolve() method: 1. clone repo (done) 2. build the cloned repo (done) 3. publish the build in a local file repo (done) 4. add this local build to the context

Is it possible to add this local build to the context? And in such a way that it will resolve it’s transitive dependencies as well.

Hope this is a little bit clear…

Thanks, Marcel


(nm2501) #15

Marcel, have a look at the Holy Gradle Plugins on BitBucket. Several plugins are coupled to Windows but they’re objective is to support retrieving source dependencies from source control (currently SVN and Hg), dealing with large c/c++ projects, packaging up large numbers of artifacts (I.e. not just one jar, but headers, libs, dlls, pdbs in separate zips).

It doesn’t support cloning Git but it should be easy to add.