Use GitHub Releases as dependency repository (Plugin?)


(IsNull) #1

GitHub provides the ability to publish/host (binary) releases. My idea is to use the Github Releases as repository in Gradle. (Travis is able to automatically deploy to Github Releases, so this would give a very smooth workflow.)

This would remove the burden of managing a 3th party maven repository.

I know there is https://jitpack.io, but its not free for private usage and it also seems to fail more complex builds involving 3th party tools. (And why should a repository build things itself, this is the job of the CI server)

Using the Github Rest API to find projects & release versions seems to be a logical step and actually just another form of repository. This would even work with private repositories, where as public maven repos obviously would not be possible.

Would it make sense to write a plugin which provides a new repository ‘github-releases’? I’d like to give this a try. Maybe someone can point out another plugin which implements a custom repo, if this is possible?


(Schalk Cronjé) #2

This is something we as community plugin authors never quite got right so far. For the JRuby plugin we added the option to firing up a little jetty servlet in order to do the Gem -> Maven conversion. That is not a great solutions IMHO, but it had to be done becasue the Gradle API curretnly offers little in the way to help plugin authors to solve this problem.

Then there was the Gradle Repositories Plugin, but it no logner works with later versions of Gradle. You can poke around in there to get some ideas.


(Lance Java) #3

Some things to consider off the top of my head

  1. Most (all?) github repositories only publish binaries as release artifacts, how will you obtain the publishing pom information (eg transitive dependencies)?
  2. Will you be able to deal with non-standard jar naming conventions?
  3. Will you support jars nested within zips?
  4. How will you determine groupId and artifactId? (the github user / repo may be different to the groupId / artifactId)
  5. If a github repo represents a multi-module build you’ll need a mechanism to support multiple artifact id’s per release version
  6. I’m guessing that you’ll need the option to provide custom mappings when github release artifacts don’t conform exactly to conventions. Where will this mapping meta-data live? in the github repo? or in the client’s build.gradle?

Points 4 & 5 sound like they may require a releaseevent webhook to maintain a mapping between github repos and groupId / artifactId


(IsNull) #4

The idea with a local ‘adapter’ maven server sounds interesting as it would not require any changes to Gradle or a Plugin. On the other hand, a plugin which can handle this would probably be more effective.

We need to have basic information of the artefact, especially its transitive dependencies. Thus we still need an ivy.xml or pom.xml as it is used when published to a normal maven/ivy repository.

Since we should not need to checkout the git repository (this would slow down builds especially if many of the dependencies would use this form of repository), we would require a ivy.xml or a pom.xml in the Github Releases, next to the jar.

For multi module build (which is quite important) this would require multiple poms in the Release downloads.

  • foo-client.jar
  • foo-client-sources.jar
  • foo-client.pom.xml
  • foo-server.jar
  • foo-server-sources.jar
  • foo-server.pom.xml

The version would be directly mapped from the Release Tag.

So as an usage example:

    repositories {
        maven {
            url "http://127.0.0.1:8077/github/IsNull/releases"
        }
    }
   dependencies {
      compile("MyRepository:MyModule:1.0.0")
   }

The above would look for a Repository named ‘MyRepository’ for a Release with the Tag 1.0.0 or v1.0.0 and pull in the jar/pom prefixed with MyModule.

Could this work? :smiley:

PS: I think it makes a lot of sense to handle each GitHub User/Organisaiton as a unique repository. Especially if you need to authenticate to them, since different GitHub Users would require different tokens.


(Lance Java) #5

I think you need to step back and think about your primary objectives. I’m guessing the main objective is for something that “just works” for github projects.

The need for pom.xml/ivy.xml is now a custom requirement and will likely require a custom “publish” step. So, the convenience seems to have faded and it now looks like you’re rolling your own repository with it’s own publishing mechanism.

Have you considered using github as a maven repository?


(Schalk Cronjé) #6

If you only want to download files, there might be a simpler solution. Configure an Ivy resolver with a custom pattern. https://gist.github.com/ysb33r/9f95bab338b912c45986


(IsNull) #7

As much as I’d love to see a “just works for Github Release” solution, I cant see how transitive dependencies could be handled by pure magic. They must be specified by some means, and if this happens with an additional file, why not reuse the pom.xml.
This indeed requires an additional step to create the pom, but Gradle seems to be able to just create the publish pom, without actually publishing:

model {
        tasks.generatePomFileForMavenJavaPublication {
            destination = file("$buildDir/libs/${jar.baseName}-pom.xml")
        }
}

In essence, this would reuse a GitHub ‘User’ as Maven repository.

Have you considered using github as a maven repository?

Yes, but pushing binaries into a git repo is not good practice. Due to the nature of git this leads to bloating the repo in no time.

Apparently the Ivy custom Url attempt would be able to handle ivy/pom files, not only binaries. This looks very promising.

A typical Url from the GitHub Releases looks like

https://github.com/MyUser/MyRepository/releases/download/v0.1.2/myArtifact-0.1.2.jar

This would yield a pattern like

repositories {
    ivy {
        url 'https://github.com/MyUser'
        layout ('pattern') {
            artifact '[organisation]/releases/download/v[revision]/[artifact]-[revision](-[classifier])(.[ext])'
        }
    }
}

I think I do some testing. :slight_smile:

Edit:

After some testing I’ve run into the same issue as reported here: gradle-fails-to-download-a-dependency-if-http-head-is-not-supported-by-a-web-server

GitHub does not support HTTP HEAD on Release Urls (which are AFAIK redirected to AWS Cloud). Gradle failes if this is not possible with

Could not HEAD ‘https://github.com/MyUser/MyRepo/releases/download/v0.1.0/my-artifact-0.1.0.jar’. Received status code 403 from server: Forbidden

Oh well but almost :slight_smile:


Gradle fails to download a dependency if HTTP HEAD is not supported by a web server
(Layton Whiteley) #8

im trying to do something similar here.

I tried using github as a maven repo, however the one hick up i see now is using api.github.com

you can get raw content but its done by passing a header instead of just using a path.
auth token is needed as a header as well.

as demonstrated here: https://gist.github.com/Integralist/9482061

not sure if there is a way to set headers with:

maven {
            url "https://api.github.com/repos/{org}/{repository}/contents"
        }



(Schalk Cronjé) #9

I’m curious why you want to publish to Github Releases if you can just publish to Bintray, which is far better suited to the task.