Gradle offline problems


(Gal Mor) #1

I am working on a gradle project that should run on a machine without internet connection.

For that, I have created a task that takes the gradle cache and copies it into another directory, to be used as a local repository.
For a smaller test project with 2-3 dependencies that task did it’s job and I was able to execute “gradle build --offline” and run the project successfully.

I have built the project in online mode and it worked. I have ran my task to copy the cache to the local repository and it worked. However, when trying to run “gradle build --offline” I am getting errors like this (for several different jar files):

Could not download commons-math3.jar (org.apache.commons:commons-math3:3.4.1): No cached version available for offline mode

Checking both my local repository and the gradle cache, I couldn’t find the relevant jar file (in this case commons-math3-3.4.1.jar).

I have tried using “gradle build --refresh-dependencies”, I have tried to delete the cache and then using “gradle build” - nothing, it wouldn’t download the jar files, only the pom files. I have even tried to download the jars manually and put them in the correct directories in my local repository did not work.

Any help would be appreciated.


(Schalk Cronjé) #2

Gradle will only download the JARs when it actually needs them. It could be that your task executes before Gradle has added the additional artifacts to the cache. As a first diagnosis, run gradle dependencies and check which configuration requires that. Then you could possibly have that task run before yours.

However, having said that an dif I understand your question correctly you are copying the cache as-is. If you try to use that directly as a repository it will not work due the way that Gradle stores the artifacts. You might be able to get it to work by changing the Gradle User Home directory when in offline mode (-g on the command-line), but that will probably end up as a unsatisfactory solution.

You might be better off to use the Ivypot plugin if want everything as a local repository.


(Gal Mor) #3

The order of commands I am using is:
“gradle build” (so as I understand, if gradle needs a jar to complete the build it should download it)
“gradle myTask” (to copy the cache, I have tested that task on a small scale project and that order of operations worked)
“gradle build --offline” (this one fails)

In my top-most gradle.build file, I have these lines inside “buildscript”:

repositories {
if(!gradle.startParameter.isOffline()) {
mavenCentral()
} else {
maven {
url uri("${rootProject.projectDir}/CacheLibraries")
}
}
maven {
url uri("${rootProject.projectDir}/CachedLibraryPlugin")
}
}

where “${rootProject.projectDir}/CacheLibraries” is where my task is copying the cache to.

I am also trying artifactory but currently ran into a different problem. The first solution I’ll get to is probably what I’ll use.


(Schalk Cronjé) #4

Does your offline cache look anything similar to this example?

org.apache.commons/commons-lang3/3.3.2/90a3822c38ec8c996e84c16a3477ef632cbc87a3/commons-lang3-3.3.2.jar
org.apache.commons/commons-lang3/3.3.2/932ed8226f371b204d04a8c4d3d5fe0f2b26339f/commons-lang3-3.3.2.pom

If it does, then it is not a Maven-layout, and it will not work by configuring a maven block to point at its root.


(Gal Mor) #5

The gradle cache looks like that.
My task transforms it to:

org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.jar (same with pom).

My main problem here is that there are several libraries of which I only have the pom file in the gradle cache, so there is no jar to copy and change the hierarchy for.

So I have
org/apache/commons/commons-lang3/3.4.1/commons-lang3-3.4.1.pom
but I don’t have
org/apache/commons/commons-lang3/3.4.1/commons-lang3-3.4.1.jar

And as a result I can’t compile my code offline.
How do I make gradle download it to the cache so it will be copied? As if I am simply downloading the jar myself and places it in my offline repository it also doesn’t work.


(Gal Mor) #6

@Schalk_Cronje I am trying to use the Ivypot plugin, but have encountered two problems.

1 - I have this sort of hierarchy:

RootProject
ProjectA
build.gradle
ProjectB
build.gradle
build.gradle

ProjectB is dependent on ProjectA, so it has “compile project(’:ProjectA)” as a dependency. How can I exclude this from the “syncRemoteRepositories” task (but I would still require ProjectA’s dependencies)?

2 - Adding the “syncRemoteRepositories” to a certain “build.gradle” file results in having only that file dependencies (and their required dependencies). How can I use it so I’ll run “gradle syncRemoteRepositories” only once to have all my dependencies in “syncRepo” (I have plenty of projects, sub-projects, sub-sub project etc, so running it from each one is a mess)?


(Schalk Cronjé) #7

You have just described a use-case which is not directly covered by Ivypot. (please feel free to raise an issue about this at https://github.com/ysb33r/ivypot-gradle-plugin/issues).

IN the interim, I can suggest that you do the following workaround. Add a subproject which applies Ivypot, then add the following in the build script

configurations
rootProject.childProjects.each { String projName, Project p ->
  String configName = "${projName}_deps"
  configurations.create configName
  syncRemoteRepositories.add configName
  p.configurations.each { Configuration c->
   c.dependencies.each { Dependency d ->
    dependencies.add configName, d
   }
  }
}

syncRemoteRepositories {
  repoRoot = "${rootProject.projectDir}/repo"
}

Now you should only need to point your other subprojects at the repo folder in the project root.

if(gradle.startParameter.isOffline) {
  repositories {
    ivy {
      url file("${rootProject.projectDir}/repo").toURI() 
    }
  }
}

I have just typed this off-the-cuff, but hopefully it should work.


(Oleksandr Gavenko) #8

I am working remotely and don’t have connection to company Intranet all the time.

I found any tricks with Gradle caching is total bullshit especially if you have SNAPSHOT dependencies which expire each day ))

Most straightforward solution is to serve local repository, No-cost Artifactory can proxy any repository, providing local cache.

To avoid any edits in project build files I keep off project ~/.gradle/init.d/localArtifactory.gradle:

allprojects {
  buildscript {
      repositories {
          maven { url 'http://localhost:8081/artifactory/all' }
      }
  }
  repositories {
      maven { url 'http://localhost:8081/artifactory/all' }
  }
}

where on /all endpoint I serve virtual repository which is join of many sources.

Just do initial build so all afrifacts are cached by Artifactory and forget about --offline in Gradle ))

My project file has nasty config which I opt-out with environment variable build.gradle:

subprojects {
    configurations {
        all {
            gradle.ext.useRepo && resolutionStrategy {
                cacheChangingModulesFor 0, 'seconds'
                cacheDynamicVersionsFor 0, 'seconds'
            }
        }
    }
}

settings.gradle:

gradle.ext.useRepo = System.getenv()["IGNORE_REPO"] == null && System.properties["IGNORE_REPO"] == null && ! hasProperty("IGNORE_REPO")

Hope that helps you.


(Gal Mor) #9

@gavenkoa, I have tried artifactory, built everything online but then got a strange error while trying to do the same when offline. Can’t remember it at the moment.

@Schalk_Cronje, my project now looks like this:

RootProject
__ProjectA
____build.gradle
__ProjectB
____build.gradle
__CacheSynchronize
____build.gradle
build.gradle

In the “CacheSynchronize” build.gradle file I have added exactly what you have written (well, changed “repo” to “syncRepo”).

Trying to run “gradle :CacheSynchronize:syncRemoteRepository” I am getting the error:

  • What went wrong:
    A problem occurred evaluating project ‘:CacheSynchronize’.

Could not find method add() for arguments [ProjectA_deps] on task ‘:CacheSynchronize:syncRemoteRepositories’ of type org.ysb33r.gradle.ivypot.OfflineRepositorySync.

I also made sure that “ProjectA” build.gradle file contains the following:

repositories {
_if (!gradle.startParameter.isOffline()) {
__mavenCentral()
__maven {
___url “http://repo.spring.io/libs-release
__}
_}
_ivy {
__url “file://${rootProject.projectDir}/syncRepo”
_}
}

subprojects {
_repositories {
__if (!gradle.startParameter.isOffline()) {
___mavenCentral()
___maven {
____url “http://repo.spring.io/libs-release
___}
__}
__ivy {
___url “file://${rootProject.projectDir}/syncRepo”
__}
_}
}

Any Idea what I did wrong?


(Schalk Cronjé) #10

It failed because you followed my instructions and I typed the wrong thing :scream:

This version works.

apply plugin : "org.ysb33r.ivypot"

configurations
rootProject.childProjects.each { String projName, Project p ->
  String configName = "${projName}_deps"
  configurations.create configName
  syncRemoteRepositories.configurations configName // This line is now corrected
  p.configurations.each { Configuration c->
   c.dependencies.each { Dependency d ->
    dependencies.add configName, d
   }
  }
}

syncRemoteRepositories {
  repoRoot = "${rootProject.projectDir}/repo"

  repositories {
    jcenter()  // Remember to tell ivypot which repoS to use
  }
}

(Gal Mor) #11

That one runs, but leaves me with an empty “repo” directory.
Could you walk me through this so I’ll know how it works and try to solve it?

I mean. maybe I’ll find out that I still need to run this from various sub projects, which is a pain, but if it will work it is still better than nothing :slight_smile:

BTW, If you could, I would very much like to talk over hangouts or skype to try and solve it once and for all.


(Schalk Cronjé) #12

Hit me up via Twitter (@ysb33r) and we can sort out a time to chat.


(Gal Mor) #13

@Schalk_Cronje I tried to send you a message but it says I can’t as you are not following me.
If you could send me a message that would be great (@galmor1007).


(Schalk Cronjé) #14

For anyone else who needs to deal with this approach, I have a sample multi-level Gradle example project at https://github.com/ysb33r/SampleProjects/tree/IvyPotGradlePlugin


(Schalk Cronjé) #15

With the 0.6 release of IvyPot all of the above can be simplified for a multi-project to something like:

syncRemoteRepositories {
  repoRoot = "${rootProject.projectDir}/repo"

  repositories {
    jcenter()  // Remember to tell ivypot which repoS to use
  }

  addAllProjects()
}