After failing to locate an artifact in remote Ivy repo, Gradle never finds it on subsequent builds

Old Forum Archive
1

When I add a new dependency to a script, and the library is missing from the remote repository, the build fails as expected. However, if I add the library to the repository, and then re-run the script, the build still fails.

Here’s the script:

apply plugin: 'java'
  dependencies {
 compile(
         [group: 'test', name: 'test', version: '1.0'],
 )
}
  configurations.all {
    resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
}
  project.ext {
 remoteUrl = "http://myivyrepo.com"
     artifactPattern = "[organisation]/[module]/[revision]/[artifact](-[classifier])-[revision](.[ext])"
 ivyPattern = "[organisation]/[module]/[revision]/[artifact](-[classifier])-[revision](.[ext])"
}
  repositories {
 add(new org.apache.ivy.plugins.resolver.URLResolver()) {
  addIvyPattern( remoteUrl + "/" + ivyPattern )
  addArtifactPattern( remoteUrl + "/" + artifactPattern
)
  checkmodified = true
  changingPattern = 'latest'
 }
}

The first time I run this script, the Cobertura library is missing, and so the error is correct. But after I go add the library to our remote Ivy repo, it still produces the following error, even though I’ve confirmed the library is there:

* What went wrong:
Could not resolve all dependencies for configuration ':testRuntime'.
> Could not find group:test, module:test, version:1.0.
  Required by:
      :projectA:unspecified

Running clean doesn’t resolve it, nor does deleting the .gradle directory by hand. And running other projects with similar dependencies all fail. (I.e., copy-and-paste this build script into a completely different project and it will still fail.)

So it appears to be a problem with the cache. If I delete the cache and the re-run the script, it finds the dependency and passes.

Also, I do not have this problem if I use a local repository for resolution. For example, if I do this instead:

project.ext {
 localUrl = "${System.properties['user.home']}/.ivy2/local"
     artifactPattern = "[organisation]/[module]/[revision]/[artifact](-[classifier])-[revision](.[ext])"
 ivyPattern = "[organisation]/[module]/[revision]/[artifact](-[classifier])-[revision](.[ext])"
}
  repositories {
 ivy {
  url localUrl
  ivyPattern ivyPattern
  artifactPattern artifactPattern
 }
}

Now the script fails when the artifact is missing, and as soon as I add it, re-running the build passes.

Any suggestions on this? Is this a bug, or is there something on my Ivy URLResolver that isn’t configured correctly?

PS. This is Gradle 1.1.

2

A little more information. I notice that the failure comes in the ‘:testRuntime’ configuration. If I exclude tests using -x then the build passes.

C:\dev\java\workspace\projectA>gradle -x test build
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:jar UP-TO-DATE
:assemble UP-TO-DATE
:check
:build
  BUILD SUCCESSFUL
  Total time: 2.962 secs

However, if I run “gradle dependencies” I get the error again, and this time in the ‘:compile’ configuration.

C:\dev\java\workspace\projectA>gradle dependencies
:dependencies
  ------------------------------------------------------------
Root project
------------------------------------------------------------
  archives - Configuration for archive artifacts.
No dependencies
  compile - Classpath for compiling the main sources.
  FAILURE: Build failed with an exception.
  * What went wrong:
Execution failed for task ':dependencies'.
> Could not resolve all dependencies for configuration ':compile'.
   > Could not find group:test, module:test, version:1.0.
     Required by:
         :projectA:unspecified

Deleting the cache gives me success:

C:\dev\java\workspace\projectA>gradle dependencies
:dependencies
  ------------------------------------------------------------
Root project
------------------------------------------------------------
  archives - Configuration for archive artifacts.
No dependencies
  compile - Classpath for compiling the main sources.
Download http://ivy.smdev.info/ivy/repo/test/test/1.0/ivy-1.0.xml
\--- test:test:1.0 [default]
  default - Configuration for default artifacts.
\--- test:test:1.0 [default]
  runtime - Classpath for running the compiled main classes.
\--- test:test:1.0 [default]
  testCompile - Classpath for compiling the test sources.
\--- test:test:1.0 [default]
  testRuntime - Classpath for running the compiled test classes.
\--- test:test:1.0 [default]
  BUILD SUCCESSFUL
  Total time: 7.86 secs
3

Gradle caches the fact that an artifact is missing from a repository for 24 hours. You can force Gradle to recheck for missing artifacts by running with the ‘–refresh-dependencies’ option.

In the future we plan to improve Gradle so that it will refresh dependencies automatically before failing the build due to a missing dependency.

4

Ah, thanks. Well, glad it’s a simple fix, at least. I was under the mistaken impression that ‘‘resolutionStrategy.cacheChangingModulesFor’’ would handle that, but I guess it’s not a changing module. :slight_smile:

5

We are using an internal repository for 3rd party libs. Is there a property to set the 24h limit to something like 5 minutes ?

6

See ResolutionStrategy in the build language reference. If you are referring to the caching of missing artifacts, I’m not aware of a way to reconfigure the TTL for that.

7

Thanks, I set those to

configurations.all {
          if (rootProject.ext.isteamcityrun) {
      resolutionStrategy.cacheChangingModulesFor 10, 'minutes'
      resolutionStrategy.cacheDynamicVersionsFor 10, 'minutes'
    }
  }

Would that mean that missing artifacts would still be ‘cached’ for 24 hours ?

8

As far as I know, missing artifacts are always cached for 24 hours. As Daz already said:

In the future we plan to improve Gradle so that it will refresh dependencies automatically before failing the build due to a missing dependency.

9

Is there a way to flush only the missing dependency cache on each run (on our build server) ? Otherwise I don’t know how to get the build stable since builds running on our CI server should only get artifacts from our internal server.

10

No there isn’t. Why do you run into the problem of cached missing artifacts on CI? Typically this only happens rarely, for example when something about the artifact repository is first misconfigured and then corrected (e.g. proxy settings).

11

Our CI builds are not allowed to download artifacts from external servers (like mavenCentral). Instead we store our 3rb party dependencies in SCM and resolve them using an Ivy resolver.

Developers usually have some work-in-progress projects etc. so they add e.g. mavenCentral() as an additional repository.

If someone now adds a dependency to ‘org.foo:bar:1.0’ it may run fine on his local dev machine, but when he submits his change the CI build will have a missing dependency to ‘org.foo:bar:1.0’ and gradle will fail the build. He can now add the ‘org.foo:bar:1.0’ to the local repository but the CI build will stay failed for 24 hours.

12

I see. Ideally, devs would resolve from the same repo, and would make sure that a new lib is resolvable before they commit. If you can’t make this happen, the only workaround that I can think of is to delete caches very often on CI, or to always run with ‘–refresh-dependencies’. That’s not nice, but at least you are always resolving from your internal repo.

As has been mentioned before, this is on our list of things to fix, but I can’t give you an ETA. If you need this to get fixed soon, I encourage you to submit a pull request, or get in touch with Gradleware. Here is how we think this should be fixed: https://github.com/gradle/gradle/blob/master/design-docs/dependency-management-bug-fixes.md#ignore-cached-missing-module-entry-when-module-is-missing-for-all-repositories

13

Ok, thanks for the info.

Can I set the “refresh-dependencies” flag also from the build script ?

14

Yes, just do ‘gradle.startParameter.refreshDependencies = true’.