Bug in dependency management with Gradle 1.9

Old Forum
1

We recently switched from Gradle 1.4 to Gradle 1.9 and saw some never-seen-before issues in our builds.

Note: We tried using Gradle 1.10 but this bug ([#GRADLE-2987] ivy xml parsing issues related to change of the xml schema to 1.1) was a showstopper for us. We also tried using the recent 1.11 but the issue described below is still present.

After the upgrade to Gradle 1.9, some of our builds (actually a small minority of them) fail with the following type of error message:

  • What went wrong: > Could not resolve all dependencies for configuration ‘:product:cbsl-bo:cbsl-bo-impl:testCompile’. > > Could not download artifact ‘com.calypso.cbsl:cbsl-core-impl:2.0.3-SNAPSHOT:cbsl-core-impl-tests.jar’ >

Artifact ‘com.calypso.cbsl:cbsl-core-impl:2.0.3-SNAPSHOT:cbsl-core-impl-tests.jar’ not found.

Not sure to really understand the syntax of this string (‘com.calypso.cbsl:cbsl-core-impl:2.0.3-SNAPSHOT:cbsl-core-impl-tests.jar’) but it looks like there’s some bug in the logic to compute the dependency id (group, artifact, version, classifier, extension) as the classifier here seems to be “cbsl-core-impl-tests” instead of “tests”.

The weird thing is: the very same branch is built with multiple profiles and only one profile fails. All the other profiles build successfully with basically the exact same build configuration.

have you ever seen this bug ?

Thanks

2

Here’s what our Nexus server contains for this artifact.

cbsl-core-impl-2.0.3-20140218.223909-20-acceptancetest-sources.jar Tue Feb 18 14:39:15 PST 2014 261

cbsl-core-impl-2.0.3-20140218.223909-20-acceptancetest-sources.jar.md5 Tue Feb 18 14:39:15 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20-acceptancetest-sources.jar.sha1 Tue Feb 18 14:39:15 PST 2014 40

cbsl-core-impl-2.0.3-20140218.223909-20-acceptancetest.jar Tue Feb 18 14:39:17 PST 2014 261

cbsl-core-impl-2.0.3-20140218.223909-20-acceptancetest.jar.md5 Tue Feb 18 14:39:17 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20-acceptancetest.jar.sha1 Tue Feb 18 14:39:17 PST 2014 40

cbsl-core-impl-2.0.3-20140218.223909-20-integrationtest-sources.jar Tue Feb 18 14:39:16 PST 2014 261

cbsl-core-impl-2.0.3-20140218.223909-20-integrationtest-sources.jar.md5 Tue Feb 18 14:39:16 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20-integrationtest-sources.jar.sha1 Tue Feb 18 14:39:16 PST 2014 40

cbsl-core-impl-2.0.3-20140218.223909-20-integrationtest.jar Tue Feb 18 14:39:15 PST 2014 261

cbsl-core-impl-2.0.3-20140218.223909-20-integrationtest.jar.md5 Tue Feb 18 14:39:15 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20-integrationtest.jar.sha1 Tue Feb 18 14:39:15 PST 2014 40

cbsl-core-impl-2.0.3-20140218.223909-20-javadoc.jar Tue Feb 18 14:39:13 PST 2014 259640

cbsl-core-impl-2.0.3-20140218.223909-20-javadoc.jar.md5 Tue Feb 18 14:39:13 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20-javadoc.jar.sha1 Tue Feb 18 14:39:13 PST 2014 40

cbsl-core-impl-2.0.3-20140218.223909-20-sources.jar Tue Feb 18 14:39:15 PST 2014 69943

cbsl-core-impl-2.0.3-20140218.223909-20-sources.jar.md5 Tue Feb 18 14:39:16 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20-sources.jar.sha1 Tue Feb 18 14:39:16 PST 2014 40

cbsl-core-impl-2.0.3-20140218.223909-20-test-sources.jar Tue Feb 18 14:39:14 PST 2014 9504

cbsl-core-impl-2.0.3-20140218.223909-20-test-sources.jar.md5 Tue Feb 18 14:39:14 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20-test-sources.jar.sha1 Tue Feb 18 14:39:14 PST 2014 40

cbsl-core-impl-2.0.3-20140218.223909-20-tests.jar Tue Feb 18 14:39:14 PST 2014 15635

cbsl-core-impl-2.0.3-20140218.223909-20-tests.jar.md5 Tue Feb 18 14:39:14 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20-tests.jar.sha1 Tue Feb 18 14:39:14 PST 2014 40

cbsl-core-impl-2.0.3-20140218.223909-20.jar Tue Feb 18 14:39:09 PST 2014 85775

cbsl-core-impl-2.0.3-20140218.223909-20.jar.md5 Tue Feb 18 14:39:09 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20.jar.sha1 Tue Feb 18 14:39:10 PST 2014 40

cbsl-core-impl-2.0.3-20140218.223909-20.pom Tue Feb 18 14:39:10 PST 2014 9169

cbsl-core-impl-2.0.3-20140218.223909-20.pom.md5 Tue Feb 18 14:39:10 PST 2014 32

cbsl-core-impl-2.0.3-20140218.223909-20.pom.sha1 Tue Feb 18 14:39:10 PST 2014 40

3

I have the same issue with another artifact this time.

  • What went wrong: > Could not resolve all dependencies for configuration ‘:modules’. > > Could not download artifact ‘com.calypso:calypso:14.0.0.22.SP2-CP-GLOBAL-NEXTSP-SNAPSHOT:calypso-tests.jar’ >

Artifact ‘com.calypso:calypso:14.0.0.22.SP2-CP-GLOBAL-NEXTSP-SNAPSHOT:calypso-tests.jar’ not found.

It seems that the classifier (which should be ‘tests’) is incorrectly evaluated to ‘calypso-tests’ (that is “<artifact-name>-<classifier>”.

4

The error occurred with a 3rd project and strangely, the previous build was green but it failed at the next run.

So this error seems to be intermittent and not reproducible on every build.

5

This is confirmed. Some builds which work with Gradle 1.9 suddenly fail with this error.

6

Hi Francois. The fact that you’re using SNAPSHOT artifacts, and that this occurs sporadically makes me suspect an issue that occurs when the artifact is in the cache. Perhaps our test coverage is not adequate for cached SNAPSHOT artifacts that have a classifier.

To help track down this issue, we’re going to need some help.

The best thing would be a small, reproducible example build that demonstrates this issue. I would suggest starting with the complete build and seeing if you can get error consistently (ie: clean cache, run build, run build again and see if it fails) Make sure cacheChangingModulesFor is NOT set. Once you have a reproducible failure, you can start removing extra pieces until you have just the code that fails: it’s likely that all you’ll need is the ‘repositories’, ‘configurations’ and ‘dependencies’ blocks. Preferably in a single project build.

If you can’t reproduce the issue, we’ll need to understand how the dependency is being resolved, which means we’ll need dependencies configuration together with the debug logging produced by a failing build. You can use Gist to provide these details.

7

I’ve done a little bit of digging and added a test for resolving SNAPSHOT artifacts with classifiers. Note that this report:

Artifact ‘com.calypso.cbsl:cbsl-core-impl:2.0.3-SNAPSHOT:cbsl-core-impl-tests.jar’ not found.

does not indicate that the classifier is incorrect.

The final part of the string is actually the ‘artifact name’, which in a maven repository is always considered to be ‘$moduleId-$classifier.$extension’. You’ll need to take a look at the debug logging to see what HTTP requests are actually being generated, but I suspect that these are correct.

8

Hi Daz,

Thanks for your reply, I’ll try to create a sample project to reproduce the issue. This could take a while given the amount of customization in our builds. I’ll update the ticket once I get a reproducible build error.

Cheers,

9

Hi,

I’ve been investigating this issue as well at Calypso ( a colleague of Francois ) and I have seen one big difference in the example that I am using. The ivy.xml’s that get generated are quite different. Here is a diff:

7c7
<
 publication="19691231155959"
---
>
 publication="20140220120637"
45c45,51
<
 <dependency org="com.calypso.clearing" name="clearing-cdml-impl" rev="2.9.0-SNAPSHOT" force="true" conf="test->runtime(*),master(*)">
---
>
 <dependency org="com.calypso.clearing" name="clearing-cdml-impl" rev="2.9.0-SNAPSHOT" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)">
>
  <exclude org="com.enterprisedt" module="edtFTPj" name="*" type="*" ext="*" conf="" matcher="exact"/>
>
  <exclude org="org.slf4j" module="jcl-over-slf4j" name="*" type="*" ext="*" conf="" matcher="exact"/>
>
  <exclude org="org.slf4j" module="slf4j-log4j12" name="*" type="*" ext="*" conf="" matcher="exact"/>
>
  <exclude org="org.apache" module="xerces-j" name="*" type="*" ext="*" conf="" matcher="exact"/>
>
 </dependency>
>
 <dependency org="com.calypso.clearing" name="clearing-cdml-api" rev="2.9.0-SNAPSHOT" force="true" conf="test->runtime(*),master(*)">
63a70,75
>
  <exclude org="com.enterprisedt" module="edtFTPj" name="*" type="*" ext="*" conf="" matcher="exact"/>
>
  <exclude org="org.slf4j" module="jcl-over-slf4j" name="*" type="*" ext="*" conf="" matcher="exact"/>
>
  <exclude org="org.slf4j" module="slf4j-log4j12" name="*" type="*" ext="*" conf="" matcher="exact"/>
>
  <exclude org="org.apache" module="xerces-j" name="*" type="*" ext="*" conf="" matcher="exact"/>
>
 </dependency>
>
 <dependency org="com.calypso.clearing" name="clearing-cdml-impl" rev="2.9.0-SNAPSHOT" force="true" conf="test->runtime(*),master(*)">"

Everything right is 1.4 and everything left is 1.9. For some reason, in 1.9, clearing-cdml-impl is not being added as a compile dependency (at least according to this diff).

Does this give any more information? This is really the only “conclusive” difference I could find, and I’m not sure if this actually is affecting the problem but it seems strange.

Thanks,

Steve

10

Hi Daz,

Since your replies, I’ve seen the issue surface and quickly disappear (2 builds fail and turn back to green). As you said, there seems to be something related to the logic for caching snapshots. This would explain why the issue is so elusive.

I just setup a CI job (in debug mode) that runs every 5 mins to try to get more info on this issue.

At first glance, it seems that creating a sample project to reproduce the bug will be tricky.

Let’s see…

11

Actually it turns out that this issue is reproducible on my laptop.

I ran the build in debug and uploaded the build log here.

12

Trying to shrink down the size of the log.

13

OK, I was able to dramatically shrink down the project to reproduce the bug. The final project structure looks like the following:

[16:34:57 francois_ritaly@fritaly-mac:~/svn/DASHBOARD_GRADLE-TROUBLESHOOTING]$ tree
.
├── build.gradle
├── gradle
│   └── wrapper
│  
   ├── gradle-wrapper.jar
│  
   └── gradle-wrapper.properties
├── gradle.properties
├── gradlew
├── gradlew.bat
└── src
    └── main
        └── java
            └── com
                └── calypso
                    └── apps
                        └── dashboard
                            └── LaunchDashboard.java

The build.gradle contains the following:

buildscript {
        dependencies {
                classpath 'com.calypso.gradle.plugins:calypso-gradle-plugins:1.3.18'
        }
}
  apply plugin: 'calypso-module'
  dependencies {
 compile "com.calypso:calypso-appkit:14.1.0.0-SNAPSHOT"
}

As you can see, the build uses some custom plugins we developed.

The build fails with the following error:

[16:40:21 francois_ritaly@fritaly-mac:~/svn/DASHBOARD_GRADLE-TROUBLESHOOTING]$ ./gradlew clean assemble
The ArtifactRepositoryContainer.getResolvers() method has been deprecated and is scheduled to be removed in Gradle 2.0.
The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead.
The Test.testReportDir property has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the Test.getReports().getHtml().getDestination() property instead.
The Test.testResultsDir property has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the Test.getReports().getJunitXml().setDestination() property instead.
Ensuring version is unique across all modules ...
:clean UP-TO-DATE
:compileJava
  FAILURE: Build failed with an exception.
  * What went wrong:
Could not resolve all dependencies for configuration ':compile'.
> Could not download artifact 'com.calypso:calypso-appkit:14.1.0.0-SNAPSHOT:calypso-appkit.jar'
   > Artifact 'com.calypso:calypso-appkit:14.1.0.0-SNAPSHOT:calypso-appkit.jar' not found.
  * Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
  BUILD FAILED
  Total time: 5.928 secs

The debug build log is here.

14

Hi Francois

Thanks for taking the time to simplify this issue. That really helps. The crux of the problem is that the SNAPSHOT module is found in the cache, but the artifact for that module is in the cache as “missing”. The challenge will be to work out how we get into this state:

[UserResolverChain] Attempting to resolve module 'com.calypso:calypso-appkit:14.1.0.0-SNAPSHOT' using repositories [maven, MavenLocal]
[CachingModuleVersionRepository] Found cached version of changing module 'com.calypso#calypso-appkit;14.1.0.0-SNAPSHOT' in 'maven'
[UserResolverChain] Using module 'com.calypso:calypso-appkit:14.1.0.0-SNAPSHOT' from repository 'maven'
  ... and later ...
  [CachingModuleVersionRepository] Detected non-existence of artifact 'com.calypso:calypso-appkit:14.1.0.0-SNAPSHOT:calypso-appkit.jar' in resolver cache

One thing you might notice is that the issue resolves itself after 24 hrs. This would be because after 24 hours Gradle will ignore the “missing” entry in the artifact cache, and check the remote repository once again.

One way that I can see this problem could arise would be that: 1) You have the ‘calypso-appkit’ module published, but delete all of the jars at some point 2) You attempt to resolve this module, which fails. Gradle caches that the jar is missing. 3) You re-publish the jar without updating the pom. (If the pom is updated, then Gradle will check for an updated artifact, so this problem would only occur if the pom stays the same).

Does it seem like that could happen in your situation?

A few questions to help us get to the bottom of this:

  • What do your repository declarations look like? * Are you using ‘cacheChangingModulesFor 0, ‘seconds’’, or some other value? * Does this problem still existing if ‘mavenLocal’ is removed from your list of repositories? * How is the “calypso-appkit” module published? * Do you have a job that purges jar files from your repository?

It could be there is a bug in the hashing algorithm in Gradle caches. But it seems more likely that there is a particular usage scenario that is not handled correctly. Once we work out how the cache gets into this unusable state, we’ll be able to fix it.

cheers Daz

15

Hi Daz and thanks for your comments.

This morning I realized that the “test” project I setup actually wasn’t relevant. I checked the Nexus server I’m using and it turns out that it contains the following:

...
calypso-appkit-14.1.0.0-20140212.093246-3294-sources.jar
Wed Feb 12 10:32:49 CET 2014
1000430
 calypso-appkit-14.1.0.0-20140212.093246-3294-sources.jar.sha1
Wed Feb 12 10:32:49 CET 2014
40
 calypso-appkit-14.1.0.0-20140212.093246-3294.jar
Wed Feb 12 10:32:46 CET 2014
2008823
 calypso-appkit-14.1.0.0-20140212.093246-3294.jar.sha1
Wed Feb 12 10:32:46 CET 2014
40
 calypso-appkit-14.1.0.0-20140212.093246-3294.pom
Wed Feb 12 10:32:47 CET 2014
22136
 calypso-appkit-14.1.0.0-20140212.093246-3294.pom.sha1
Wed Feb 12 10:32:47 CET 2014
40
 calypso-appkit-14.1.0.0-20140305.090118-3611.pom
Wed Mar 05 10:01:19 CET 2014
22340
 calypso-appkit-14.1.0.0-20140305.090118-3611.pom.sha1
Wed Mar 05 10:01:19 CET 2014
40

As you can see, the latest snapshot (TS: 20140305.090118-3611) only defines a POM and not JAR file hence the error I consistenly got on my laptop. Sorry I should have checked that before. In fact I’m using a local Nexus mirror which could explain why the POM was only fetched from the master server. This would be a Nexus bug in this case.

However the exact same issue happens too in San Francisco on our Jenkins CI server where artifacts are fetched straight from the Nexus master. But in this case, as said before, the issue is very elusive: 2 builds fail and the next one is green.

I’ll try to reproduce the issue when fetching snapshots from the master Nexus server and come back with elements for you.

Thanks and sorry again for the inconvenience.

16

How are these ivy files being generated? Are you using ‘uploadArchives’? ‘ivy-publish’? Something else?

Can you post a more complete snippet of the ivy.xml generated with 1.4 and with 1.9? Can you also try with 1.11?