Concurrent builds and corrupted ivy.xml in cache


(Kenny Stridh) #1

Hello, in our continuous integration environment there are several jobs running in parallel. Now and then we get an error in a build

  • What went wrong: A problem occurred evaluating root project ‘e’. > Could not resolve all dependencies for configuration ‘:runtime’.

    Could not resolve group:a.b.c, module:d version:1.0.0-SNAPSHOT.

Required by:

a.b.c.e:unspecified

java.text.ParseException: [[Fatal Error] 2b319a4946a8f9d8b7afd88d835b8d65.ivy.xml:1:1: Premature end of file. in file:/local/.gradle/caches/artifacts-13/module-metadata/a.b.c/d/1.0.0-SNAPSHOT/2b319a4946a8f9d8b7afd88d835b8d65.ivy.xml

I believe that this error happens when several gradle instances write the same object in the cache and then try to read while another instance write again. Not sure but it feels like the write lock is abandoned before the read.


(Peter Niederwieser) #2

Which Gradle version?


(Kenny Stridh) #3

Sorry, should have mentioned that before. I’m still running 1.0-rc3 but I couldn’t find anything like this in the release notes for 1.0.

I can change and see if it happens again.


(Peter Niederwieser) #4

rc3 is the same as 1.0 (unless we are talking about m3). Gradle’s artifact cache is designed to be multi-process safe, and it is used heavily in this manner. Of course there could still be a bug. The more information you can provide, the better.


(Kenny Stridh) #5

It’s rc3 not m3, I’ve always try to keep up with new releases although haven’t had time to change to 1.0 yet.

The scenario is like follows: There are five concurrent jobs kicking of at the same time in jenkins, all of them have the same .gradle directory and all of them are dependent on a “snapshot” artifact that change quite often.

The exception happens quite rarely, probably once every thirty invocations. I’ve changed so that we start the jobs with --stacktrace and --debug to get more information.


(Peter Niederwieser) #6

I assume you mean the same $userHome/.gradle, not the same $projectDir/.gradle?


#7

An alternative explanation would be if this was the result of resolving the dependency while the updated SNAPSHOT was being published. Publication is Gradle is not transactional, and will overwrite the existing files in place.

Does that sound possible in your scenario? How is the snapshot being published?


(Kenny Stridh) #8

Peter: It might be that you’re onto something here…

Two of the builds scripts are in the same directory and would therefore share the same $projectDir/.gradle. All builds share the same $userDir/.gradle but it might absolutely be that two build are sharing the same directory.

The problem haven’t happened today so I don’t have a stacktrace for you yet

Daz: The snapshot is published to our internal repository a Nexus installation. I haven’t seen a connection between the publish and the error, but I’ll try to keep that in mind.

Both: Thank you so much for your help so far, I will actually migrate our whole build system to gradle on friday. The change will shorten the feedback loops by over 50% and make the build easier to understand for the average developer here.


(Peter Niederwieser) #9

Two of the builds scripts are in the same directory and would therefore share the same $projectDir/.gradle. All builds share the same $userDir/.gradle but it might absolutely be that two build are sharing the same directory.

Make sure that each CI worker process operates on its own working copy (which should normally be the case). Otherwise, disaster will strike. :slight_smile:


(Kenny Stridh) #10

Damn, I missed that one. So setting --project-cache-dir would to the trick then?

Thanks!


(Peter Niederwieser) #11

You don’t want two builds to run concurrently in the same directory. Even if you separated the project caches, they could still be overwriting each other’s build output. This is better dealt with by configuring the CI server, but I would be surprised if CI workers wouldn’t already use separate working copies.


(Kenny Stridh) #12

Hello, sorry to reopen this case. Got a similar problem, and now in the “global” cache.

java.text.ParseException: [[Fatal Error] b3172ca99dfe876c53bba1d5425310ae.ivy.xml:1:1: Premature end of file. in file:/local/.gradle/caches/artifacts-13/module-metadata/com.ericsson.em/gradle-plugins/1.0.0-SNAPSHOT/b3172ca99dfe876c53bba1d5425310ae.ivy.xml

]

GRADLE_USER_HOME=/local/.gradle

The builds themself doesn’t produce any output, the scripts are there to start several test processes using JavaExec with the correct dependencies.