Gradle-1.0-milestone-8a doesn't seem to be honoring changingPattern in ivy resolvers


(swaters) #1

I define ivy resolvers directly still since the gradle ivy DSL does not yet have the changing* properties for ivy resolvers. It seems that with gradle-1.0-milestone-8 gradle has stopped honoring the changingPattern though. For instance, if I define a local filesystem resolver like so:

add(new org.apache.ivy.plugins.resolver.FileSystemResolver()) {

name = ‘localRepository’

changingPattern = ‘.*-SNAPSHOT’

checksums = “sha1,md5”

addArtifactPattern “${System.properties[‘user.home’]}/.ivy2/local-repo/$artifactPattern”

addIvyPattern “${System.properties[‘user.home’]}/.ivy2/local-repo/$ivyPattern”

}

even if I set

project.buildscript.configurations.all {

resolutionStrategy.cacheChangingModulesFor 0, ‘minutes’

}

Gradle will never download a new jar with a -SNAPSHOT version. I have verified that if I use dynamic versions (i.e. latest.integration) and set the resolution strategy for dynamic versions it does work.

Is there some way to communicate to gradle that it should honor the changing pattern of the ivy resolvers with the new resolution code?


(Luke Daley) #2

Hi Sean,

Can you confirm that this worked for ‘1.0-milestone-7’?


(swaters) #3

I know it is working with milestone-6 and before, I will try it with milestone-7 and report back.


(swaters) #4

I have the same problem with milestone-7. It seems to handle dynamic versions just fine, but not changingVersions.


(Luke Daley) #5

Thanks for taking the time to find that out. That will make it easier to track down.


(swaters) #6

No problem, let me know if there is anything else I can provide to help out, I’d really like to upgrade to take advantage of the offline mode and faster resolution in general.


#7

Try adding: ‘changingMacher = ‘regexp’’ to your resolver definition. We have a test that this works with a URLResolver, and it should work for FileSystemResolver as well.

On a separate note, is there any reason (besides SNAPSHOT support) that you can’t switch to using a built-in ivy repository?


(swaters) #8

Unfortunately, no that does not seem to fix the problem. I had tried that when I first discovered the problem and just tried it again setting both changingMather and changeMatcherName (just to be sure since the variable on the resolver is changingMatcherName). It did not pick up a new artifact in either case when a newer one was available. Based on the ivy documentation the default matcher name should be exactOrRegexp, so I wouldn’t expect setting the changingMatcher would effect this. FWIW, I did also have resolutionStrategy.cacheChangingModulesFor set to 0 seconds.

To your second question, at this point I the gradle ivy DSL does not provide support for changingPattern, changingMatcher (which I don’t need, but I’m sure some people who use changingPattern use that field) and I don’t think it has support for the m2compatible flag which I do use as well. Once those are present, I would be happy to switch over to the gradle Ivy declarations.


#9

This is working for me (with changingMatcher set), but you’ll probably need to run once with ‘–refresh=dependencies’ for it to work.

This is because after first finding the module (without the changing matcher specified) Gradle caches the module descriptor as NOT changing in the repository.

So there are 2 minor bugs at play here: 1) Gradle requires changingMatcher to be specified for a changingPattern to be recognised on an ivy DependencyResolver 2) Gradle continues to use cached values for a DependencyResolver after the changingPattern/changingMatcher configuration for that resolver has been changed.

Can you please check if it works for you with ‘changingMatcher=‘regexp’’ and running once with ‘–refresh=dependencies’?


(swaters) #10

Interesting, so I don’t know if it changes things, but I am declaring this particular resolver in an init script in init.d. It sets up dependencies for the buildscript classpath (in order to add some plugins to the classpath).

In that scenario, if I add changingMatcher = ‘regexp’ to my localResolver, and run with --refresh dependencies it will pick up the new jar. However, if I push a new copy of the jar and run without --refresh dependencies it will not pick up the new jar.

I will try testing this for dependencies that are not at the buildscript/classpath level, but for me at least I can’t seem to get it to recognize that my resolver is a changing resolver.


(swaters) #11

Breakthrough!

If I change my repo definition from:

add(new org.apache.ivy.plugins.resolver.FileSystemResolver()) {
     name = 'localRepository'
     changingPattern = '.*-SNAPSHOT'
     checksums = "sha1,md5"
     addArtifactPattern "${System.properties['user.home']}/.ivy2/local-repo/$artifactPattern"
     addIvyPattern "${System.properties['user.home']}/.ivy2/local-repo/$ivyPattern"
 }

to:

add(new org.apache.ivy.plugins.resolver.FileSystemResolver()) {
     setName 'localRepository'
     setChangingPattern '.*-SNAPSHOT'
     setChangingMatcher 'regexp'
    setChecksums "sha1,md5"
     addArtifactPattern "${System.properties['user.home']}/.ivy2/local-repo/$artifactPattern"
     addIvyPattern "${System.properties['user.home']}/.ivy2/local-repo/$ivyPattern"
 }

This fixed it. So I guess there are interceptors on the set methods to capture whether it is a changing repo. So when I was setting the instance variables directly gradle wasn’t picking up that it was a changing repo.

Also, just to be sure I tried it without setting changingMatcher and that broke it. So the combination of using setters vs directly setting the instance variables and setting changingMatcher seems to have fixed things.

Thanks for all your help on this guys!


(swaters) #12

Okay, one more note on this issue. While I am now getting new jars as needed, I am also getting new jars when not needed. It seems like setting resolutionStrategy.cacheChangingModulesFor 1, ‘minutes’ every 1 minute it re-downloads the ivy.xml, jar, and hash files. I would have expected that it would just download the hash file or ivy.xml and based on that and the cache determine if it needed to download the jar.

Am I misunderstanding how cacheChangingModulesFor works, or is this another issue?


#13

Two things: 1) I’m happy that you got the FileSystemResolver to function correctly, but it’s a bit confusing. There are no magic interceptors used. Here is the build script we use to test this behaviour:

repositories {
    add(new org.apache.ivy.plugins.resolver.URLResolver()) {
        name = "repo"
        addIvyPattern("http://localhost:${server.port}/repo/[organization]/ivy-[module]-[revision].xml")
        addArtifactPattern("http://localhost:${server.port}/repo/[organization]/[module]-[revision].[ext]")
        changingMatcher = 'regexp'
        changingPattern = '.*SNAPSHOT.*'
    }
}
  1. When you are using an ivy resolver directly you miss out on a lot of Gradle’s caching smarts, like using checksums to avoid downloading things that have not changed. (Although this isn’t really relevant when you’re using a local filesystem resolver, since the cache is bypassed in this case).

The only way to get this functionality would be to switch to an ‘ivy{}’ repository, using ‘changing=‘true’’ on any SNAPSHOT dependencies. (Although this won’t work for transient dependencies).


(swaters) #14

Well…that certainly is weird as to why it isn’t picking up the changing on the filesystem resolver. We are also using URLResolvers in the same way (old-style configuration), the filesystem one is just for local publishing when developing a module that is depended on by another project, but having changing on that resolver is obviously very important.

I would love to use the gradle ivy repository definitions and every release I try them, but for now we are stuck with them it seems. We have a large number of projects and dependencies and it would just a giant pain to go set changing=true on all of the individual dependencies right now.

We have plans to eventually move to dynamic versions rather than changing versions, but due to some other weird environmental issues we are stuck with changing versions for the forseeable future. I also imagine that we will always need the ability to use a changing pattern for local publishes since those will almost certainly follow the maven style SNAPSHOT versioning. Surely I’m not the only one that could get some use out of adding changingPattern to the ivy DSL :slight_smile: Do you know of an existing JIRA for this feature or should I go ahead and create one?

Thanks for all your help, I will definitely keep playing around with this


(detelinyordanov) #15

Hi Sean,

We are experiencing the same issue and are stuck with upgrade to Gradle 1.0-m8a due to this. I have created a sample test case that is meant to verify whether new Ivy repository DSL works, unfortunately this test fails even when setting ‘changing=true’ on the dependency definition. The test case consists of:

  • a ‘helloplugin’ module which is build and uploaded with a SNAPSHOT version - a test module that depends on this plugin using the following dependency definition:
buildscript {
 dependencies {
  classpath (
   [ group: 'org.foo', name: 'helloplugin', version: '1.0-m1-SNAPSHOT' ], { changing = true }
  )
 }
}
  • a test that builds and uploads the helloplugin, then runs the test module to verify it has the latest one on its classpath

If you go and create a JIRA for this I will attach this test there as I’m unable to attach it here.

Regards,

Detelin


(Luke Daley) #16

GRADLE-2128 is the associated issue for this.


#17

Please attach the test case to GRADLE-2128 (ignore the title of the issue as it appears multiple things may be involved here).


(Gradle Jira) #18

A sample test case to verify whether changing module dependencies are refreshed by Gradle when there are new versions on the repository. Requires: - Gradle 1.0-m8 - Ivy http repository using Maven2 layout (see {{remoteRepo}} configuration in {{gradleUserHome/init.gradle)

The test case contains:

  • A {{org.foo:helloplugin}} module using a changing version ({{1.0-m1-SNAPSHOT}} but can be any) - A test module that defines a buildscript dependency to the {{helloplugin}} setting {{changing=true}} - A build script that builds and uploads the {{helloplugin}}, then runs the test module - this is done twice, each time checking whether the test module uses latest {{helloplugin}}

Please let me know if there are any misconfigurations or anything that needs to be changed, I also tried setting {{cacheChangingModulesFor 0, ‘seconds’}} resolution strategy in the test module but it did not seem to help.

Regards,

Detelin


(Gradle Jira) #19

Thanks for the detailed test cases Detelin.

The fundamental issue here is that Gradle currently has no concept of the ‘newest’ version of a changing module. So Gradle is unable to find the ‘newest’ version of a changing module across multiple repositories, which is what you require.

When Gradle resolves a dependency, it iterates over the list of repositories until it finds a match. The first matched module is used, and subsequent repositories are not searched. This is true also for changing modules: we will check in repositories for a version that is different to the currently cached version and use is if we find it, but we will only search until we find it in one of the repositories.

(For dynamic versions we have the concept of the ‘newest’ version, so we look in all repositories and use the newest version available.)

So I think the only current solution to this issue is to not specify both the ‘localRepo’ and ‘remoteRepo’ in your set of buildscript repositories, but instead specify the repository that you wish use to provide the changing module. For example, I can get your tests to pass by removing the part of init.gradle that sets up the buildscript repositories for each project, creating a separate test-local and test-remote project with different buildscript repositories. (This is a pretty naive approach: but it demonstrates the solution clearly).

So in test-remote/build.gradle I have:

buildscript {

configurations.all {

resolutionStrategy.cacheChangingModulesFor 0, ‘seconds’

}

repositories {

add project.repositories[‘remoteRepo’]

}

dependencies {

classpath (

[ group: ‘org.foo’, name: ‘helloplugin’, version: ‘1.0-m1-SNAPSHOT’ ], { changing = true }

)

}

}

I hope this helps to explain the behaviour you are seeing, and points you in the direction of a solution. I understand that diagnosing issues with dependency resolution can be frustrating; I’m adding a bit more logging to help track down these sorts of things in the future.


#20

This is a separate issue that also impacts on the behaviour of changing modules. I’ve created GRADLE-2140 to track this.