Artifact only dependency notation still resolves pom/ivy.xml and does so across repositories


(swaters) #1

I have a potentially corner-case dependency issue that came up when I upgraded to 1.0-milestone-8a.

For the sake of simplicity let’s say I have 2 repositories I am resolving against:

// internally published 3rd party artifacts (generally not available in public maven repo)
            ivy {
                name 'thirdPartyMavenRepository'
                url "http://internal.artifactory/artifactory/thirdparty-maven/"
                layout 'maven'
            }
              // internal cache of the relevant public maven repo
            maven {
                name "RemoteRepos"
                url "http://internal.artifactory/artifactory/remote-repos/"
            }

I have a self-published artifact in the thirdPartyMavenRepository with no pom or ivy.xml. It is the activemq binary distribution and I have it published as org.apache.activemq:apache-activemq:5.5.0:bin@tar.gz so it is available at http://internal.artifactory/artifactory/thirdparty-maven/org/apache/activemq/apache-activemq/5.5.0/apache-activemq-bin.tar.gz

In the remoteRepos repository there is a pom published under org.apache.activemq:apache-activemq:5.5.0 from the public maven repo. In gradle 1.0-milestone-8a if I add a dependency of compile( “org.apache.activemq:apache-activemq:5.5.0:bin@tar.gz” ) The build resolves the tar.gz from thirdPartyMavenRepository, but then it also resolves the pom in RemoteRepos.

Since the bin classifier and the tar.gz are not in the pom, dependency resolution fails:

Resource missing. [HTTP GET: http://internal.artifactory/artifactory/thirdparty-maven/org/apache/activemq/apache-activemq/5.5.0/ivy-5.5.0.xml]
Resource found. [HTTP HEAD: http://internal.artifactory/artifactory/thirdparty-maven/org/apache/activemq/apache-activemq/5.5.0/apache-activemq-5.5.0-bin.tar.gz]
Checksum SHA-1 matched cached resource: [HTTP GET: http://internal.artifactory/artifactory/remote-repos/org/apache/activemq/apache-activemq/5.5.0/apache-activemq-5.5.0.pom.sha1]
Resource missing. [HTTP GET: http://internal.artifactory/artifactory/thirdparty-maven/org/apache/activemq/activemq-parent/5.5.0/ivy-5.5.0.xml]
Resource missing. [HTTP HEAD: http://internal.artifactory/artifactory/thirdparty-maven/org/apache/activemq/activemq-parent/5.5.0/activemq-parent-5.5.0.jar]
Checksum SHA-1 matched cached resource: [HTTP GET: http://internal.artifactory/artifactory/remote-repos/org/apache/activemq/activemq-parent/5.5.0/activemq-parent-5.5.0.pom.sha1]
Resource missing. [HTTP GET: http://internal.artifactory/artifactory/thirdparty-maven/org/apache/apache/7/ivy-7.xml]
Resource missing. [HTTP HEAD: http://internal.artifactory/artifactory/thirdparty-maven/org/apache/apache/7/apache-7.jar]
Checksum SHA-1 matched cached resource: [HTTP GET: http://internal.artifactory/artifactory/remote-repos/org/apache/apache/7/apache-7.pom.sha1]
Resource missing. [HTTP HEAD: http://internal.artifactory/artifactory/remote-repos/org/apache/apache/7/apache-7.jar]
Resource missing. [HTTP HEAD: http://internal.artifactory/artifactory/remote-repos/org/apache/activemq/activemq-parent/5.5.0/activemq-parent-5.5.0.jar]
Resource missing. [HTTP HEAD: http://internal.artifactory/artifactory/remote-repos/org/apache/activemq/apache-activemq/5.5.0/apache-activemq-5.5.0.jar]
Resource missing. [HTTP GET: http://internal.artifactory/artifactory/remote-repos/org/apache/activemq/apache-activemq/5.5.0/apache-activemq-5.5.0-bin.tar.gz]
    FAILURE: Build failed with an exception.
  * Where:
Build file 'C:\dev\netsoft\boa\build.gradle' line: 12
  * What went wrong:
A problem occurred evaluating root project 'boa'.
> Could not resolve all dependencies for configuration ':activemq'.
   > Artifact 'org.apache.activemq:apache-activemq:5.5.0:bin@tar.gz' not found.
  * Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output.
  BUILD FAILED

So, from my view there are two issues at play here:

  1. In the documentation it says that if I am using the @tar.gz notation then gradle will only grab the artifact and not the descriptor, but this is clearly not the case. 2. It is possible to resolve a descriptor from one repo and an artifact from another

The first issue may just be a documentation update if this is intended behavior. The second issue seems scary to me, but maybe that is intended behavior as well…

I fixed this by just re-publishing to org.apache.activemq:activemq-bin:5.5.0@tar.gz So it’s an easy fix from that perspective, but seems like it may have exposed a couple areas of concern.


#2

So, from my view there are two issues at play here:

  1. In the documentation it says that if I am using the @tar.gz notation then gradle will only grab the artifact and not the descriptor, but this is clearly not the case.

You are correct: this is GRADLE-1194. At this time we’re not sure exactly how this will be addressed.

  1. It is possible to resolve a descriptor from one repo and an artifact from another

This is not correct: if Gradle resolves a descriptor from a particular repository, it should only use artifacts taken from the same repository.

Your problem has prompted me to write a short outline of the dependency resolution process for the user guide, which will be available soon. In brief:

  1. Given a required dependency, Gradle first attempts to resolve the module for that dependency. Each repository is inspected in order, searching first for a module descriptor file (pom or ivy file) that indicates the presence of that module. If no module descriptor is found, Gradle will search for the presence of the primary module artifact file indicating that the module exists in the repository.
  2. Once each repository has been inspected for the module, Gradle will choose the 'best' one to use. This is done using the following criteria:
    1. For a dynamic version, a 'higher' static version is preferred over a 'lower' version.
    2. Modules declared by a module descriptor file (ivy or pom file) are preferred over modules that have an artifact file only.
    3. Modules from earlier repositories are preferred over modules in later repositories.

    When the dependency is declared by a static version and a module descriptor file is found in a repository, there is no need to continue searching later

    repositories and the remainder of the process is short-circuited.

  3. If the module descriptor is a pom file that has a parent pom declared, Gradle will recursively attempt to resolve each of the parent modules for the pom.
  4. All of the artifacts for the module are then requested from the same repository that was chosen in the process above.

#3

So, from my view there are two issues at play here:

  1. In the documentation it says that if I am using the @tar.gz notation then gradle will only grab the artifact and not the descriptor, but this is clearly not the case.

You are correct: this is GRADLE-1194. At this time we’re not sure exactly how this will be addressed.

  1. It is possible to resolve a descriptor from one repo and an artifact from another

This is not correct: if Gradle resolves a descriptor from a particular repository, it should only use artifacts taken from the same repository.

Your problem has prompted me to write a short outline of the dependency resolution process for the user guide, which will be available soon. In brief:

  • Given a required dependency, Gradle first attempts to resolve the module for that dependency. Each repository is inspected in order, searching first for a module descriptor file (pom or ivy file) that indicates the presence of that module. If no module descriptor is found, Gradle will search for the presence of the primary module artifact file indicating that the module exists in the repository.

  • Once each repository has been inspected for the module, Gradle will choose the ‘best’ one to use. This is done using the following criteria:

    • For a dynamic version, a ‘higher’ static version is preferred over a ‘lower’ version.
  • Modules declared by a module descriptor file (ivy or pom file) are preferred over modules that have an artifact file only.

  • Modules from earlier repositories are preferred over modules in later repositories.

    • When the dependency is declared by a static version and a module descriptor file is found in a repository, there is no need to continue searching later

repositories and the remainder of the process is short-circuited.

Does this help explain the results you’re seeing? It’s further complicated by the fact that Gradle looks for jar artifacts when searching for a parent pom file.


(swaters) #4

Yes! That helps immensely.

Thank you for the thorough description of the dependency resolution. I was being thrown off by the http output thinking that it was grabbing both the artifact and the descriptor. Based on what you are saying though it is really only using the descriptor and ignoring the artifact without the descriptor.

I will keep an eye on the JIRA to see, what happens with this. Thanks again for the detailed response.


#5

Yep - since we find the pom file in ‘remote-repos’, we will only use artifacts from that repository. This is because we prefer a ‘declared’ module to an ‘implicit’ one (without a decriptor).

In the future we plan to make it easier to declare if a repository should allow ‘implicit’ modules or not. Until then the behaviour (based on ivy) may remain a little baffling.