Oddity in the output of dependencyInsight task


(Mark Petrovic) #1

Hi. I’m using Gradle 1.3 to analyze some dependency resolution. I am having trouble interpreting the output, however.

I run this

gradle -i dependencyInsight --dependency xml-apis

The task tells me that xml-apis:xml-apis:2.0.2 is the resolved artifact after conflict resolution, but I have no idea where this version 2.02 artifact is coming from. My read of the report is that it just shows up out of nowhere. My project does not depend directlry on it (and I would expect that to be reflected in the output if it did).

Selected primary task 'dependencyInsight'
Tasks to be executed: [task ':services:ws-server:dependencyInsight']
:services:ws-server:dependencyInsight (Thread[Daemon,5,main] - start
:services:ws-server:dependencyInsight
Executing task ':services:ws-server:dependencyInsight' due to:
  Task has not declared any outputs.
xml-apis:xml-apis:2.0.2 (conflict resolution)
  xml-apis:xml-apis:1.3.04 -> 2.0.2
\--- xalan:serializer:2.7.1
     \--- xalan:xalan:2.7.1
          +--- org.acme.payment:payment-integration-card-vendor:12.0.0-SNAPSHOT
          |
  \--- org.acme.payment:payment-integration-card-service:12.0.0-SNAPSHOT
          |
       \--- org.acme.payment:payment-integration-service:12.0.0-SNAPSHOT
          |
            \--- org.acme.qa:atf:master-SNAPSHOT
          |
                 \--- compile
          \--- net.sourceforge.htmlunit:htmlunit:2.12
               \--- org.seleniumhq.selenium:selenium-htmlunit-driver:2.35.0
                    \--- org.seleniumhq.selenium:selenium-java:2.35.0
                         \--- org.seleniumhq.selenium:selenium-server:2.35.0
                              \--- org.acme.qa:atf:master-SNAPSHOT (*)
  xml-apis:xml-apis:1.4.01 -> 2.0.2
\--- xerces:xercesImpl:2.10.0
     +--- flux:flux-jar:7.9.9-local-01
     |
  +--- org.acme.qa:atf:master-SNAPSHOT
     |
  |
  \--- compile
     |
  \--- org.acme.cli-app-instances:jobs-server-client:12.0.0-SNAPSHOT
     |
       \--- org.acme.qa:atf:master-SNAPSHOT (*)
     \--- net.sourceforge.htmlunit:htmlunit:2.12
          \--- org.seleniumhq.selenium:selenium-htmlunit-driver:2.35.0
               \--- org.seleniumhq.selenium:selenium-java:2.35.0
                    \--- org.seleniumhq.selenium:selenium-server:2.35.0
                         \--- org.acme.qa:atf:master-SNAPSHOT (*)
  (*) - dependencies omitted (listed previously)
:services:ws-server:dependencyInsight (Thread[Daemon,5,main]) - complete

Here is some supporting information from “gradle dependencies”:

> grep xml-apis bad.deps
 |
  |
  |
       \--- xml-apis:xml-apis:1.4.01 -> 2.0.2
|
  |
  |
  |
  |
       \--- xml-apis:xml-apis:1.3.04 -> 2.0.2 (*)
|
  |
  |
       \--- xml-apis:xml-apis:1.4.01 -> 2.0.2
|
  |
  |
  |
  |
       \--- xml-apis:xml-apis:1.3.04 -> 2.0.2 (*)
|
  |
  |
       \--- xml-apis:xml-apis:1.4.01 -> 2.0.2
|
  |
  |
  |
  |
       \--- xml-apis:xml-apis:1.3.04 -> 2.0.2 (*)
|
  |
  |
       \--- xml-apis:xml-apis:1.4.01 -> 2.0.2
|
  |
  |
  |
  |
       \--- xml-apis:xml-apis:1.3.04 -> 2.0.2 (*)
|
  |
  |
       \--- xml-apis:xml-apis:1.4.01 -> 2.0.2
|
  |
  |
  |
  |
       \--- xml-apis:xml-apis:1.3.04 -> 2.0.2 (*)

which also shows the v2.0.2 artifact apparently showing up out nowhere.

I must be missing something, and am baffled.

Can anyone comment?

Thank you.


(Peter Niederwieser) #2

You seem to have both direct and indirect dependencies on ‘xml-apis’. Maybe try with a later Gradle version, to rule out that it’s a Gradle problem that has meanwhile been fixed.


(Peter Niederwieser) #3

PS: This line reflects that it’s a direct dependency: ‘xml-apis:xml-apis:2.0.2 (conflict resolution)’


(Mark Petrovic) #4

Here is the output of the same task, but now using Gradle 1.8. Seems the same output.

By direct, you mean my project depends on xml-apis v2.0.2?

Executing task ':services:ws-server:dependencyInsight' due to:
  Task has not declared any outputs.
xml-apis:xml-apis:2.0.2 (conflict resolution)
  xml-apis:xml-apis:1.3.04 -> 2.0.2
\--- xalan:serializer:2.7.1
     \--- xalan:xalan:2.7.1
          +--- org.acme.payment:payment-integration-card-vendor:12.0.0-SNAPSHOT
          |
  \--- org.acme.payment:payment-integration-card-service:12.0.0-SNAPSHOT
          |
       \--- org.acme.payment:payment-integration-service:12.0.0-SNAPSHOT
          |
            \--- org.acme.qa:atf:master-SNAPSHOT
          |
                 \--- compile
          \--- net.sourceforge.htmlunit:htmlunit:2.12
               \--- org.seleniumhq.selenium:selenium-htmlunit-driver:2.35.0
                    \--- org.seleniumhq.selenium:selenium-java:2.35.0
                         \--- org.seleniumhq.selenium:selenium-server:2.35.0
                              \--- org.acme.qa:atf:master-SNAPSHOT (*)
  xml-apis:xml-apis:1.4.01 -> 2.0.2
\--- xerces:xercesImpl:2.10.0
     +--- flux:flux-jar:7.9.9-local-01
     |
  +--- org.acme.qa:atf:master-SNAPSHOT
     |
  |
  \--- compile
     |
  \--- org.acme.cli-app-instances:jobs-server-client:12.0.0-SNAPSHOT
     |
       \--- org.acme.qa:atf:master-SNAPSHOT (*)
     \--- net.sourceforge.htmlunit:htmlunit:2.12
          \--- org.seleniumhq.selenium:selenium-htmlunit-driver:2.35.0
               \--- org.seleniumhq.selenium:selenium-java:2.35.0
                    \--- org.seleniumhq.selenium:selenium-server:2.35.0
                         \--- org.acme.qa:atf:master-SNAPSHOT (*)
  (*) - dependencies omitted (listed previously)
:services:ws-server:dependencyInsight (Thread[Daemon,5,main]) - complete

(Peter Niederwieser) #5

Yes, that’s what I mean. Perhaps the dependency is declared by a parent build script?


(Mark Petrovic) #6

Here is how I would chop through that analysis, from the top level directory of my build:

$ find . -name build.gradle -print0 | xargs -0 grep -l ‘xml-apis’

which produces no output. I have no dependency on this artifact.

I’m a bit concerned about this. This dependency is preventing my app from starting; it’s inducing a CNF exception for this class org.w3c.dom.ElementTraversal at Spring startup.

What else might I try?


(Peter Niederwieser) #7

Not sure. Maybe 2.0.2 is a transitive dependency then. Try grepping through ‘gradle dependencies’ (for the subproject).


(Mark Petrovic) #8

That grep output appears above my “am baffled” comment above. And nowhere in that output do I see the v2.0.2 artifact appear on a line by itself, which would indicate that something is bringing it in to predominate in conflict resolution.

What is the syntax to force a dependency? Maybe I can learn something by forcing the last known good version for this artifact. I know that’s cowardly. It’s where I am.


(Mark Petrovic) #9

Here’s another oddity.

I have this task in my subproject

task printRuntimeClasspath << {
    def classpath = sourceSets.main.runtimeClasspath
    classpath.each { File f ->
        println "cp " + f.name
    }
}

When I execute it, the xml-apis artifact is simply not present in the output. Something therefore appears to be keeping the real resolved xml-apis jar off the classpath, which would explain the CNF exception.

Very strange.


(Mark Petrovic) #10

Here is additional information, generated with “gradle -d dependencies”. Here I do see a conflict resolution in favor of xml-apis v2.0.2 arriving via xalan 2.6.0 somewhere along the graph. My project has a transitive dependency on xalan 2.7.1 which has no dependency on xml-apis v2.0.2.

Somehow it seems gradle is both selecting the wrong xml-apis version and then not including it on the runtime classpath. In fact, when I clear the gradle cache, the v2.0.2 artifact is apparently not downloaded into the cache when I build. In my “good project”, which is defined as my project when my app could start several project-git-commits earlier, xml-apis v1.4.01 was downloaded and which contains the original class I need on the classpath org/w3c/dom/ElementTraversal.class.

Sorry for not using gists earlier for large text chunks.

https://gist.github.com/ae6rt/6951063

What other information can I provide?


(Mark Petrovic) #11

So I was able to get my app back in something resembling working order by forcing the version I can live with:

configurations.all {
        resolutionStrategy {
            force 'xml-apis:xml-apis:1.4.01'
        }
    }

which I included in allprojects {} of my top level.

I can now instantiate a Spring context successfully.

I still cannot explain why I need to do this, nor have I read anything here that explains it.

I’m willing to work with folks here to get to the true root cause. As buildmaster for this large automated testing app, I’ve not experienced anything like this since migrating to Gradle last year. I’ve read the books, the User Guide, the DSL, and I cannot reconcile why v2.0.2 is selected, or why it’s not put on the classpath after it is.

So now that I believe I’m back on my feet, how can I help?


(Szczepan Faber) #12

Hey Mark,

The output may not indicate that 2.0.2 is a direct dependency. It may indicate that 2.0.2 version is selected but it “does not exist” in the graph. This may happen when version is forced. However the report says it’s “conflict resolution” not “forced” and I don’t know why.

Is there a chance you can dig further? A small project that can repro this issue would be awesome.

Thanks a lot for your efforts on digging into it.


(Mark Petrovic) #13

I’ll be happy to dig into it, and thanks for travelling with me on this.

A small project to reproduce is going to be a challenge. The artifact that tipped my build from working to not working is not something I can share with the outside the company. That artifact has transitive dependencies that somehow fouled things up, but I don’t have a quantitative analysis of why yet.

To summarize what I did so far:

  1. prove the effect exists in Gradle 1.8, as well as the version 1.3 I’m currently working under

  2. prove that the xml-apis v2.0.2 does not appear in my direct or transitive dependencies using “gradle dependencies”. Direct dependency is doubly-proven absent by grepping through all my build.gradle files (25 or so of them, all handcrafted by me)

  3. prove that xml-apis v2.0.2 appears to be selected out of thin air by running “gradle dependInsight --dependency xml-apis”

  4. prove that forcing xml-apis v1.4.01 via the DSL resolution strategy nominally fixes my immediate problem

  5. prove that xml-apis v2.0.2 is referenced in “gradle -d dependencies” but I cannot decipher that output to the degree needed to understand what ultimately called for it. xalan 2.6.0 uses it in the immediate sense, but nothing in my project, directly or transitively, uses anything but xalan 2.7.x. But at least there is some obscure reference to xml-apis v2.0.2 in that debug output.

Your choice of words “selected but does not exist in the graph” is probably the best phrasing of the problem. That describes the situation.

I’ve run out of tricks and don’t know what to try next. I can send you full debug output via email - I’d rather not post all that here because it contains package names and functionality that I’d rather not air publicly.

What can I try next?

Thanks again. I think this is worth pursuing to the fullest degree.

Mark


(Mark Petrovic) #14

Whoa, check this out.

I just ran the insight task with nightly-latest and I see this on the console

$ ~/tools/gradle-1.10-20131011231056+0000/bin/gradle
--no-daemon dependencyInsight --dependency xml-apis
 ...
:buildSrc:build UP-TO-DATE
The groovy configuration has been deprecated and is scheduled to be removed in Gradle 2.0. Typically, usages of 'groovy' can simply be replaced with 'compile'. In some cases, it may be necessary to additionally configure the 'groovyClasspath' property of GroovyCompile and Groovydoc tasks.
POM relocation to an other version number is not fully supported in Gradle : xml-apis#xml-apis;2.0.2 relocated to xml-apis#xml-apis;1.0.b2.
Please update your dependency to directly use the correct version 'xml-apis#xml-apis;1.0.b2'.
Resolution will only pick dependencies of the relocated element.
Artifacts and other metadata will be ignored.
The ConfigurationContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead.
The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead.

See the relocation of xml-apis? That looks significant.


(Mark Petrovic) #15

There’s a lot of state to wade through in debugging this, and I’m still gathering facts. Not knowing precisely what I’m looking for leads me to just start digging and look for anything related to xm-apis 2.0.2.

So in the output of “gradle -d dependencies” in my project of interest, I noticed some other project’s configuration was being “visited”. I found that odd because the project of interest does not depend on that project. I then set a breakpoint in

org.gradle.api.internal.artifacts.ivyservice.resolveengine.DependencyGraphBuilder#traverseGraph

and found that xml-apis 2.0.2 is a dependency of a Cobertura plugin (https://github.com/stevesaliman/gradle-cobertura-plugin) I’m using in another project of this multi-project build. My project does not apply this Cobertura plugin. I grant, contrary to what I said earlier, that xml-apis 2.0.2 may be a transitive dep of some other artifact I’m depending on, but it may be. But for now, no matter - I’ll study that more in the days to come.

But answer me this: consider this test project I came up with

https://github.com/ae6rt/gradle-buildscript-visiting

Go into the theone/ subdirectory and execute this to see that the other project in the build has its configuration being visited:

$ cd theone/
theband:theone> ../gradlew -d depend --configuration compile | grep Visiting.*cob | awk '{print $4 " " $5 " " $6}' | grep with-cob
Visiting configuration parent:with-cobertura:1.0(classpath).
Visiting dependency parent:with-cobertura:1.0(classpath)

Is that to be expected? I find it odd because my project does not depend on it, and could put at risk my project’s classpath in conflict resolution. Remove the “buildscript” clause in the other project’s build and the visiting goes away.

More later.

Thanks. Mark


(Mark Petrovic) #16

I just found this

http://forums.gradle.org/gradle/topics/gradle_does_not_report_missing_failed_artifact_downloads

and this

http://issues.gradle.org/browse/GRADLE-2812

which sound similar to what I’ve reported in the current thread.


(Mark Petrovic) #17

Ok, I think I figured it out.

Given

http://issues.gradle.org/browse/GRADLE-2812

and the fact that one of my dependencies has a dependency on xalan:xalan:2.7.0 which has a dependency on xml-apis:2.0.2, I now see how xml-apis:2.0.2 is selected but never makes it on the classpath of the app.