Gradle does not handle repositories where directory listings are XML

Naresh_Baliga · October 23, 2013, 2:43am

Hello, I’m on Gradle 1.8 and I have a local Ivy repository and one in SVN. Prior to upgrade to Gradle 1.8, I was using the add(FileSystemResolver) and add(SvnResolver) respectively.

After seeing the deprecated warnings for add, I’ve attempted to use the ivy DSL, however, for the Ivy via svn repository, I’m seeing this error:

A problem occurred evaluating root project 'myproject-ws-trunk'.
> Could not resolve all dependencies for configuration ':compileCopy'.
   > Could not resolve com.mycompany:my-api:5.2.+.
                Required by:
                                                              com.mycompany:myproject-ws-trunk:unspecified
                     > Could not list versions using Ivy pattern 'https://<mysvnrepository>/svn/ctsvn/ivy/repository/[organisation]/[module]/[revision]/ivy.xml'.
         > Unable to parse Http directory listing
                                                                                                                    > Unsupported ContentType text/xml for DirectoryListing

The build.gradle snippet is:

ivy {
      name "local"
      ivyPattern "${USER_HOME}/.build/local/[organisation]/[module]/[revision]/ivy.xml"
      artifactPattern "${USER_HOME}/.build/local/[organisation]/[module]/[revision]/[artifact](-[classifier])-[revision].[ext]"
   }
   ivy {
      name "ivysvn"
      url "https://<mysvnrepository>/svn/ctsvn/ivy/repository"
      layout "pattern", {
         ivy "[organisation]/[module]/[revision]/ivy.xml"
         artifact "[organisation]/[module]/[revision]/[artifact](-[classifier])-[revision].[ext]"
      }
      credentials {
         username 'jsmith'
         password 'mypassword'
      }
   }

Web server and svn details: Apache/2.2.15 (Unix) mod_ssl/2.2.15 OpenSSL/0.9.8e-fips-rhel5 DAV/2 SVN/1.6.16 PHP/5.3.3

The stack trace is:

Caused by: org.gradle.api.internal.resource.ResourceException: Could not list versions using Ivy pattern 'https://<mysvnrepository>/svn/ctsvn/ivy/repository/[organisation]/[module]/[revision]/ivy.xml'.
        at org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister$1.visit(ResourceVersionLister.java:60)
        at org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver.listVersionsForAllPatterns(ExternalResourceResolver.java:316)
        at org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver.findDynamicResourceUsingPatterns(ExternalResourceResolver.java:307)
        at org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver.findResourceUsingPatterns(ExternalResourceResolver.java:283)
        at org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver.findIvyFileRef(ExternalResourceResolver.java:254)
        at org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver.getDependency(ExternalResourceResolver.java:155)
        at org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver.getDependency(ExternalResourceResolver.java:148)
        at org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CacheLockingModuleVersionRepository$1.run(CacheLockingModuleVersionRepository.java:45)
        at org.gradle.internal.Factories$1.create(Factories.java:22)
        at org.gradle.cache.internal.DefaultCacheAccess.longRunningOperation(DefaultCacheAccess.java:214)
        at org.gradle.cache.internal.DefaultCacheAccess.longRunningOperation(DefaultCacheAccess.java:276)
        at org.gradle.cache.internal.DefaultPersistentDirectoryStore.longRunningOperation(DefaultPersistentDirectoryStore.java:142)
        at org.gradle.api.internal.artifacts.ivyservice.DefaultCacheLockingManager.longRunningOperation(DefaultCacheLockingManager.java:50)
        at org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CacheLockingModuleVersionRepository.getDependency(CacheLockingModuleVersionRepository.java:43)
        at org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleVersionRepository.getDependency(CachingModuleVersionRepository.java:87)
        at org.gradle.api.internal.artifacts.ivyservice.ivyresolve.memcache.CachedRepository.getDependency(CachedRepository.java:54)
        at org.gradle.api.internal.artifacts.ivyservice.ivyresolve.UserResolverChain$RepositoryResolveState.resolve(UserResolverChain.java:183)
        at org.gradle.api.internal.artifacts.ivyservice.ivyresolve.UserResolverChain.findLatestModule(UserResolverChain.java:97)
        at org.gradle.api.internal.artifacts.ivyservice.ivyresolve.UserResolverChain.findLatestModule(UserResolverChain.java:80)
        at org.gradle.api.internal.artifacts.ivyservice.ivyresolve.UserResolverChain.resolve(UserResolverChain.java:55)
        ... 95 more
Caused by: org.gradle.api.internal.resource.ResourceException: Unable to parse Http directory listing
        at org.gradle.api.internal.externalresource.transport.http.HttpResourceLister$1.transform(HttpResourceLister.java:57)
        at org.gradle.api.internal.externalresource.transport.http.HttpResourceLister$1.transform(HttpResourceLister.java:49)
        at org.gradle.api.internal.externalresource.AbstractExternalResource.withContent(AbstractExternalResource.java:60)
        at org.gradle.api.internal.externalresource.transport.http.HttpResourceLister.list(HttpResourceLister.java:49)
        at org.gradle.api.internal.externalresource.transport.DefaultExternalResourceRepository.list(DefaultExternalResourceRepository.java:119)
        at org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister$1.listAll(ResourceVersionLister.java:145)
        at org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister$1.listRevisionToken(ResourceVersionLister.java:73)
        at org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister$1.visit(ResourceVersionLister.java:53)
        ... 114 more
Caused by: org.gradle.api.internal.resource.ResourceException: Unsupported ContentType text/xml for DirectoryListing
        at org.gradle.api.internal.externalresource.transport.http.ApacheDirectoryListingParser.parse(ApacheDirectoryListingParser.java:45)
        at org.gradle.api.internal.externalresource.transport.http.HttpResourceLister$1.transform(HttpResourceLister.java:54)
        ... 121 more

Any help would be appreciated.

Naresh_Baliga · October 23, 2013, 3:06am

I did a GET on the URL and the content is indeed XML.

And through an XSL stylesheet, the content is made pretty.

So, one option is to ask the administrators to change the Apache listing option to HTML instead of XML.

Though given that it’s a possibility that web servers could be configured this way, is there any existing support for Gradle Ivy for this?

daz · October 23, 2013, 4:33am

There’s no existing support for this, but we’d be open to a patch for this support: would you be interested in working on a fix?

We’d probably need to update ‘org.gradle.api.internal.externalresource.transport.http.HttpResourceLister’ to detect the content type and handle XML.

As well as this, we’d want to add some integration test coverage by adding functionality to send XML directory listing to ‘org.gradle.test.fixtures.server.http.HttpServer’, and by adding another test case to ‘org.gradle.integtests.resolve.ivy.IvyDynamicRevisionRemoteResolveIntegrationTest’ that tests with XML directory listings.

daz · October 23, 2013, 4:40am

I’ve raised GRADLE-2928 to track this issue.

Naresh_Baliga · October 23, 2013, 6:52am

I could try. Which branch should I clone? I’ve cloned 1.8 rc 2 just to get started. I’ve followed the instructions for generating the IntelliJ project. However I’m running into build errors for the Gradle run configuration.

It complains about:

/home/nbaliga/.IntelliJIdea11/system/compiler/gradle.gradlefa164059/.generated/groovyStubs/gradlefa164059/plugins/tests/org/gradle/api/internal/tasks/testing/junit/ATestClassWithSuiteMethod.java
error: cannot find symbol class Test

Is there any “getting started” resource for budding contributors that you could point me to?

luke_daley · October 23, 2013, 10:39am

Some docs on development here: https://github.com/gradle/gradle/blob/master/README.md Docs on contribution requirements: https://github.com/gradle/gradle/blob/master/CONTRIBUTING.md

You should work off of master, as that will reduce the likelihood of merge conflicts. You should continuously rebase your work on top of the moving master.

daz · October 23, 2013, 9:25pm

Regarding errors in IntelliJ, you need to do a few things: 1) Disable “Use external build” in the compiler settings 2) Use “Exclude from stub generation” for the 1 or 2 classes that fail in Groovy compilation (giving the error you mention above) 3) Compile multiple times when you get errors about visibility of setup() methods in Spock specs.

These are all to workaround shortcomings in the Groovy support in IntelliJ IDEA.

Naresh_Baliga · October 24, 2013, 7:41am

How do I go about debugging Gradle in IntelliJ once the project has been created?

I’ve made some changes to AbstractDirectoryListingParser for example. And I want to debug the project in IntelliJ by running the modified Gradle on a sample build.gradle.

daz · October 24, 2013, 7:45am

If you have executed ‘gradle idea’ to generate your IntelliJ project, then you should have a pre-built “Gradle” run configuration. You can set the working directory to something different, and then execute (& debug) a Gradle build from within your IDE.