How can you filter/avoid unnecessary metadata look-ups from the repository manager


(R. Fitzner) #1

Hello,

we are facing the following problem:

We are using Nexus 3.6 and have noticed that the more public proxy repositories we add to our public repository group the more time Gradle’s configuration phase needs. Also from time to time our Jenkins jobs are running into timeouts or cannot resolve the requested artifact because the Nexus is very busy trying to resolve metadata about /my/company artifacts in ALL repositories of a group (it doesn’t matter if it is a public proxy repository or an internal hosted repository) and then Nexus has to merge the results including failed metadata scans.

Sonatype describes in “Troubleshooting Artifact Download Failures” (see https://support.sonatype.com/hc/en-us/articles/213464518-Troubleshooting-Artifact-Download-Failures ) that “SNAPSHOT look-ups, or version ranges, or unversioned dependencies (typically plugins) will trigger metadata look-up”. It also describes how you can tweak it for maven on the client side (see chapter “Maven settings.xml or POM repository config blocking downloads”).

But Gradle’s internal maven plug-in does not support all maven features like the mentioned policy configurations for mirrors or repositories/pluginRepositories. Gradle supports maven policies only for uploads (see mavenDeployer.snapshotRepository and mavenDeployer.repository), but not for downloads (see https://docs.gradle.org/current/userguide/maven_plugin.html ).

What we would like to do is to filter or avoid unnecessary metadata look-ups from the client side.
For example something like this with a made up notation for filtering:

subprojects {
repositories {
    mavenLocal()
    maven {
        url 'http://nexus.company.my/repository/snapshot-maven2/'
        releases(enabled: false) // this is the notation from the maven plugin for upload, it is not taken into account for download at the moment
        snapshots(enabled: true)   // this is the notation from the maven plugin for upload, it is not taken into account for download at the moment
        include(group: 'my.company') // this is a made up notation to describe our requirement for allowing a specific group for download
    }
    maven {
        url 'http://nexus.company.my/repository/release-maven2/'
        releases(enabled: true)
        snapshots(enabled: false)
        include(group: 'my.company') // this is a made up notation to describe our requirement for allowing a specific group for download
    }
    maven {
        url 'http://nexus.company.my/repository/public-group-maven2/'   // this repository hosts all our proxy repository to public mirrored repositories for example maven-central
        releases(enabled: true)
        snapshots(enabled: false)
        exclude(group: 'my.company') // this is a made up notation to describe our requirement for excluding all artifacts from a specific group for download
    }
    dependencies {
        compile "my.company:my-module-a:1.5-SNAPSHOT" // still under development
        compile "my.company:my-module-b:2.0" // already released
        testCompile "org.mockito:mockito-core:1.15 "
    }
}
}

Background information we’ve already checked:
Since Nexus 3 routing rules for repositories are not supported anymore (see https://help.sonatype.com/repomanager3/repository-manager-2-to-3-feature-equivalency).
So you cannot configure “block/deny” all requested artifacts with a starting groupId/path (e.g. /my/company) for all or any proxy maven repositories anymore, but for Nexus 2 you still can (see https://blog.sonatype.com/2010/01/how-to-control-nexus-groups-with-effective-routing-rules/ ).

Can you please give me a hint where I can start to hook into Gradle’s lifecycle process to program it by myself? Is there perhaps a similar concept of Maven settings.xml in Gradle for resolving artifacts?
Or maybe I’ve missed something important in your manuals?

Thank you for your reply in advance.


(Daniel G) #2

It’s not exactly the solution you are looking for, but when we found ourselfs in the same situation with ivy, it helped to use virtual repositories to merge the smaller ones together. This way your buildtool will request the repository manager only once.

Anyways I’d love to see the integrated solution you proposed.