Making Gradle find a JAR in a flatDir repo when it is missing from a Maven repo

Old Forum
1

I have my projects that depend (directly and transitively) on Log4j 1.2.15.

Because of the problem described here: http://onemanwenttomow.wordpress.com/2007/12/31/maven2-log4j-and-jmx-dependencies/ and here: http://unitstep.net/blog/2009/05/18/resolving-log4j-1215-dependency-problems-in-maven-using-exclusions/

adding the depedency to log4j:log4j:1.2.15 in my build files make Gradle 1.0-milestone-7 fail because of the changes made in the resolution of the missing JMX reference implementation JARs (milestone-5 and 6 simply skipped the missing JARs).

So, what I would like to do is to add those missing JARs (jmxtools-1.2.1.jar and jmxri-1.2.1.jar) to my flatDir repository provided with the Gradle project. However, even if my build.gradle says:

repositories {

flatDir dirs: “${rootProject.projectDir}/repo”

mavenRepo url: “${main_repo}”

mavenLocal () }

and I put those jars in “${rootProject.projectDir}/repo”, Gradle ignores them and fails with (for instance):

Could not resolve all dependencies for configuration ‘:MyProject:compile’. Cause: Artifact ‘com.sun.jdmk:jmxtools:1.2.1@jar’ not found

Changing my dependency declaration towards log4j adding (for instance) “exclude: module ‘jmxtools’” is not a viable solution, because log4j 1.2.15 is also required transitively by other dependencies, so I would need to add that exclusion to ALL the dependencies that I know that require Log4j… It’s ugly and unmaintainable.

So, my question is: is there a way to tell Gradle to pick up those missing JARs from the “repo” directory?

Actually, this is not the only case where I would need Gradle to look in that directory when it does not find something in the Maven repository. For instance, there are dependencies for which the source/javadoc archives are not in the Maven repository, but I added them in the “repo” dir… However, that is another case in which Gradle just ignores them and says it can’t find source/javadoc archives :frowning:

2

Here are a few options:

You can exclude the jmxtools and jmxri globally (this works for both direct and transitive dependencies):

configurations.all {
    exclude group: 'com.sun.jdmk', module: 'jmxtools'
    exclude group: 'com.sun.jmx', module: 'jmxri'
}

Another option would be to include the jars in a repo directory, but rather than using a flat directory repository, arrange them as a maven repository. You can then define a composite maven repo, that searches in multiple locations:

repositories {
     mavenRepo url: "${main_repo}", artifactUrls: ["${rootProject.projectDir}/repo"]
    mavenLocal ()
 }
3

Hi Adam, both your solutions are very interesting. However, the second one sounds like it’s the best one for me. I have a problem, though. After changing the repositories definition like you said, I now get the following error:

Could not resolve group:javax.mail, module:mail, version:1.4.3.
    Required by:
        com.cardinis.cardinis.cmod:COMMON:Trunk
        com.cardinis.cardinis.cmod:COMMON:Trunk > log4j:log4j:1.2.15
    Cause: unsupported protocol: 'file'

The stacktrace is (I’m reporting just the cause and some lines):

Caused by: java.lang.IllegalStateException: unsupported protocol: 'file'
 at org.apache.commons.httpclient.protocol.Protocol.lazyRegisterProtocol(Protocol.java:149)
 at org.apache.commons.httpclient.protocol.Protocol.getProtocol(Protocol.java:117)
 at org.apache.commons.httpclient.HttpHost.<init>(HttpHost.java:107)
 at org.apache.commons.httpclient.HttpMethodBase.setURI(HttpMethodBase.java:280)
 at org.apache.commons.httpclient.HttpMethodBase.<init>(HttpMethodBase.java:220)
 at org.apache.commons.httpclient.methods.GetMethod.<init>(GetMethod.java:89)
 at org.gradle.api.internal.artifacts.repositories.transport.http.HttpResourceCollection.getResource(HttpResourceCollection.java:84)
 at org.gradle.api.internal.artifacts.repositories.transport.http.HttpResourceCollection.getResource(HttpResourceCollection.java:58)
 at org.gradle.api.internal.artifacts.repositories.ResourceCollectionResolver.getResource(ResourceCollectionResolver.java:121)
 at org.gradle.api.internal.artifacts.repositories.ResourceCollectionResolver.findStaticResourceUsingPattern(ResourceCollectionResolver.java:91)
 at org.gradle.api.internal.artifacts.repositories.ResourceCollectionResolver.findResourceUsingPattern(ResourceCollectionResolver.java:76)
 at org.apache.ivy.plugins.resolver.AbstractPatternsBasedResolver.findResourceUsingPatterns(AbstractPatternsBasedResolver.java:93)
 at org.gradle.api.internal.artifacts.repositories.MavenResolver.findArtifactRef(MavenResolver.java:157)

What sounds strange for me is that javax.mail:mail:1.4.3 should be found in the remote repository, not in the repo directory.

Any suggestion?

4

Should I open an issue in JIRA for the problem I’m encountering now?

5

I did some experiments and I’m convinced there’s a bug here. With 1.0-milestone-6 and below that error is not produced, although the pick up of the JAR from the repo dir still doesn’t work. I opened GRADLE-2046

6

The actual bug is that you can’t mix http and file urls in the same maven repository. Unfortunately we are not catching this early and reporting it (like we do with ivy repositories).

I’ve updated GRADLE-2046 to reflect this.

7

Thank you Daz. But what will you do? Fix it in order to report early or make the mixed case work? It would be extremely usefule if the mixed scenario worked (as explained in my original question here).

8

Yes, we’d like you to be able to mix file and http urls in a single repository. Originally we didn’t think this would be a common use case, but yours is a valid example.

Please vote for this issue, and we’ll get to it as priorities dictate. We may well do the trivial thing first (report early) and address the underlying issue a little later.

9

Unfortunately I can’t vote, since the issue is mine :stuck_out_tongue: Anyway, thanks for the info. Mauro.

10

As an additional example, please consider the case when in the remote repository there is the jar but not the javadoc/source bundle and you want to provide one through the use of a local directory.

11

Just to clarify what’s going on here:

  1. If Gradle finds a jar file in a repository, but no meta-data file (ivy/pom), then Gradle will generate a module descriptor based on the presence of this jar. (This descriptor contains no dependency information, and a single ‘jar’ artifact). This is exactly how the “flatdir” repository works. 2. Gradle prefers to use a module with a real meta-data file than one with a generated module descriptor. So if Gradle finds only ‘jmxri-1.2.1.jar’ in one repository, but ‘jmxri-1.2.1.pom’ in a second, it will use the second repository to provide the module. Gradle considers the presence of a meta-data file to be more authoritative. 3. MavenCentral contains the POM, but not the jarfile for JMX. So that screws up our logic, since the jar-only repository is actually the better one to use. This is not great behaviour: we have an issue to remind us to improve this (GRADLE-2034)

A solution that should currently work would be very similar to your original design, but use a local ‘maven’ repository instead of a ‘flatdir’ repository. This local repository would contain pom files as well as artifacts, so there would be no need to search in subsequent repositories for the JMX module.

Something like:

repositories {
     maven { url
"${rootProject.projectDir}/repo" }
    maven { url "${main_repo}" }
    mavenLocal ()
 }

PS - do you really need mavenLocal() here?

12

To be honest, including source/javadoc artifacts is the “valid example” I am referring to.

I think in the case of Log4j including JMX unnecessarily, you have 2 better (imo) alternatives:

  1. Use global excludes if you don’t require the JMX functionality of Log4j 2. Create a fully-fledged local Maven repository to provide the JMX modules (including the pom files) if you do require JMX functionality
13

I see. However, please consider that if you want to be more “agile” and you don’t know in advance whether you’ll need JMX functionality or not, having to put up a whole local repository (including POMs) is not as straightforward as it could be having a local repository with just the missing JARs… After all, the mavenRepo() artifactUrls parameter aim sounds like to be exactly this…

14

I don’t know if I really need maveLocal()… I thought it could be useful to speed up things when I have artifacts in the local Maven cache… however the user guide is not very exhaustive on this topic…

Anyway, thank you for clarifying how things work. However, from what you write it seems like the typical case is that Gradle finds both POMs and JARs in one repository OR that it finds JARs first and then looks for POMs on other repositories… if it finds them, then the secondary repositories are used. However the documentation of mavenRepo() seems to suggest that the typical case is that Gradle find both JARs and POMs in one repository OR that it finds POMs in the first repository, than looks for JARs in other repositories. So the case of jmxri or jmxtools seems to fit perfectly in this second case. I know it is something “unusual”, however it’s real and since it rises when you require log4j (which is very common) I think it should be taken care of. Thank you very much for your support!

15

I think you are confusing the term “repository” with the url and artifactUrls properties of a single repository.

A single maven repository to Gradle consists of a single ‘url’ that will be searched for pom files and jar files, and a possible set of additional ‘artifactUrls’ that will be searched for jar files only. Together these urls are treated as a single, self-contained unit by Gradle, called a repository.

My description above refers to multiple, separate repositories. The maven repository documentation you are referring to is discussing multiple urls for the same repository. I hope this clarifies things a bit.

16

PS: You should remove mavenLocal() unless you are actually consuming artifacts that you publish locally using maven. Any speedup you gain from using it will be negligable, but it can serve to make your build less portable.

Gradle knows about the artifacts in the local maven repository, and will attempt to use them where possible. You can read about artifact reuse and other cache concepts in this new section of the user guide.

17

Please note GRADLE-2034 was closed without covering this use case. Could you please open a new issue?

18

Added GRADLE-2887

19

Why don’t you just copy the pom to the local repository along with the jar?

20

This would require to setup a local Maven repo and manually copy the poms+JARs there. And this should be done by all the team members.