We have code that depends on org.apache.crunch:crunch-core:0.8.0-cdh4.3.0 and org.apache.crunch:crunch-hbase:0.8.0-cdh4.3.0, neither of which can be resolved because of the following problems:
> Could not resolve org.apache.crunch:crunch-core:0.8.0-cdh4.3.0.
...
> Could not parse POM /Users/Shared/m2_repo/org/apache/crunch/crunch-core/0.8.0-cdh4.3.0/crunch-core-0.8.0-cdh4.3.0.pom
> Unable to resolve version for dependency 'org.apache.hadoop:hadoop-client:jar'
> Could not parse POM https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/crunch/crunch-core/0.8.0-cdh4.3.0/crunch-core-0.8.0-cdh4.3.0.pom
> Unable to resolve version for dependency 'org.apache.hadoop:hadoop-client:jar'
> Could not resolve org.apache.crunch:crunch-hbase:0.8.0-cdh4.3.0.
...
> Could not parse POM /Users/Shared/m2_repo/org/apache/crunch/crunch-hbase/0.8.0-cdh4.3.0/crunch-hbase-0.8.0-cdh4.3.0.pom
> Unable to resolve version for dependency 'org.apache.hbase:hbase:jar'
> Could not parse POM https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/crunch/crunch-hbase/0.8.0-cdh4.3.0/crunch-hbase-0.8.0-cdh4.3.0.pom
> Unable to resolve version for dependency 'org.apache.hbase:hbase:jar'
If you look at the crunch-core-0.8.0-cdh4.3.0.pom and crunch-hbase-0.8.0-cdh4.3.0.pom you’ll see that they inherit their versions for hadoop-client and hbase from a crunch-parent POM. That parent POM shows that those versions are defined by hadoop.version and hbase.version properties that are set in a “hadoop-1” profile automatically-activated when the crunch.platform property is not specified.
To summarize, it appears that Gradle does not see that this profile is active and consequently does not realize the hadoop.version and hbase.versions.
You are right, and it’s a known limitation. Having profiles in published POMs is problematic, and highly discouraged even by the Maven folks. Nevertheless, there are some plans to at least add limited support (e.g. honoring profiles that are ‘activeByDefault’) at some point.
Well that’s rather unfortunate. I was pursuing switching some of our projects over to Gradle, but this blocks us. Granted, we might have been blocked for other undiscovered reasons, too, but this blocked me on step 1. Is there a JIRA I can track for this enhancement? Are you aware of any workaround? I tried declaring the hadoop.version and hbase.version properties to their appropriate values in the gradle.properties file, just in case that magically made it work, but not surprisingly it did not. I also tried adding hadoop-client and hbase as first-class dependencies in the build.gradle file, and that, too, did not work.
While I understand what you mean about having build profiles in published POMs and why that’s problematic, I find it interesting that the Maven folks discourage it. As far as I can tell Maven doesn’t allow you to not do it.
You can either edit the POMs in your enterprise repository, or add the transitive dependencies referred to by the profiles directly. (I don’t see why this wouldn’t work.) I’m pretty sure there is already a JIRA for this (see http://issues.gradle.org).
Sean, we have Maven POM profile support coming up for Gradle 1.12-RC-1 (please also see GRADLE-1997). For now we will only be handling profiles that are marked ‘activeByDefault’. We would love to hear your feedback.
In my particular case the support of activeByDefault wouldn’t fix the problem since it’s “active by default” through the non-existence of a system property as opposed to the actual activeByDefault element, but regardless, I can still just modify the POM to switch to use the activeByDefault element to help verify the functionality. As long as your support addresses both dependencies and properties, not just one or the other, it seems like it would fit the needs of both me and others I’ve seen on this forum mentioning the same issue. I’ll await the 1.12-RC-1 to try it out.
The inital implementation is going to parse and evaluate ‘properties’, ‘dependencies’ and ‘dependencyManagement’ elements for profiles that are ‘activeByDefault’. We might add profile switches in a future Gradle release.
I am using Gradle 1.12 and i still see these failures.
Looks like some issue with crunch-core pom which is not handled by gradle.
> Could not resolve org.apache.crunch:crunch-core:0.7.0+16-cdh4.3.0.
Required by:
com.x.x.-executables:10.0.1-SNAPSHOT
> Could not parse POM https://nexus....com/content/groups/public/org/apache/crunch/crunch-core/0.7.0+16-cdh4.3.0/crunch-core-0.7.0+16-cdh4.3.0.pom
> Unable to resolve version for dependency 'org.apache.hadoop:hadoop-client:jar'
Kavin, I believe it’s because Gradle added support for profiles that have activeByDefault = true defined, but Crunch does not define its profile in this manner; instead it activated it through the absence of a system property.
FYI, I submitted a patch to Crunch to work around this problem in Gradle onto the 0.10.0-SNAPSHOT version and I believe it was patched back to the 0.8.3 version. Of course since you’re on an even older version that won’t do much good for you.
That’s correct. Profile activation through the absence of a system property is implemented in Gradle 2.0. 1.12 doesn’t have that feature yet. Once the 2.0-rc-1 is out you’ll be able to try it.