I would like to copy dependencies of project to specific directories when creating distribution.
Requirements were simplified from my previous approach, so I think it can be done better ( previous approach: http://forums.gradle.org/gradle/topics/organizing_downloaded_artifacts ). In fact previous approach was not correct and cause a lot of problems, so I need to start again…
My files (simplified):
common.gradle:
configurations {
sabreLib {
description = 'Sabre libraries'
transitive = true
}
mavenLib {
description = 'Common libraries'
transitive = false
}
compile { extendsFrom sabreLib, mavenLib }
}
ext.distributionLibDirs = [
'logging' : [ 'org.slf4j:.*:.*', 'log4j:.*:.*' ],
'oracle' : [ 'com.oracle:.*:.*' ]
]
apply from: 'externals/gradle/distribution.gradle'
distribution.gradle
import java.util.regex.Pattern;
def apply(Map libraries, closure) {
if (libraries == null) {
return;
}
for ( entry in libraries ) {
String[] artifacts = entry.value
for ( lib in artifacts ) {
closure( entry.key, lib )
}
}
}
def dependencySpec(String pattern, boolean negate) {
return new Spec<Dependency>() {
boolean isSatisfiedBy(Dependency element) {
String group = element.getGroup() == null ? "" : element.getGroup()
String name = element.getName() == null ? "" : element.getName()
String version = element.getVersion() == null ? "" : element.getVersion()
String[] array = pattern.split(":")
boolean result = group ==~ array[0] && name ==~ array[1] && version ==~ array[2]
result = negate ? !result : result
//println "negate: $negate; pattern: $pattern <-> $group:$name:$version; result: $result"
return result
}
}
}
task createDistributionDir(dependsOn: jar) << {
description = 'Create project distributution.'
delete distributionDir
copy {
into distributionDir
from( jar.archivePath )
Configuration collection = configurations.compile.copyRecursive()
apply( distributionLibDirs ) { directory, pattern ->
Set<File> files = collection.files( dependencySpec( pattern, false ) )
from( files ) { into "lib/$directory" }
collection = collection.copyRecursive( dependencySpec( pattern, true ) )
}
from( collection ) { into "lib" }
}
}
task createDistribution(type: Zip, dependsOn: createDistributionDir ) {
archiveName = "${baseLine}.zip"
from distributionDir
}
In fact biggest problem is with following code:
copy {
into distributionDir
from( jar.archivePath )
Configuration collection = configurations.compile.copyRecursive()
apply( distributionLibDirs ) { directory, pattern ->
Set<File> files = collection.files( dependencySpec( pattern, false ) )
from( files ) { into "lib/$directory" }
collection = collection.copyRecursive( dependencySpec( pattern, true ) )
}
from( collection ) { into "lib" }
}
I would like that this code take all dependencies from configuration compile and copy artifacts to specific directories according to defined regexps…
I tried a lot of different possibilities but no one seems to work…
Could you please suggest how to make it work?
Can you explain in words what you are trying to achieve here?
Sure! I would like to iterate over all dependencies (with transitive ones from super configurations). Then based on definitions from ext.distributionLibDirs I would like to copy specific jars to specific directories.
Algorithm in above snippet: 1. Get all dependencies (collection) 2. Based on predicate (dependencySpec(pattern, false)) I want to take all dependencies meeting this predicate 3. Copy dependencies from previous step to specific directory 4. Remove from collection all dependencies, which were already copied in this step 5. Repeat until all definitions in ext.distributionLibDirs are used. 6. Rest of libraries, which were not copied to subdirectories just copy to lib directory.
We are using Gradle in our team in Sabre Holdings. And I get requirement to divide jars into subdirectories when creating distribution. It should be dead simple, but I can not achieve my goal… I must do something very wrong…
I can’t tell outright what’s wrong with the code. Can you be more specific than “no one seems to work”? What exactly isn’t working? Have you tested and/or debugged the code?
A potential simplification is to make a single pass over all artifacts (e.g. ‘configuration.resolvedConfiguration.resolvedArtifacts’) and add a copy action for each of them. Instead of making this an ad-hoc task, I’d use a task of type ‘Sync’. By replacing ‘from(jar.archivePath)’ with ‘from(jar)’, you can drop the explicit ‘dependsOn’.
Thanks a lot for your hints! I have changed my code as below, and indeed it works:
common.gradle:
ext.distributionLibDirs = [
'logging' : [ 'org\.slf4j:.*:.*', 'log4j:.*:.*' ],
'oracle' : [ 'com\.oracle:.*:.*' ],
'mom' : [
'com\.sabre\.messaging:sems:.*',
'com\.ibm:com\.ibm\.mq.*:.*',
'com\.ibm:com\.ibm\.dhbcore:.*',
'javax\.transaction:jta:.*'
]
]
distribution.gradle:
task createDistributionDir << {
delete distributionDir
copy {
into distributionDir
for(artifact in configurations.compile.resolvedConfiguration.resolvedArtifacts) {
String dir = "lib/" + artifactSubdir(artifact)
from( artifact.getFile() ) { into dir }
}
}
}
private String artifactSubdir(ResolvedArtifact artifact) {
String group = artifact.getModuleVersion().getId().getGroup()
group = group == null ? "" : group
String name = artifact.getModuleVersion().getId().getName()
name = name == null ? "" : name
String version = artifact.getModuleVersion().getId().getVersion()
version = version == null ? "" : version
String artifactName = "$group:$name:$version"
for(entry in distributionLibDirs) {
for (pattern in entry.value) {
if ( artifactName ==~ pattern ) {
return entry.key;
}
}
}
return "";
}
task createDistribution(type: Zip, dependsOn: createDistributionDir ) {
archiveName = "${baseLine}.zip"
from distributionDir
}
I would like also to comment a bit about Gradle, which is really great tool, but has it’s own shortcomings:
- API for getting information about group/name/version from resovedArtifact is really unintuitive. I would never look for something like: artifact.getModuleVersion().getId().getVersion() 2. Why task of type Copy can not be executed when you use it as: task createDistributionDir (type: Zip) << { } I couldn’t make it work this way and have to use copy {} block, which seems superfluous, when there is Copy task type.
Thanks for your support and great work with Gradle!
ad 1. I hear you, but the consensus is that it is the price to pay for a correct, versatile and future-proof API.
ad 2. Your declaration is wrong. You need to get rid the left-shift operator. Since this is such a common mistake, I recommend to never use this operator and always use the ‘doLast’ method instead.
- Maybe some kind of facade for scripting? Just for consideration… 2. Well, when I remove << then there is no ‘lib’ directory at all in my distribution. What should I do to make it in correct way? Additionally why should I make this task in configuration phase, when in fact it should be done only on demand…
- lib is not appearing probably because I apply common.gradle before dependencies {} block. But I can not move applying common.gradle after dependencies block, as common.gradle defines also configurations for dependencies block. I could probably split common.gradle somehow, but it makes solution even more complicated. I think that better solution is to delay createDistribution task, so it is configured and executed just when it is needed.
If you wish I can send you all my scripts…
Unfortunately removing explicit dependency (dependsOn: jar) is not working. I have changed task createDistribution as you proposed: 1. I have removed dependsOn 2. I have dropped .archivePath part from copy definition
After these changes jar was no longer built. But if it was already built before, it was copied.
So something is wrong with such implicit dependencies. It also tells me that scripting non standard things in Gradle is still more complicated than necessary…
Did you switch to a ‘Copy’ (or ‘Sync’) task? Otherwise it won’t work. That’s one of the reasons why task types should, whenever possible, be preferred over the corresponding methods (‘Copy’ vs. ‘copy’).
ad 1. I don’t know what you mean by that. ad 2. Task configuration must happen in the configuration phase. ad 3. I can’t comment on this because I’m not familiar with your code. It’s not clear to me why you’d have to move the plugin application behind the ‘dependencies’ block.
PS: Your logic is complicated enough to warrant a task/plugin class along with some tests. This might help to derive a working solution.
When I switched to following code:
task createDistributionDir(type: Copy) {
delete distributionDir
into distributionDir
from( "conf" ) { into "conf" }
from( "scripts" )
from( jar )
for(entry in verbatimCopy) {
from( entry.key ) { into entry.value }
}
for(artifact in configurations.compile.resolvedConfiguration.resolvedArtifacts) {
String dir = "lib/" + artifactSubdir(artifact)
from( artifact.getFile() ) { into dir }
}
}
I got hundreds of java compile errors, so I dropped this approach… (Errors about missing dependencies, which in fact are provided.)
The ‘delete’ line is wrong (it’s performing a delete in the configuration phase, no matter which tasks will be run). A simple fix is to use a ‘Sync’ task (and remove the ‘delete’). The loop in its current form would have to go into a ‘doFirst { … }’, and you’d have to add ‘inputs.file configurations.compile’ somewhere (say before ‘into distributionDir’). This should give a ‘Sync’ task that has the right dependencies and knows whether it is up-to-date.
I would never suspect delete in above case, and to say true I still don’t understand what’s a difference between delete and other statements. Distribution dir have nothing to do with compilation, so what’s the impact? Whats more distribution dir have timestamp inside it’s name, so every time it’s different. This delete is kind of “defensive programming”, to avoid unexpected problems…
I will check above changes when I come back from vacations… Anyway it is really difficult to know what is executed when.
It definitely takes time and effort to learn and understand how Gradle works. All the task-like methods (‘delete’, ‘copy’, etc.) execute immediately when invoked. Hence they always have to go into a task action (‘doFirst { … }’ or ‘doLast { … }’).