Hi
I would like to speed up the build by deploying (with a system call to scp) in parallel threads, but I need to notify the user at the end of the build if any of the deployments failed. (It’s not enough to print an error message to the console in the middle of the build because that would force the users to watch the entire thing.) There is an afterEvaluate life cycle method but no afterExecute. Is there something else I can hook into?
Why not simply fail the build when a deploy didn’t work?
Can you provide some more information? How are you going to structure these scp calls? In a Task?
@Leonard: The would be optimal, but the system call to “scp” happens in a new new thread. An execption in that thread is just written to the console and the main thread continues. At least when I rethrew the execption that occured.
@Luke: My ‘deploy’ task takes a closure that starts a thread to run the “scp” and “ssh” system commands. A semaphore limits it to 10 concurrent threads (The default max number of concurrent scp calls on my system).
The code is below. The action is factored out into a method because other sub-projects have different source and destination name for the deployments.
...
configure(ears) {
....
task deploy(dependsOn: 'ear') {
outputs.upToDateWhen(deployUpToDateWhen(earPath, earName))
} << deployAction(earPath, earName)
}
...
// Utilities
semaphore = new Semaphore(10, true)
deploymentErrors = Collections.synchronizedList([])
shas = [:]
def deployAction(path, name) {
{ ->
if(semaphore.tryAcquire(180, TimeUnit.SECONDS)) {
new Thread().start {
try {
"scp -q $path $jbHost:$jbDir/$name~".execute()
"ssh $jbHost mv -f $jbDir/$name~ $jbDir/$name".execute()
} catch(Exception e) {
def message = "Deployment of $path failed. $e"
System.err.println message
deploymentErrors << message
// Rethrowing here doesn't stop the build
} finally {
semaphore.release()
}
}
} else {
throw new RuntimeException("Timed out waiting for a sempahore while deploying")
}
}
}
def deployUpToDateWhen(path, name) {
// Saves about 1 second of each deployment
{ task ->
def shaSrc = "openssl sha $path".execute().text.replaceAll(/.*= /, '').trim()
def shaDst = getDestSha(shas, name) // "ssh $jbHost openssl sha $jbDir/$name".execute().text.replaceAll(/.*= /, '').trim()
shaSrc == shaDst
}
}
def getDestSha(shas, module) {
if(!shas) {
// Cache destination shas (saves ca. 0.15 seconds per module)
def text = "ssh $jbHost cd $jbDir; openssl sha [0-9]*" .execute().text
(text =~ /SHA\(([^\)]*)\)= (.*)/).each { whole, name, sha ->
shas[name] = sha.trim()
}
}
return shas[module]
}
How about using GPars (http://www.gpars.org/0.12/guide/guide/3.%20Data%20Parallelism.html#3.1%20Parallel%20Collections) parallel collections according to the api the first exception is re-thrown in the calling thread and you can set the executor size too.
@Leonard: I considered GPars very briefly, but I have no experience with it and I didn’t immediatly see a way to limit the number of concurrent threads to 10 or how to apprach it when there is no collection to give to GPars, so a semaphore seemed a quicker solution. But, it is an argument from ignorance (of GPars).
Couldn’t you restructure your code to create a collection of deployables an then use gpars to deploy them?
You’re going to be fighting an uphill battle trying to parallelise execution, but it might work.
You want org.gradle.api.invocation.Gradle:buildFinished(groovy.lang.Closure) (You can get the gradle instance via org.gradle.api.Project:gradle)
@Leonard: Probably I could. I would like to know the answer to the question anyway since I think could be useful in other cases. For example to display some sort of statistics after the build. Gradle is such a general framework that I’ld be surpriced if there isn’t a way to do this.
@Luke: I agree in general, but since ‘deploy’ never has dependent tasks, it should work. Thanks for the answer. Seems like what I want.