Parallel task execution supported in gradle?


(gradle-shorn) #1

The gradle task I’ve partially listed at the end of the post is intended to read an eclipse classpath and copy the various jar files and other compile / runtime files necessary for building a project that I’m working on (whose build structure and process I have no control over, so please hold the “you’re doing it wrong” feedback).

I’ve gotten lots of different errors - NullPointerExceptions, ConcurrentModificationExceptions, IllegalStateException and probably others I can’t remember. I’ve tried both milestone-5 and milestone-3 (I’ve never had M5 succeed fully, but I’ve seen M3 get through the full list of files twice out of about 10 runs or so). I’ve tried with “-C rebuild”, I’ve tried deleting both $HOME/.gradle and the project .gradle cache directories (it actually seemed like M3 was more likely to actually finish after I’d cleaned out the caches, but nothing I can reproduce reliably).

So, my questions: * 1) the sheer variety of different kinds of exceptions indicates to me that this may be totally unsupported, is that because ** 1.1) parallel task execution is just not supported? ** 1.2) parallel task execution is supported, but I’m doing it wrong (i.e. using gpars to tall Task.execute() methods on different threads)? Is there a proper gradle API way to do this or something? * 2) It seems like it always fails in the up-to-date checking - is there some way I could skip that part with the aim of not constantly reading and writing to the gradle cache, thus skipping the contention issues? * 3) parallel execution aside, is the serial execution branch correct? i.e. creating the tasks via the Project.task() method and then calling their execute() method - is there a better or more “gradle”-ish way to do it? * 4) I’m very new to gradle and groovy, so I’m going crazy using different features and what-not - any feedback or recommendations you’d like to provide on better ways to do things would be cheerfully accepted :slight_smile:

task copyJarFiles() << {
  File jcopyDestDir = new File(jcopyDest)
    println "destination directory: [${jcopyDestDir}]"
  println "libRootPath: [${libRootPath}]"
  println "copyFromBaseDir: [${copyFromBaseDir}]"
    List copyTasks = []
  forEachLibEntryInEachClasspath {
eclipseLib ->
    copyTasks << createCopyLibTask(eclipseLib.@path.text() as File, jcopyDestDir)
  }
    println "starting execution of ${copyTasks.size()} copy tasks in ${parallelCopy?"parallel":"serial"}"
  if( parallelCopy ){
    GParsPool.withPool {
      copyTasks.eachParallel{ Copy copyTask ->
        println "thread [${Thread.currentThread().name}] executing task [$copyTask.name]"
        copyTask.execute()
      }
    }
  }
  else {
    copyTasks.each{ it.execute() }
  }
  println "finised execution"
}
private Copy createCopyLibTask(File libPath, File jcopyDestDir) {
  File libPathDir = libPath.parentFile
  String libEntryDirStructure = libPathDir.path.substring(libRootPath.length())
  File copyFromDir = getCopyFromDir(libPathDir, copyFromBaseDir, libEntryDirStructure)
  assert copyFromDir.exists(): "[$copyFromDir] does not exist"
  File copyToDir = new File(jcopyDestDir, libEntryDirStructure)
    println "path to process: [$libPath], libPathDir: [$libPathDir]"
  println "libEntryDirStructure: [$libEntryDirStructure]"
  println "copyFromDir: [$copyFromDir]"
  println "copyToDir: [$copyToDir]"
    return task("copy lib jars $libPath", type: Copy) {
    from copyFromDir
    into copyToDir
    include { currentFile -> isCompileRunFile(currentFile.file.name) }
    eachFile { FileVisitDetails i -> println "${->name}: copying $i.file.name" }
  }
}

Edit: maaaany edits to get the code listing right.


(Peter Niederwieser) #2

Parallel task execution is not yet supported but planned for post-1.0. Executing a task from another task by calling its execute() method is not supported in general. The only supported way is to use a dependsOn relation. This won’t give you parallel task execution today but it will in the future.


(gradle-shorn) #3

Parallel task execution is not yet supported but planned for post-1.0

Fair enough.

Executing a task from another task by calling its execute() method is not supported in general. The only supported way is to use a dependsOn relation.

Should I be creating the copy tasks in the configuration phase? But then how do I hook them up to execute? Just have an empty copyJarFiles task and stuff all the copy tasks into the dependsOn list in the configuration phase?

Perhaps I shouldn’t be using tasks for this kind of thing at all? In the end I suppose I can just use ant.copy in the execution phase - just doesn’t seem very gradle-ish. Plus I won’t get gradle’s nicer up-to-date processing for free, or the ability to decorate the tasks with a ProgressLogger.

Maybe I’m just trying too hard to find nails for my new hammer :slight_smile:


(Peter Niederwieser) #4

Should I be creating the copy tasks in the configuration phase? But then how do I hook them up to execute? Just have an empty copyJarFiles task and stuff all the copy tasks into the dependsOn list in the configuration phase?

Yes. However I’d start out with only a few Copy tasks with “natural” boundaries (e.g. one task for copying all Eclipse libs). Don’t optimize prematurely. Copying many files in parallel could as well result in a slowdown.

Perhaps I shouldn’t be using tasks for this kind of thing at all? In the end I suppose I can just use ant.copy in the execution phase

I recommend to use a Copy task instead of ant.copy except when you need a particular ant.copy feature that’s not (yet) supported by Copy tasks.