How do you run multiple javaexec tasks in parallel?


(Tristram Coffin) #1

I’m attempting to run several javaexec processes in parallel (same project). I have a sample project on github with my attempts (outlined below), at https://github.com/theaberrant/gradle-concurrent

The end goal is to actually run Cucumber tests in parallel - running each feature file as a separate process (or thread?). That’s another project though, and this is the scaffolding to make that happen :slight_smile:

build.gradle:

apply plugin: 'groovy'
  repositories {
    mavenCentral()
}
  dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.1.0'
}
  task runScript (dependsOn: 'classes', type: JavaExec) {
    main = 'myscript'
    classpath = sourceSets.main.runtimeClasspath
}
  task runConcurrently () {
    logger.quiet "Running task"
    ['1000', '2000', '1000', '3000'].each {
        project.ext.sleepTime = it
        logger.quiet "runScript with ${project.ext.sleepTime}"
        runScript
    }
}

myscript.groovy (in src/main/groovy):

Integer sleepTime = (args[0] as Integer ?: 500)
  println "Sleeping ${sleepTime}ms"
sleep(sleepTime)
println "I have awoken!"

Right now I get them all to run, however not in parallel. Additionally, I can’t figure out how to pass the ‘project.ext.sleepTime’ into the runScript task.

Any help is greatly appreciated. It’s a bit of a unique need, and I’m still learning gradle, so my google searches have been fruitless (although I will admit my search queries are probably too vague).

If it matters, I’m running gradle version 1.4:

$ gradle --version
  ------------------------------------------------------------
Gradle 1.4
------------------------------------------------------------
  Gradle build time: Monday, January 28, 2013 3:42:46 AM UTC
Groovy: 1.8.6
Ant: Apache Ant(TM) version 1.8.4 compiled on May 22 2012
Ivy: 2.2.0
JVM: 1.7.0_13 (Oracle Corporation 23.7-b01)
OS: Windows 7 6.1 amd64

Thanks in advanced,

Tris


(Matias Bjarland) #2

Perhaps something along the lines of the following would help?

import groovyx.gpars.GParsPool
  ext.startTime = System.currentTimeMillis()
  buildscript {
   repositories {
     mavenCentral()
  }
    dependencies {
     classpath "org.codehaus.gpars:gpars:1.1.0"
  }
  }
  def concurrentMethod(int index, String threadName) {
  println "thread $threadName with index $index at ${System.currentTimeMillis() - startTime}"
 }
  String getThreadName(names) {
  String currentName = Thread.currentThread().name
  if (!names.contains(currentName)) names << currentName
    "t${names.indexOf(currentName).toString().padLeft(2, '0')}"
}
    task run << {
  def cores
 = Runtime.runtime.availableProcessors()
   def threads = 10
  println "
  > Using $threads threads on $cores cores..."
  def names = []
  GParsPool.withPool(threads) {
    (1..100).eachParallel {
       def name = getThreadName(names)
      concurrentMethod(it, name)
     }
  }
}
  task wrapper(type: Wrapper) {
  jarFile = 'wrapper/gradle-wrapper.jar'
  gradleVersion = "1.8"
}

This uses (the most excellent) groovy parallel systems library gpars to make parallel executions more beautiful. I threw this into a git repo at:

https://github.com/mbjarland/gradle-concurrent-execution

where you can run the parallelized flow with:

$ ./gradlew run

Hope that helps.


(Peter Niederwieser) #3

It isn’t currently possible to run multiple tasks in the same project in parallel, but you can write a task that internally spawns multiple threads (e.g. as shown in the post above) and uses the ‘javaexec’ method instead.

As for the Cucumber tests, I thought that the worst problems have been sorted out and that they can now be run with Gradle’s regular ‘test’ task (as JUnit tests). If that’s the case, then you just need to set ‘test.maxParallelForks = n’.


(Tristram Coffin) #4

Thanks all for the replies. I haven’t tried the first solution yet, but I think that does what I need it to do…I’ll test it sometime this weekend.

GRADLE-2615 is still open and unresolved, so I believe that it’s still not possible to run cucumber as a straightup jUnit test. I’ll experiment with that as well, since that would be the best option (although I may have to adjust the junit output for my purposes).

I’ll update once I’ve had a chance to investigate further.


(Tristram Coffin) #5

I got the code Matias supplied working, however I can’t figure out how to run a groovy script, rather than just println statement.

I’ve updated my repo with my attempt at integration with the javaexec. I’m getting the following error:

17:33:51.240 [ERROR] [system.err] Exception in thread "main" Caused by: java.lang.ClassNotFoundException: groovy.lang.Script
17:33:51.244 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: FAILED
17:33:51.244 [ERROR] [system.err]
     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
17:33:51.248 [INFO] [org.gradle.process.internal.DefaultExecHandle] Process 'command 'C:\dev\tools\Java\jdk1.7.0_13\bin\java.exe'' finished with exit value 1 (state: FAILED)
17:33:51.251 [ERROR] [system.err]
     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
17:33:51.258 [ERROR] [system.err]
     at java.security.AccessController.doPrivileged(Native Method)
17:33:51.260 [ERROR] [system.err]
     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
17:33:51.264 [ERROR] [system.err]
     at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
17:33:51.267 [ERROR] [system.err]
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
17:33:51.270 [ERROR] [system.err]
     at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
17:33:51.274 [ERROR] [system.err]
     ... 13 more
17:33:51.277 [ERROR] [system.err] Exception in thread "main"

My guess is that this is classpath related, however I’m not really sure how to resolve it. Any suggestions are appreciated :slight_smile:


(Matias Bjarland) #6

AFAIK executing an extrernal groovy script from a gradle build file is best done via groovy. Should perhaps be noted that since gradle adds its own DSL to the groovy syntax, it is when possible preferable to write any build logic using the gradle DSL rather than executing external groovy scripts.

With that being said, the following modification should execute your external groovy script in parallel using gpars:

def concurrentMethod(int index, String threadName) {
    Binding binding = new Binding();
    binding.variables['args'] = [index]
        binding.variables['threadName'] = threadName
          def shell = new GroovyShell(binding)
          def groovySrc = file('src/main/groovy/myscript.groovy')
    Object result = shell.evaluate(groovySrc)
}

I created a pull request on your repo demonstrating the above code.

Hope that helps.


(Tristram Coffin) #7

Matias - that works great for the code I supplied, unfortunately for my other project I still need to run a javaexec command.

Basically my goal is to run a cucumber task, which looks similar to:

task cucumber() {
    dependsOn assemble
    doLast {
        javaexec {
            main = "cucumber.api.cli.Main"
            classpath = configurations.cucumberRuntime
            args = ['-f', 'pretty', '--glue', 'src/test/groovy', 'src/test/resources']
        }
    }
}

I’d need to be able to pass a specific file name to the args as well, specifying which feature file to run. I was able to figure out how to build that as a file list:

def features = fileTree(dir: 'src/test/resources').include '**/*.feature'

I’ve branched my cucumber-jvm-groovy-rest-example repo to ‘concurrent’ with my attempts at calling the cucumber task (using cucumber.execute(), however that’s returning an error:

* What went wrong:
Execution failed for task ':run'.
> java.lang.IllegalStateException: Cannot start long running operation, as the task artifact state cache (C:\dev\projects\cucumber-jvm-groovy-rest-example\.gradle.4\taskArtifacts) has not been locked.

I’m guessing this is part of the lack of concurrency support for javaexec task :frowning:

All this help is definitely much appreciated. I’m really close, just need to a) figure out how to call a java exec via method, and b) pass parameters to said method to be able to specify arguments passed to the java class.

Thanks again for all the help, let me know if anything I’ve said is confusing.


(Tristram Coffin) #8

I pushed another solution using Process to the gradle-concurrent repo. This allows me to run any java application, although likely has issues with environment configs, etc. I wasn’t able to get a JavaExec task working :frowning:

I’m having issues with Cucumber not closing, however I’ll bring that up to the cucumber forums since it’s unrelated to Gradle.

Thanks all for the help - if there are better ways to do the same thing let me know, but otherwise I have something that works for my needs :slight_smile:


(Tristram Coffin) #9

In case anyone comes across this with the goal of running cucumber features concurrently via gradle, I’ve posted to the cukes google group: https://groups.google.com/forum/#!topic/cukes/neLoyCmosGI


(Ranjit Singh) #10

I have one question we can run parallel test from gradle

  1. I am running test class which calls mine cucumber tag 2. So with the above code provided i am able run the scripts in parallel, but it calls my test classes randomly . how can i specify which test classes i want to run in parallel?

Thanks in advance