Build maven dependencies automatically before build

[zufarfakhurtdinov]

Hi all! I have a big maven multi-module project. I develop applications for multiple platforms: gwt, android, robovm(ios).
I want to migrate to gradle for android modules. So, I want to build multi-platform dependency modules.
And I cannot just migrate all modules to gradle. It need a lot of time.
I see some ways:

1. create simple parent pom with contain all maven modules with paths.
   Create gradle exec task to invoke something like mvn install -f PATH_TO_PARENT -am -pl :dep1, :dep2, :dep3
   Execute it task before each build.
   Problems: I need to filter local dependency and external dependencies. Duplicate each local dependency with custom configuration?
2. use parent pom. Generate new pom with dependencies and invoke mvn install -f PATH_TO_PARENT -am -pl :my-pom-with-dependency-list
   Problems: yet another semi-temporary module.
3. duplicate each local dependency pom.xml with build.gradle config
   Problems: Time consuming and I think my ide(intellij idea) will explode on that config.

[st_oehme (Stefan Oehme)]

If I understand you correctly, your problem is that you need to know which Maven modules need to be built, right?

@zufarfakhurtdinov

Duplicate each local dependency with custom configuration?

Why duplicate them? Your Maven projects should have a groupId that should allow you to tell them apart from other external dependencies.

[zufarfakhurtdinov]

@st_oehme

If I understand you correctly, your problem is that you need to know which Maven modules need to be built, right?

Yes, it’s first problem. Second problem is how to execute maven build before dependency resolution in gradle?

@st_oehme

Why duplicate them? Your Maven projects should have a groupId that should allow you to tell them apart from other external dependencies.

I can create custom configuration:
configurations {
localMaven
}
dependencies {
compile ‘mygroup:myLocal:1.0’
localMaven ‘mygroup:myLocal:1.0’
}
task installMavenDeps(type: Exec) {
def mvnDeps = configurations.localMaven.allDependencies
.findAll { it.name != “unspecified”}
.collect { dep → “:” + dep.name }
.join(“,”)
environment System.env
commandLine “mvn”, “install”, “-f”, “…”, “-DskipTests”, “-am”, “-pl”, “$mvnDeps”
}
But I think that approach uncomfortable and error-prone.

upd. I understand. Yes, I think I can filter required dependencies by groupId. But I have local maven dependencies(sources) and dependencies from remove private maven repository(jars). They can have same groupId. I can rename group for remote dependencies or add additional filter on artifactId.

[st_oehme (Stefan Oehme)]

@zufarfakhurtdinov

I can rename group for remote dependencies or add additional filter on artifactId.

A popular convention is to have a unique groupID per multi-module project in Maven. If you can change that, then go ahead. Otherwise filter on artifactId.

@zufarfakhurtdinov

Second problem is how to execute maven build before dependency resolution in gradle?

Gradle does not do dependency resolution up-front. So as long as your Maven build task runs before any other task that wants to resolve dependencies (like the Java compile task), you’ll be fine.

[zufarfakhurtdinov]

Thank you!
I have a gradle parent module, and several child gradle modules. Child modules have different dependencies.
And copy-paste installMavenDeps task and invoke it (with different arguments) every time doesn’t seem efficient.
How can I write task one time? And invoke that task one time with set of all needed dependencies?

[st_oehme (Stefan Oehme)]

Please show as an example of what you currently do and explain what you are not sure about in a little more detail

[zufarfakhurtdinov]

Example:

parent module:
allprojects {
ext {
isLocalDep = {
it.name != “unspecified” &&
!it.name.contains(“mybadname”) &&
it.group.startsWith(“mygroup”)
}
createMvnCommand = { String deps ->
def cl = []
if (System.getProperty(‘os.name’).toLowerCase().contains(‘windows’)) {
cl += [‘cmd.exe’, ‘/c’];
}
cl += [“mvn”, “install”, “-f”, “…”, “-DskipTests”, “-am”, “-pl”, “$deps”]
return cl
}
}

child1:
apply plugin: ‘com.android.application’

dependencies {
compile ‘mygroup:dep:1’
}

task installMavenDeps(type: Exec) {
def mvnDeps = configurations.compile.allDependencies
.findAll { isLocalDep(it) }
.collect { dep -> “:” + dep.name }
.join(",")
onlyIf {
!mvnDeps.isEmpty()
}

environment System.env
commandLine (createMvnCommand(mvnDeps))
}

configurations {
res
}
dependencies {
res(“mydata:myBinaryResourceJar:1.0”) {
transitive = false
}
}

tasks.withType(JavaCompile) {
compileTask ->
compileTask.dependsOn installMavenDeps
compileTask.dependsOn processData
}

android.applicationVariants.all { variant ->
configurations.res.each {
jar ->
copy {
includeEmptyDirs false
from zipTree(jar)
into variant.javaCompile.destinationDir
include “**/*.class”
}
}
}

task processData {
def to = tempRes + “/raw”;
file(to).mkdirs()

configurations.res.each {
jar ->
copy {
includeEmptyDirs false
from { zipTree(jar) }
into to
include “/*.jet"
eachFile { FileCopyDetails fcp ->
// remap the file to the root
String[] segments = [fcp.getName()]
println "processFile: flatten ${fcp.relativePath}"
fcp.relativePath = new RelativePath(true, segments)
}
}
copy {
includeEmptyDirs false
from { zipTree(jar) }
into tempPath
include "assets/”, “res/**”
}
}
}
child2,child3,child4(I think there will be around 6 similar apps) are mostly copy of child1. Dependencies and android settings are different. Maybe other differences.

There are several problems:

1. I have a resource jars. Some of they contains custom binary format files and compiled accessors to they ( .class files ).
   We need add .class to classpath and copy binary data to res/raw.
2. Some of they just contains assets: icons, fonts, etc. But packaging is jar. Can I don’t use handwritten copy task?
3. android.applicationVariants.all {} and processData {configurations.res.each…} invoke before installMavenDeps. So if I don’t have dependency in local maven, then build just failed. It’s BIG problem.
4. If I want build several modules, then Maven build will invoke for every project. Even all maven modules are up-to-date it cost 3 to 10 seconds every time. It’s significant slowdown.
5. There are lot of copy-paste. Can I write installMavenDeps, processData tasks once?

[st_oehme (Stefan Oehme)]

I see what you mean. You’ll probably have to do something like this

• put the installMavenDeps task in the root project
• iterate over all the sub projects and look which Maven deps they will need, add that to the list of Maven projects to build
• make all tasks that need those Maven deps depend on the installMavenDeps task

There are some big problems in your build script, which are probably causing you much of your problem:

@zufarfakhurtdinov

task processData {
def to = tempRes + “/raw”;
file(to).mkdirs()

configurations.res.each {

This will resolve the res dependencies at configuration time and copy the files at configuration time. You want this to happen at execution time. Wrap the logic in a doLast block and add the res configuration to the task`s inputs, so it knows when to re-run. Roughly like this:

task processData {
  inputs configurations.res
  doLast {
    // copy stuff
  }
}

The same goes for this:

@zufarfakhurtdinov

android.applicationVariants.all { variant -> configurations.res.each {

Create a task for this.

@zufarfakhurtdinov

def mvnDeps = configurations.compile.allDependencies .findAll { isLocalDep(it) } .collect { dep -> “:” + dep.name } .join(“,”)

This will eagerly walk over the maven dependencies, which will not work. You need to do this lazily when you build the argument string for the Maven execution. Use Groovy’s GString freature for that: "${lazy.computation.here}". Or wrap that part of the configuration in an afterEvaluate block.

[zufarfakhurtdinov]

I solved it somehow.
installMavenDeps becomes ugly. I do echo by default, but it’s ugly. And will not work on windows. I can check os.name, I know. installMavenDeps onlyIf doesn’t work too, I’ve needed to add doFirst with hack.

ext {
createMvnCommand = { String deps ->
  def cl = []
  if (System.getProperty('os.name').toLowerCase().contains('windows')) {
    cl += ['cmd.exe', '/c'];
  }
  cl += ["mvn", "install", "-DskipTests", "-am", "-pl", "$deps"]
  return cl
}
mavenDeps = new HashSet<String>()
}

task installMavenDeps(type: Exec) {
//  onlyIf {!rootProject.mavenDeps.isEmpty() && !project.hasProperty('skipMvn')}

environment System.env
commandLine "echo", "installMavenDeps: skip"
doFirst {
  if (!rootProject.mavenDeps.isEmpty() && !project.hasProperty('skipMvn')) {
    commandLine createMvnCommand(mavenDeps.join(","))
  }
}
}

subprojects {
configurations {
  res
  compile
}

afterEvaluate {
  tasks.withType(JavaCompile) {
    compileTask -> compileTask.dependsOn ":installMavenDeps"
  }
  if (plugins.hasPlugin('com.android.application')) {
    tasks.withType(JavaCompile) {
      compileTask ->
        compileTask.dependsOn ":scene-graph-android:buildJni"
        compileTask.dependsOn processData
    }
  }
}

task writeDeps << {
  def allDependencies = new HashSet()
  allDependencies.addAll(configurations.compile.allDependencies)
  allDependencies.addAll(configurations.res.allDependencies)
  def a = allDependencies
      .findAll { isLocalDep(it) }
      .collect { dep -> ":" + dep.name }
  rootProject.mavenDeps.addAll(a)
}
rootProject.installMavenDeps.dependsOn(writeDeps)

task copyDataClasses << {
//  copy constellation data *.class to each variant
  android.applicationVariants.all { variant ->
    configurations.res.each {
      jar ->
        copy {
          includeEmptyDirs false
          from zipTree(jar)
          into variant.javaCompile.destinationDir
          include "**/*.class"
        }
    }
  }
}

task processData << {
  def to = tempRes + "/raw";
  file(to).mkdirs()

  configurations.res.each {
    jar ->
      logger.info "processData: jar flatten .jet to ${to}"
      copy {
        includeEmptyDirs false
        from { zipTree(jar) }
        into to
        include "**/*.jet"
        eachFile { FileCopyDetails fcp ->
          // remap the file to the root
          String[] segments = [fcp.getName()]
          fcp.relativePath = new RelativePath(true, segments)
        }
      }

      copy {
        includeEmptyDirs false
        from { zipTree(jar) }
        into tempPath
        include "assets/**", "res/**"
      }
  }
}
}

child:

apply plugin: 'com.android.library'

dependencies {
compile "mvnlocal:mymodule"
}

android {
  compileSdkVersion 23
  buildToolsVersion "23.0.2"
  lintOptions {
    abortOnError false
  }
}

And after all it doens’t work because android resolve dependency before installMaven execution.

Cannot change dependencies of configuration ‘detachedConfiguration1’ after it has been resolved.

Can I find who invoke dependency resolving? If in child use “apply plugin:java” then all work perfectly.

[st_oehme (Stefan Oehme)]

Sorry, I won’t have time to go through that wall of code Please try to trim your example down for further discussion. Below is a simple example of what I think you need:

//only for demonstration, yours needs to extend Exec
class InstallMavenDependencies extends DefaultTask {

  @Input
  List<String> artifactIds = []

  @TaskAction
  def install() {
    println(artifactIds)
  }
}

task installMavenDependencies(type: InstallMavenDependencies)

subprojects {
  apply plugin: 'java'
  configurations.compile.dependencies.all { dep ->
    //do any filtering you need here
    rootProject.installMavenDependencies.artifactIds << dep.name
  }
  compileJava.dependsOn(rootProject.installMavenDependencies)
}

This successfully prints all dependencies for me before the compile task is executed. Of course, you’ll have to find out which of the Android tasks resolves dependencies first and make sure you run before that.

[zufarfakhurtdinov]

ok, I did custom installMavenDeps task.
Your approach works for ‘java’ subprojects perfectly.
but if I use

apply plugin: ‘com.android.library’

there is

Cannot change dependencies of configuration ‘detachedConfiguration1’ after it has been resolved.

P.s. I removed all my additional tasks, so there is my config

class InstallMavenDependencies extends AbstractExecTask {
  public InstallMavenDependencies() {
    super(InstallMavenDependencies.class)
  }
  @Input
  List<String> artifactIds = []

  @TaskAction
  void exec() {
    commandLine createMvnCommand(artifactIds)
    super.exec()
  }
}
task installMavenDependencies(type: InstallMavenDependencies)

subprojects {
  apply plugin: 'com.android.library' 

  dependencies {
     compile "mvnlocal:mymodule"
  }
  android {
    compileSdkVersion 23
    buildToolsVersion "23.0.2"
  }
  configurations.compile.dependencies.all { dep ->
  //do any filtering you need here
  rootProject.installMavenDependencies.artifactIds << dep.name
  }
  
tasks.withType(JavaCompile) {
  compileTask ->  compileTask.dependsOn(rootProject.installMavenDependencies)
  }
}

I’m sorry for wall of code, but it’s near to minimal config.

[st_oehme (Stefan Oehme)]

Thank you for cutting it down, It’s much more manageable now and I think I understand the problem: There are just too many points where dependencies are resolved and so your installation task comes too late.

But we can work around this Instead of installing the maven projects, how about referencing their jars directly:

//as before, but don't install, just run 'jar' on the maven projects
task jarMavenDependencies(type: JarMavenDependencies)

subprojects {
  apply plugin: 'java'
  //this basically adds a new function to the `dependencies` block on the sub-projects
  dependencies.ext.mavenProject = { artifactId ->
      //you need to change this to your project structure. 
      //I just assume all projects are under the root 
      //and their folder name equals their artifactId 
      //and they all have the same version.
      files("rootProjectDir/$artifactId/target/${artifactId}-${rootProject.version}.jar").builtBy(rootProject.jarMavenDependencies)
  }
}

And in your subprojects:

dependencies {
  compile mavenProject('foo')
}

The builtBy information will make sure that the maven projects are built before any task that wants to resolve them.

[zufarfakhurtdinov]

What does JarMavenDependencies mean?
If it ordinary maven invocation, it will be slow. I want invoke maven once or never.
Now I think that duplicate build logic with gradle(in addition to maven) can be solution(despite of error-prone and supporing two build engines).

[st_oehme (Stefan Oehme)]

JarMavenDependencies is the same as in the previous answer, we had already talked about it It will only be in the root project and executed once.

[zufarfakhurtdinov]

Failed with compile errors because my maven module have own dependencies.

[zufarfakhurtdinov]

Are there ways to solve it?
Can I handle which task invoke dependencies resolve?

[st_oehme (Stefan Oehme)]

I think the best thing you can do is use my approach above and also parse the pom.xml files to get the transitive dependencies.

[zufarfakhurtdinov]

What about transitive dependencies of local maven modules?
So:

1. parse all local maven pom.xml include transitive
2. dynamically add all transitive dependencies with mavenProject … builtBy
3. Profit?
   Do I understand correctly?

[st_oehme (Stefan Oehme)]

Yes, that’s exactly what I mean

Please note though: If anyone does dependency resolution before the execution phase, then you will still be out of luck.

[zufarfakhurtdinov]

Thank you very much! I’ll try it.
But how can I find which task resolve my dependencies? And when?
add DependencyResolutionListener, throw exception in beforeResolve() and read stacktrace?))