Guidance on using Gradle for building a legacy project

plugins

(Dave Barnett) #1

We have a legacy application and legacy build tool we are using to build both our native and Java code. For Java, we tied “JMake” into our build tool, however, it is no longer maintained, does not support JDK 9 and above, and we’re needing to move to Java 9 / 10 / 11.

In our environment, we have “baseline” builds (with full source code) and the developers use “sparse work areas” (only the code they are changing).

I’ve figured out how to get Gradle to find the source files (by giving the required srcDirs and using include “seta/setb/foo/**/*.java”) and that works okay, but when I give both the baseline src area and the developer src are, I get complaints about classes being duplicated.

So, it seems to me that I need to write a plugin that adapts the “java” plugin so that it understand our model and only takes the first java file with a given name that it finds. In other words, when given these areas to search for source files, with “MyFile.java” in both, only the developer’s file should be used:
/developer/src/com/slb/test/MyFile.java <== should be used
/baseline/build-01/src/com/slb/test/MyFile.java <== should be ignored

[it is possible to have multiple layers, all of which may have a MyFile.java; in all cases only the “first” should be used. e.g. /developer/src, /baseline/build-03, /baseline/build-02, /baseline/build-01 – effectively stacked incremental build areas]

  1. am I correct that I need a special plugin for this because I cannot just give the java plugin a list of files to build?
  2. if so, can I extend the “java” plugin?
    … or must I duplicate all the code from it and then make the changes I need? [I really don’t want to do that]

Alternatively, we already have an Ant-based class that understands the above “search path” model (called GFileSet based on Ant’s FileSet), but I don’t see how I can get this to the java plugin so it will be used instead of the conventional “include **/*.java”.

Thank you


(Dimitar Dimitrov) #2

I assume that the “sparse work areas” are to reduce compilation times. You’ll likely find that Gradle’s change detection and dependency tracking eliminate this need.

Just try to compile all the sources, change a single file and recompile. If the time is unacceptable, do not hesitate to post to this forum.

One more thing, unlike Ant and Maven, Gradle tries hard to figure inter-class dependencies and recompile the impacted files (not sure if it works for primitive constants changing), but in general I have rarely seen a case when I have needed to do a clean build.


(Dave Barnett) #3

Yes, that’s part of it, but the issue is we have tons of code (10GB in our src area) and so having a complete copy of “src” locally is not feasible in our environment. Our legacy build tool swallows all of that and builds it all, in order, in one pass.

Our Java code is interspersed with C, C++, Fortran, Fortran90, Perl, and Python among others which I’m probably forgetting.

Replacing our legacy build tool with Gradle isn’t an option right now, and may never be. What I was looking to do was use Gradle to handle Java (due to the points you made about it) and leave the rest to our existing tools…

Our existing tool can be made to handle Java, but it is a very dumb “oh, you changed Foo.java, so I will rebuild it as Foo.class” but does not take into account the need to rebuild dependents, too – which is why we were using Jmake instead.

I don’t like this idea, but I guess I could rsync the required code – after doing the filtering we require – into “src/main/java” and then just use the defaults; surely there’s a better way though…

Thank you


(Dave Barnett) #4

It looks like “Plugin Composition” may allow me to customize the “java” plugin to do what I need it to do. Still investigating, but that seems more promising than maintaining my own fork of the java plugin.


(uklance) #5

Since the java plugin is driven by SourceSet and SourceDirectorySet I think you’ll manually configure the java tasks rather than using the java plugin. Possibly something like

apply plugin: 'base' 

configurations {
   compile
   runtime { extends configurations.compile } 
} 
dependencies {
   compile 'com.foo:bar:1.0'
}
FileTree tree3 = fileTree('src/level3/java')
FileTree tree2 = fileTree('src/level2/java').matching {
   exclude { FileTreeElement el ->
      return file("src/main/level3/java/$el.path").exists() 
   } 
}
FileTree tree1 = fileTree('src/level1/java').matching {
   exclude { FileTreeElement el ->
      return file("src/main/level3/java/$el.path").exists() ||
           file("src/main/level2/java/$el.path").exists()
   } 
} 
task compileJava(type: JavaCompile) {
   source = tree1.add(tree2).add(tree3)
   classpath = configurations.runtime
}
task test(type:Test) { ... } 
assemble.dependsOn compileJava
check.dependsOn test

(Dimitar Dimitrov) #6

See Lance’s reply for how to customize the compile task.

If you have many filesets, you may want to express them declaratively in some map structure and use Task Rules to create the compile tasks on demand.

This way, adding a new “working set” becomes as simple as adding another key/value in the map, which would probably live in its own build file or be loaded from some data (json, csv, or groovy config slurper).


(uklance) #7

I was thinking this could be turned into a plugin to abstract the complexity.

Eg:

apply plugin: 'com.foo.mylegacyjava'

myLegacyJava {
    sourceSet('level1') {
       srcDir 'src/level1/java'
    )
    sourceSet('level2') {
       srcDir 'src/level2/java'
       overrides 'level1'
    }
    sourceSet('level3') {
       srcDir 'src/level3/java'
       overrides 'level2' 
    }
} 

(Dave Barnett) #8

Thanks, let me have a try with this and see how I get along.


(Dave Barnett) #9

I started a new project area, ran “gradle --init” and updated build.gradle with contents as above, changing “src/main/…” entries to match our source code structure.

When I run, I get:

startup failed:
build file ‘…/master_build_2019_2/build.gradle’: 11: unexpected token: extends @ line 11, column 14.
runtime { extends configurations.compile }
^

1 error

Perhaps I’m missing a plugin? or I’ve not understood where to put the above…?

Thank you.


(uklance) #10

That’s untested code as you’ve found out.

It’s extendsFrom not extends

See Configuration.extendsFrom(…)


(Dave Barnett) #11

Thanks, that gets me one step further.

Afterwards, I get:
A problem occurred evaluating root project ‘master_build_2019_2’.

Directory ‘/baselines/src_extern_004’ does not allow modification.

It’s complaining about this line:
source = tree1.add(tree2).add(tree3

I’ve not yet figured out how to concatenate FileTree objects. Any hints?

Thanks.


(uklance) #12

Maybe

source = tree1.plus(tree2).plus(tree3) 

See FileTree.plus(…)


(Dave Barnett) #13

That changes removes the complaint, but now yields:
Caused by: java.lang.NullPointerException

org.gradle.api.tasks.TaskExecutionException: Execution failed for task ‘:compileJava’.
at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:49)
at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)

perhaps either source isn’t getting set, or one [or all 3?] of my “tree” objects have no content…?

I get similar NPE from using source = tree3

EDIT: For what it’s worth, “tree3” isn’t empty; I did an each / println on it.


(uklance) #15

Perhaps try a small standalone project first with src/level1/java and src/level2/java etc with a couple of java files to test the override functionality. If you commit it to github I might even take a look if you can’t get it working.

Also try adding --stacktrace to the command line to see the full trace.


(uklance) #16

Here’s a proof of concept I threw together

task makeFiles {
 doLast {
  def makeMap = [
   level1: ['A', 'B', 'C', 'D', 'E', 'L1'],
   level2: ['A', 'B', 'L2'],   
   level3: ['A', 'L3']
  ]
  makeMap.each { level, simpleNames ->
   simpleNames.each { simpleName ->
    File javaFile = file("src/$level/java/com/foo/${simpleName}.java")
    javaFile.parentFile.mkdirs()
    javaFile.text = 
"""package com.foo;
public class $simpleName {
 public String getValue() {
  return \"${level}${simpleName}\";
 }
}
"""    
    logger.lifecycle "Created $javaFile" 
   }
  }
 }
}

task printTrees {
 dependsOn makeFiles
 doLast {
  FileTree tree3 = fileTree('src/level3/java')
  FileTree tree2 = fileTree('src/level2/java').matching {
   include { FileTreeElement el ->
    return el.directory || !file("src/level3/java/${el.path}").exists()
   }
  }
  FileTree tree1 = fileTree('src/level1/java').matching {
   include { FileTreeElement el ->
    return el.directory || (!file("src/level3/java/${el.path}").exists() && !file("src/level2/java/${el.path}").exists())
   }
  }
  tree3.plus(tree2).plus(tree1).each {
   println "$it"
  }
 }
}

Output

C:\path>gradle printTrees

> Task :makeFiles
Created C:\path\src\level1\java\com\foo\A.java
Created C:\path\src\level1\java\com\foo\B.java
Created C:\path\src\level1\java\com\foo\C.java
Created C:\path\src\level1\java\com\foo\D.java
Created C:\path\src\level1\java\com\foo\E.java
Created C:\path\src\level1\java\com\foo\L1.java
Created C:\path\src\level2\java\com\foo\A.java
Created C:\path\src\level2\java\com\foo\B.java
Created C:\path\src\level2\java\com\foo\L2.java
Created C:\path\src\level3\java\com\foo\A.java
Created C:\path\src\level3\java\com\foo\L3.java

> Task :printTrees
C:\path\src\level3\java\com\foo\A.java
C:\path\src\level3\java\com\foo\L3.java
C:\path\src\level2\java\com\foo\B.java
C:\path\src\level2\java\com\foo\L2.java
C:\path\src\level1\java\com\foo\C.java
C:\path\src\level1\java\com\foo\D.java
C:\path\src\level1\java\com\foo\E.java
C:\path\src\level1\java\com\foo\L1.java

(Dave Barnett) #17

As far as I can see, the stack trace is saying that “source” is empty.

  • What went wrong:
    Execution failed for task ‘:compileJava’.

java.lang.NullPointerException (no error message)

  • Try:
    Run with --info or --debug option to get more log output. Run with --scan to get full insights.

  • Exception is:
    org.gradle.api.tasks.TaskExecutionException: Execution failed for task ‘:compileJava’.
    at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:49)
    at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
    at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
    at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
    at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
    at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
    at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:136)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.execute(DefaultTaskPlanExecutor.java:201)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.executeWithTask(DefaultTaskPlanExecutor.java:192)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
    Caused by: java.lang.NullPointerException
    at org.gradle.jvm.platform.internal.DefaultJavaPlatform.generateName(DefaultJavaPlatform.java:63)
    at org.gradle.jvm.platform.internal.DefaultJavaPlatform.(DefaultJavaPlatform.java:30)
    at org.gradle.api.tasks.compile.JavaCompile.getPlatform(JavaCompile.java:157)
    at org.gradle.api.tasks.compile.JavaCompile_Decorated.getPlatform(Unknown Source)
    at org.gradle.api.internal.tasks.properties.bean.AbstractNestedRuntimeBeanNode$DefaultPropertyValue$1$1.create(AbstractNestedRuntimeBeanNode.java:83)
    at org.gradle.util.SingleMessageLogger.whileDisabled(SingleMessageLogger.java:200)
    at org.gradle.api.internal.tasks.properties.bean.AbstractNestedRuntimeBeanNode$DefaultPropertyValue$1.get(AbstractNestedRuntimeBeanNode.java:80)
    at com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:125)
    at org.gradle.api.internal.tasks.properties.bean.AbstractNestedRuntimeBeanNode$DefaultPropertyValue.getValue(AbstractNestedRuntimeBeanNode.java:138)
    at org.gradle.api.internal.tasks.properties.annotations.NestedBeanAnnotationHandler.visitPropertyValue(NestedBeanAnnotationHandler.java:46)
    at org.gradle.api.internal.tasks.properties.bean.AbstractNestedRuntimeBeanNode.visitProperties(AbstractNestedRuntimeBeanNode.java:62)
    at org.gradle.api.internal.tasks.properties.bean.RootRuntimeBeanNode.visitNode(RootRuntimeBeanNode.java:32)
    at org.gradle.api.internal.tasks.properties.DefaultPropertyWalker.visitProperties(DefaultPropertyWalker.java:41)
    at org.gradle.api.internal.tasks.TaskPropertyUtils.visitProperties(TaskPropertyUtils.java:39)
    at org.gradle.api.internal.tasks.execution.DefaultTaskProperties.resolve(DefaultTaskProperties.java:77)
    at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:53)
    … 21 more


(Dave Barnett) #18

This works fine. I get the files generated and they are listed in the output as you’ve shown.

I still cannot get "source = " whatever to work using the earlier example. I get the stack trace as above.

EDIT: I’ll work on getting this into a github project based on what I have right now.
EDIT2: Project url is:


(Dave Barnett) #19

After much trial-and-error, I am able to get a uniq list of java files and pass them to the JavaCompile class.

The next hurdle is how to turn this into a jar file. I get a jar file, but it has only the manifest in it.

I’ve not yet figured out the right things to enter for the jar section:
version = “0.0.1”
task myJar (type: Jar) {
manifest {
attributes(‘Implementation-Title’: project.name,
‘Implementation-Version’: project.version)
}
from sourceSets.main.output
include “**/*.class”
}


(uklance) #20

I managed to get it working using the standard java plugin.
See https://github.com/uklance/legacy-java

apply plugin: 'java' 
repositories {
	mavenCentral()
}
dependencies {
   testCompile 'junit:junit:4.12'
}
sourceSets {
	main {
		java {
			FileTree overridden1 = fileTree('src/level1/java').matching {
				include { FileTreeElement el -> el.directory || file("src/level2/java/${el.path}").exists() || file("src/level3/java/${el.path}").exists() }
			}
			FileTree overridden2 = fileTree('src/level2/java').matching {
				include { FileTreeElement el -> el.directory || file("src/level3/java/${el.path}").exists() }
			}
			Set<File> overridden = overridden1.plus(overridden2).files
			srcDirs = ['src/level1/java', 'src/level2/java', 'src/level3/java']
			exclude { FileTreeElement el -> overridden.contains(el.file) }
		}
	}
}

test {
	testLogging.showStandardStreams = true
}

You can see from the test case that the overrides are applied.

Test output

com.foo.MyTest > doTest STANDARD_OUT
    A = level3A
    B = level2B
    C = level1C
    D = level1D
    E = level1E
    L1 = level1L1
    L2 = level2L2
    L3 = level3L3