Up-to-date checks for tasks contributing to source dir subfolder

Help/Discuss
1

I have a situation where in project foo, I need to grab archives (mainly from different projects in the build), expand them in a folder like extract-parent-dir/${archive-dir}/, a third party tool takes a single path, the extract-parent-dir/, and processes the content to produce another file index-dir/file.

Also the project foo needs to have both extract-parent-dir and index-dir as resources.

As tasks should not have overlapping outputs the expand tasks, are emitting in their own folder extract-parent-dir/${archive-dir}/, however I have a small issue here as declaring extract-parent-dir/ as a srcDir (like sourceSets.main.resources.srcDirs(extractParentDir)) do not take into account whether the expand task is up-to-date.

And there’s another issue to think about the third party tool task needs to be aware that something is not up-to-date I think.

I have somewhat worked this around using this, but this feel awkward.

def extractParentDir = project.layout.buildDirectory.dir("generated/extract-parent-dir")
def extractParentFileTree = fileTree(extractParentDir) {
  builtBy(project.tasks.withType(Copy).matching { it.name.startsWith("expand") })
}
// this seems awkward as it should be doable using srcDir only ?
tasks.named("processResources") {
  dependsOn(extractParentFileTree)
}


sourceSets.main.resources.srcDir(extractParentDir)

// expand task are registered this way for each archives
project.tasks.register("extract${archiveDir.capitalize()}", Copy) {
  // ...
  from(zipTree(archiveTask.map { it.archiveFile })) { ... }
  into extractParentDir.map { it.dir(archiveDir) }
}

// then the third party tool
def generateIndex = tasks.register('generateIndex', JavaExec) {
  // ...
  it.inputs.files(extractParentFileTree)
}
sourceSets.main.resources.srcDir(generateIndex)

With this the processResources task operates well, and generateIndex task properly sees it’s dependent.

When I was using extractParentDir (instead of its fileTree) for srcDir and input.files the expand tasks where not checked if up-to-date.

2

FYO I edited my earlier post as the previous code didn’t produce the right archive via a shadowJar task, that wasn’t properly taking all extract folders in "generated/extract-parent-dir".

3

If the shown code is an exact copy, I even doubt it works as intended, besides that even if it works as intended, it might have some downsides.
Some things that come to mind:

  • by using builtBy(project.tasks.withType(Copy).matching { it.name.startsWith("expand") }) the matching tasks are iterated, which means that you break task-configuration avoidance for each and every task of type Copy
  • you configure extractParentFileTree to be builtBy all Copy tasks where the name starts with expand, but there will be none, at least none set up by your snippet which just registers ones starting with extract
  • if needing such a construct, you could explicitly configure the tasks you register as builtBy tasks instead of using matching task collection
  • Copy is btw. seldomly what you want, you usually want Sync to get rid of stale files that may still be lurking around on a non-clean build
  • instead of using the parent dir as srcDir and extracting into subdirs, you could maybe instead have one Sync (or Copy task) that has the extractParentDir as destination dir and use that task as srcDir directly, then you also have the necessary task dependencies implicitly
  • alternatively each of the extract tasks could extract into a subdirectory of its destination directory and then use those extract tasks as srcDirdirectly if you prefer having separate tasks
  • for the generateIndex task you would either copy the outputs of the other tasks together (e. g. with a sync { ... } as doFirst action into its temporaryDir) so that the indexing can work on the aggregated files if that is necessary for the indexing task
4

:folded_hands:

Useful points ! I completely missed the Copy semantics there ! Also there was a typo in my original snippet example in the matching operator, but anyway this is not anymore here. Completely forgot that one can call builtBy later outside the configure closure.

The alternatives didn’t work yet, there are other things at play there ! This is what I got so far.

def extractParentDir = project.layout.buildDirectory.dir("generated/extract-parent-dir")
def extractParentFileTree = fileTree(includedAgentDir)

tasks.named("processResources") {
  dependsOn(extractParentFileTree)
}

sourceSets {
  // ...
  main.resources.srcDir(extractParentDir)
}

// register expand tasks is different
  def expandTask = project.tasks.register("extract${archiveDir.capitalize()}", Sync) {
    // ...
    it.into providers.provider { new File(extractParentFileTree.dir, agentDir) }
    it.from(zipTree(archiveTask.map { it.archiveFile })) { ... }
  }

  extractParentFileTree.builtBy(expandTask)

// then the third party tool
def generateIndex = tasks.register('generateIndex', JavaExec) {
  // ...
  it.inputs.files(extractParentFileTree)
}
sourceSets.main.resources.srcDir(generateIndex)
5

The alternatives didn’t work yet, there are other things at play there ! This is what I got so far.

Mind sharing what hinders you to use the alternatives? :slight_smile: