Gradle erroneously reports a task is UP-TO-DATE?


(Sašo Muševič) #1

This is a quite common scenario, when two flavours of 3rd party JavaScript dependencies are included in a project. For development purposes a non-minified versions of JavaScript files are used, while deploy scenario typically only includes the minified versions (*.min.js). Let’s assume both (minified and non-minified) versions of all the dependencies are in the ‘repo’ folder. Further, there’s 2 versions of the ‘main’ file, one uses the minified deps ‘main.min.js’ and while ‘main.js’ uses the non-minified ones. Let’s assume both ‘main’ files can be generated by some means from the ‘deps.json’ where all the dependencies are declared. The file structure is as follows:

public/
    lib/
repo/
        angular/
            ...
        angular-resource/
            ...
        angular-route/
            ...
build.gradle
deps.json
main.js
main.min.js

The ‘public’ folder is where all the output files should appear, so I wrote the corresponding ‘build.gradle’ file:

task createMain {
 inputs.file 'deps.json'
 // TODO: read deps.json and create
main.min.js and './main.js
 outputs.file 'main.min.js'
 outputs.file 'main.js'
}
  task copyMain(type: Copy, dependsOn: createMain) {
 from('.') {
  include 'main.js'
 }
 into('public')
}
  task copyMainForDeploy(type: Copy, dependsOn: createMain) {
 from('.') {
  include 'main.min.js'
 }
 rename('main.min.js','main.js')
 into('public')
}
  task installJSDeps(type: Copy, dependsOn: copyMain){
 from('repo')
 into('public/lib')
 outputs.dir 'public/lib'
 inputs.file 'deps.json'
}
  task installJSDepsForDeploy(type: Copy, dependsOn: copyMainForDeploy){
 from('repo'){
  include '**/*.min.js'
 }
 into('public/lib')
 outputs.dir 'public/lib'
 inputs.file 'deps.json'
      doFirst {
  //clean up any existing files before copying new ones
  FileTree tree = fileTree (dir: "public/lib");
          delete(tree)
 }
}

What I was hoping to achieve is: if I call ‘installJSDepsForDeploy’ only the minified files appear in ‘public/lib’, if I call ‘installJSDeps’ all the files appear in ‘public/lib’ (in addition to ‘main’ file being copied/renamed). What happens is the following:

$ gradle installJSDepsForDeploy
:createMain UP-TO-DATE
:copyMainForDeploy
:installJSDepsForDeploy
  BUILD SUCCESSFUL
  Total time: 3.698 secs
  $ gradle installJSDeps
:createMain UP-TO-DATE
:copyMain
:installJSDeps
  BUILD SUCCESSFUL
  Total time: 2.484 secs
  $ gradle installJSDepsForDeploy
:createMain UP-TO-DATE
:copyMainForDeploy
:installJSDepsForDeploy UP-TO-DATE
  BUILD SUCCESSFUL
  Total time: 2.41 secs

The second time ‘:installJSDepsForDeploy UP-TO-DATE’ is reported, which is not desired.

Am I missing something? Thanks in advance,

Sash


(Luke Daley) #2

I don’t quite understand what you’re doing, but some comments…

  1. You shouldn’t have to declare inputs and outputs for ‘Copy’ tasks (they are inferred from the from/into statements) 2. You probably want to use the ‘Sync’ task instead of ‘Copy’.

(Sašo Muševič) #3

Hi Luke thanks for your answer. I have realized recently that Copy tasks do not need explicit input/output declaration, thanks for clarifying it. However, the Copy tasks were only used to mimic the effect of ‘bower’ installer, which effectively does a very similar thing. The ‘bower’ is invoked via ‘Exec’ task, and fetches some files from a specific GIT repo etc… To clear out the confusion, let me explain what I need to do. My problem boils down to the following. I want ‘taskA’ copy from:

/repo/**/*.js

/repo/**/*min.js

/repo/**/*min.js.map

/repo/**/*.css

/repo/**/*.md

to:

/public/lib/**/*min.js

/public/lib/**/*min.js.map

And I want ‘taskB’ copy from:

/repo/**/*.js

/repo/**/*min.js

/repo/**/*min.js.map

/repo/**/*css

/repo/**/*.md

to:

/public/lib/**/*.js

/public/lib/**/*.css

/public/lib/**/*.md

The tasks DO NOT depend on each other, that’s in fact the main point here. Further, I want ‘TaskA’ NOT to copy the files that ‘TaskB’ is copying. Both tasks need to make sure the files that the other task is copying over ARE NOT present after completion.

Basically, after running ‘gradle taskA’ the ‘public/lib’ should include ONLY:

/public/lib/**/*min.js

/public/lib/**/*min.js.map

and NOT:

/public/lib/**/*.js

/public/lib/**/*.css

/public/lib/**/*.md After running ‘gradle taskB’ the result should be the opposite.

Now, in my experience even when not using a Copy task (only using the available Gradle/Groovy copy commands) and declaring the ‘outputs.dir’ the following happens. Running ‘gradle taskB’ AFTER ‘gradle taskA’ results in gradle reporting ‘taskB UP-TO-DATE’ which is not desirable. I can understand the logic behind it, but from the relevant chapter ‘15. More About tasks’ subchapter ‘15.9. Skipping tasks that are up-to-date’, in particular ‘15.9.2 How does it work?’ it is clearly stated that

Gradle takes note of any files created, changed or deleted in the output directories of the task. Gradle persists both snapshots for next time the task is executed.

As far as I can understand, ‘taskA’ should cause ‘taskB’ to be out of date and vice-versa, or am I missing something?

Thanks in advance,

Sash


(Perryn Fowler) #4

Hi Sash

I think you want your Copy tasks to be Sync tasks to get the desired behaviour.

Perryn


(Sašo Muševič) #5

Hi Perryn,

thanks for the answer. I am sorry to be inconclusive, but my tasks need to be ‘Exec’ as I am calling the ‘bower’ command which then fetches the files from GIT and copies the files over. So, Copy and Sync are out of question. What I am trying to achieve is to declare the inputs/outputs correctly, so the ‘Exec’ task (ultimately a call to ‘bower’) would only execute if there was a change in input file(s) or output directory.

So, the ‘taskA’ makes sure all the files are in the target dir, while ‘taskB’ makes sure only ‘*min.js’ (and nothing else!!) are in the target dir. In a sense, ‘taskB’ is a subtask of ‘taskA’, but it’s imperative that running ‘taskB’ removes any possible extra files from the target directory. Is this possible at all with Gradle?

Thanks,

Sash


(Luke Daley) #6

@Sašo: what you are describing as your desired behaviour is how it should be working.

Can you provide a small self contained build that I can use to experiment with this?


(Perryn Fowler) #7

Hi Sašo

Now I’m confused about what you are asking :slight_smile:

Are you asking how to make TaskB delete any extra files in its output directory, or does it already do this?


(Sašo Muševič) #8

OK, a minimal but real world example:

task fetchJSDeps {
 // SIMULATING THE FIRST STEP OF THE BOWER COMMAND
    // Let's pretend it reads 'deps.json', fetches
    // all the deps from some URL and puts them into 'repo' folder
    // This task is always UP-TO-DATE for the purpose of this
    // example - all the files are already in the 'repo' folder
  outputs.dir 'repo'
 inputs.file 'deps.json'
}
  task installAllJSDeps(dependsOn: fetchJSDeps){
   // SIMULATING THE SECOND STEP OF THE BOWER COMMAND
  // Caution: this cannot be a "Copy" or "Sync" task:
 //
  remember, this task simulates what bower does,
 //
  which is essentially a Exec task, so iputs/outputs
 //
  *must* be declared explicitly!
 def repoDir = 'repo'
 def publicLibDir = 'public/lib'
   // need to wrap into doLast, otherwise the copy
 // command executes during the 'config' time
 doLast {
  copy {
   // simply copies everything
   from(repoDir)
   into(publicLibDir)
  }
 }
 // the whole 'public/lib' folder is considered the output
 outputs.dir publicLibDir
 // inputs are all the file in the repo - produced by 'installAllJSDeps'
 inputs.files fileTree(dir: repoDir, include: '**/*' ).getFiles()
}
  task installMinifiedJSDeps(dependsOn: fetchJSDeps){
 // SIMULATING THE SECOND STEP OF THE BOWER COMMAND
 // Caution: this cannot be a "Copy" or "Sync" task:
 //
  remember, this task simulates what bower does,
 //
  which is essentially a Exec task, so iputs/outputs
 //
  *must* be declared explicitly!
 def includeStrings = ['**/*.min.js', '**/*.min.js.map']
   def repoDir = 'repo'
 def publicLibDir = 'public/lib'
 doLast {
  copy {
   // *ONLY* copies the minified files
   from(repoDir) {
    include includeStrings
   }
   into(publicLibDir)
  }
  def filesToRemove = fileTree(dir: publicLibDir, include: '**/*', exclude: includeStrings)
  // removes any non-minified files!
  delete filesToRemove.getFiles()
 }
 // the whole 'public/lib' folder is considered the output
 outputs.dir publicLibDir
 // inputs are all the file in the repo - produced by 'installAllJSDeps'
 inputs.files fileTree(dir: repoDir, include: '**/*' ).getFiles()
}

Here are all the files. This is what happens:

smusevic@ /c/Development/Workspace/GradleTest
$ gradle installMinifiedJSDeps
:fetchJSDeps UP-TO-DATE
:installMinifiedJSDeps
  BUILD SUCCESSFUL
  Total time: 3.626 secs
  smusevic@ /c/Development/Workspace/GradleTest
$ gradle installAllJSDeps
:fetchJSDeps UP-TO-DATE
:installAllJSDeps
  BUILD SUCCESSFUL
  Total time: 2.439 secs
  smusevic@ /c/Development/Workspace/GradleTest
$ gradle installMinifiedJSDeps
:fetchJSDeps UP-TO-DATE
:installMinifiedJSDeps UP-TO-DATE
  BUILD SUCCESSFUL
  Total time: 2.282 secs
  smusevic@ /c/Development/Workspace/GradleTest
$

Clearly, I do not want ‘installMinifiedJSDeps’ to be UP-TO-DATE the second time. The problem seems to be that adding files to a folder (pleas note: by adding I mean not changing/deleting any of the existing files, just adding more extra files) which is declared as ‘outputs.dir’ does not cause task to be out of date, which according to the Gradle manual chapter 15.9. (15.9.2.) should happen, or I am missing something. Thanks a lot guys! Sash


(Perryn Fowler) #9

Hi Sash,

When considering whether a tasks outputs are up to date, Gradle deliberately ignores any extra files that were added since the last time the task was run. This is so completely unrelated tasks that write files to the same output directory do not interfere with each other.

I think the sentence in the manual is referring to changes that are made by the task. I agree though, that as it is currently phrased it is misleading. I will try to make it clearer.

In order to get your example working as you wanted, I added the following configuration to your ‘installMinifiedJSDeps’ task

outputs.upToDateWhen {
    fileTree(dir: publicLibDir,
                 include: '**/*',
                exclude: includeStrings).isEmpty()
}

(Sašo Muševič) #10

Great answer Perryn, thanks a lot!!! Since my knowledge of Gradle is very basic, I made a number of wrong assumptions on this particular topic. Here’s the list:

  • when do the ‘inputs’/‘outputs’ get evaluated:

  • right AFTER the task has executed or

  • after the whole build has finished + if I make changes to code of the task in question:

  • are inputs/outputs hashes of the task discarded (i.e.: is the task forced to run after a code change) + how exactly does Gradle resolve the UP-TO-DATE status if you define all of the following:

  • ‘inputs.files’ and ‘inputs.file’ and ‘inputs.dir’ and ‘inputs.source’ and ‘inputs.sourceDir’

  • ‘outputs.files’ and ‘outputs.file’ and ‘outputs.dir’ and ‘outputs.upToDateWhen’

  • ‘onlyIf’ closure on the task + why does the task always run if there’s no ‘outputs’ defined (some tasks genuinely don’t have an output, only an input…I ended-uo setting output to input, which worked fine but it’s a bit confusing) + what you just said in your second paragraph (don’t know how to put it properly)


(Perryn Fowler) #11

Hi Sash,

Sorry, are you just listing your assumptions or did you want answers to those questions?


(Sašo Muševič) #12

No, I don’t need the answers, but when the documentation is updated, adding the answers the above questions will be very helpful for most of beginners. Thanks again!