Make the copy task dependent on file changes only

Help/Discuss
1

Hi,

I have a large legacy project written in several languages including Java. This project is assembled by the make utility for several targets and platforms. The last important thing is that the project may be assembled out of the project root folder.

When I decided to rewrite the assembling and distribution of the Java part of the project on Gradle, I faced the following problem. The make utility runs Gradle several times, ones for every combination of platform and target. The several Gradle runs is not a problem by itself. But coping of the build artifacts occurs every time when the dist folder is changed regardless of the presence of artifacts in the dist folder.

The simple example is below:

// gradle-test/build.gradle
plugins {
	id 'java'
}

ext {
	distRoot = file(getPropertyValueOrDefault('distDir', "${rootProject.projectDir}/build/dist"))
}

public <T> T getPropertyValueOrDefault(String propertyName, T defaultValue) {
    T value = rootProject.findProperty(propertyName) as T
    if (value == null) {
        value = defaultValue
    }
    return value
}

sourceSets {
    main.java.srcDirs = ['src']
}

tasks.register('dist', Copy) {
    from jar
    into distDir

// gradle-test/src/Dummy.java
public class Dummy {}
  1. gradle dist -PdistDir=./dist1 -i
    Three tasks are executed (classes, jar and dist)
  2. gradle dist -PdistDir=./dist2 -i
    One task is executed (dist). The build directory is the same, the dist directory has been changed.
  3. gradle dist -PdistDir=./dist1 -i
    Dist task is executed again. The build directory is the same, the dist directory has been changed. But gradle-test.jar already existed in dist1 directory before running Gradle.

I know that this behavior is default for Gradle. But is there any way to exclude the value of distDir property from Gradle cache?

I believe similar questions have already been asked, but I couldn’t find relevant discussions. I would appreciate any solutions.

2

This has nothing to do with caching.
A Copy task is never cached.
It might be up-to-date or not.
But as the output of the task changed since the last run, it cannot be up-to-date and thus has to be re-executed.

You could instead have two separate copy tasks, so that the outputs of the tasks do not change and they can be up-to-date.

Or you could maybe use outputs.upToDateWhen { check-the-file-exists-here }, but that would not be a good idea imho, as you don’t know whether what you find is proper, if it works at all.

3

Thank you for quick response.

But as the output of the task changed since the last run, it cannot be up-to-date and thus has to be re-executed.

I meant that gradle saves the task output from the last run when I talked about caching. I apologize for the confusion in gradle terms. I’m still thinking in make terms.

You could instead have two separate copy tasks, so that the outputs of the tasks do not change and they can be up-to-date.

Unfortunately I can’t. The reason is the following. The most part of the project is still assembled via make. Also, the distribution name is determined by the configure script and can be arbitrary. For example:

  • distributions/x64-linux
  • distributions/x64-windows
  • distributions/custom-developer-pack-name
    Therefore, I can’t write separate task for every possible case.

Or you could maybe use outputs.upToDateWhen { check-the-file-exists-here }

I thought about this option. But I also want to check the the output of the jar task has not changed since the last run, i.e. like this outputs.upToDateWhen { check-the-file-exists-here && check-the-jar-task-output-didn't change }. Is it possible? I could use this option as a temporary solution until the entire project build is done using gradle.

4

I meant that gradle saves the task output from the last run when I talked about caching. I apologize for the confusion in gradle terms. I’m still thinking in make terms.

Yes, that is what caching is. For cacheable tasks if the cache is enabled, it stores the file outputs of a task in the cache and can get it from the cache if the inputs are the same as for the cache entry. That way you can for example easily switch between branches without always having to rebuild everything. But as I said, for a Copy task that would not make sense.

The up-to-date check, just remembers checksums and paths and looks whether the inputs and outputs since the last successful execution have changed. If not, the task will be considered up-to-date, if they are, the task is re-executed. And as your output files change by changing the property, they are not the same since the last execution and thus are re-executed.

Unfortunately I can’t. The reason is the following. The most part of the project is still assembled via make. Also, the distribution name is determined by the configure script and can be arbitrary.

Sure you can, you could for example include a sanitized form of your property in the task name and then generate an own task for that specific run that is exclusively for that property value. Those should then be able to be properly up-to-date I think.

like this outputs.upToDateWhen { check-the-file-exists-here && check-the-jar-task-output-didn't change } . Is it possible?

Sure, if you put enough effort in it. Or short, don’t. You would probably have to re-implement a completely own up-to-date system, calculating, remembering, and comparing checksums. Just don’t, but adjust your usage as described above. :slight_smile:

I could use this option as a temporary solution

Really not worth the effort. :wink:

1 Like
5

You would probably have to re-implement a completely own up-to-date system, calculating, remembering, and comparing checksums.

Your first option became more preferred after these words. :smile:

With the help of your hint, I modified the script in this way.

import org.gradle.internal.os.OperatingSystem

plugins {
    id 'java'
}

ext {
    def target = OperatingSystem.current().nativePrefix
    distRoot = file(getPropertyValueOrDefault('distDir', "${rootProject.projectDir}/build/distros"))
    distName = getPropertyValueOrDefault('distName', "${target}")
}

public <T> T getPropertyValueOrDefault(String propertyName, T defaultValue) {
    T value = rootProject.findProperty(propertyName) as T
    if (value == null) {
        value = defaultValue
    }
    return value
}

sourceSets {
    main.java.srcDirs = ['src']
}

tasks.register("dist-${distName}", Copy) {
    from jar
    into "${distRoot}/${distName}"
}

tasks.register('dist') {
    dependsOn "dist-${distName}"
}

As a result

$ gradle dist
3 actionable tasks: 3 executed (got `build/distros/linux-amd64/gradle-test.jar`)
$ gradle dist -PdistName=name1
3 actionable tasks: 1 executed, 2 up-to-date (got `build/distros/name1/gradle-test.jar`, `dist-name1` is executed)
$ gradle dist -PdistName=name2
3 actionable tasks: 1 executed, 2 up-to-date (got `build/distros/name2/gradle-test.jar`, `dist-name2` is executed)
$ gradle dist -PdistName=name1
3 actionable tasks: 3 up-to-date
$ gradle dist -PdistName=name2
3 actionable tasks: 3 up-to-date

Doesn’t this solution look like a crutch? Are there points that could be improved?

6

Doesn’t this solution look like a crutch?

No, why?

Are there points that could be improved?

Quite some. :smiley:
Here some of them:

  • use Kotlin DSL, you immediately get type-safe build scripts, much more helpful error messages if you mess up the syntax and amazingly better IDE support
  • not not use internal Gradle classes
  • do not abuse ext / extra properties when a simple local variable is applicable
  • instead of getPropertyValueOrDefault you could simply use findProperty('distDir') ?: "..." (in both DSLs)
  • assign the tasks.register return value to a variable, then you can use that later on to depend without the need to depend on the task by string
  • Do not use Copy tasks unless you really want to copy with several tasks into the same directory, which is bad per-se in 98.7 % of cases, use a Sync task instead

:slight_smile:

7

My intermediate solution didn’t contain dummy dist task and I had to run Gradle like this gradle -PdistName=name dist-name which was inconvenient. Then I thought if a dummy dist task could help. I wasn’t sure if it would work the way I expected. But it was. Now I have dist task which has made my life easier. :smile: But since I haven’t seen examples doing this kind of thing, I’m not sure about the solution. this feeling will go away once I gain more experience as a Gradle user.

Thanks a lot for your review and advice!

1 Like