File Hashing Incredibly Slow / Redundant

I feel like this has probably been discussed, but I haven’t really been able to find satisfying workarounds or any clear idea of whether the issue is going to be addressed. Let me know if there’s more information somewhere else.

Gradle uses file hashing to check if files have been changed, but this is incredibly slow for large assets. We have a number of tasks in the middle of our dependency tree that spit out ~300mb archives that are then used elsewhere in the tree. Unfortunately even if the file hasn’t been changes, in the up-to-date check Gradle goes through and hashes these files taking anywhere from 30 seconds to a few minutes for each one. This completely ruins our ability to make quick compilations, as it takes time even if we only made a small change somewhere downstream.

It seems like there should be some way to set certain tasks to check outputs based on timestamp, even if the resulting structure would be less correct. Or perhaps some way to swap out for a faster hashing algorithm?

Is this something that has been discussed? How do people normally deal with this issue in big projects?

Hi,

If the file length and timestamp haven’t changed since the last time, Gradle doesn’t recompute the hash. It sounds like something is bumping the timestamp on these files. Is that possible?

Huh, that’s interesting. I’ll take a look and see what might be happening. If that’s indeed the intended behavior, something along the line is getting caught up.

Running the build with --info will at least allow you to identify which files Gradle thinks have changed which might help you track this down.

So, oddly – running with --info shows no file changes:

Skipping task ':desktop:dist' as it is up-to-date (took 29.288 secs). :desktop:dist UP-TO-DATE :desktop:dist (Thread[Daemon worker Thread 3,5,main]) completed. Took 29.299 secs. :desktop:dist_packr_windows (Thread[Daemon worker Thread 3,5,main]) started. :desktop:dist_packr_windows

Yet the up-to-date check took 29 seconds. So the input files haven’t changed at all. :\ Additionally, I’m positive the timestamp hasn’t changed. Is the file-length checking something on a newer version of gradle? I’m on gradle distributions-snapshots/gradle-2.5-20150422220021+0000-all.zip

Thanks!

The file-length checking isn’t anything new and is definitely part of the snapshot you’re working with. Are you working on a special FS like NFS or similar?

Nope, nothing out of the ordinary.

NTFS file system, happens on both Win7 and Win10. The task is Jar task if that helps anything.

We’re using a few gradle files we didn’t write – Is it possible that they globally overwrote the behavior of the up to date check? Or is there some way I can get more information?

Thanks!