# Transparent/pluggable input/output checksums

**URL:** <https://discuss.gradle.org/t/transparent-pluggable-input-output-checksums/35176>\
**Category:** Help/Discuss\
**Created:** [March 7, 2020, 8:27am UTC](https://discuss.gradle.org/t/transparent-pluggable-input-output-checksums/35176 "2020-03-07T08:27:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![bgeradz](https://avatars.discourse-cdn.com/v4/letter/b/ed655f/32.png) [@bgeradz](https://discuss.gradle.org/u/bgeradz)\
**Post date:** [March 7, 2020, 8:27am UTC](https://discuss.gradle.org/t/transparent-pluggable-input-output-checksums/35176/1 "2020-03-07T08:27:23Z")

</div>

According to docs gradle keeps track of “checksums” of task inputs / outputs to understand whether a task should execute.  
However there is no explanation on when and how these checksums are calculated.

Is it possible to make gradle’s checksuming mechanism pluggable or at least more transparent?

Use case: let’s say your project tasks operate on huge amount of data as inputs and outputs. Let’s assume naively calculating md5 of files is inefficient and does not scale well. What if there is an external service that efficiently keeps track of checksums and gradle could be instructed to ask the external service for checksums of given files instead of calculating them by itself?

---

<div class="post-metadata">

**Author:** ![Lance](https://sea1.discourse-cdn.com/gradle/user_avatar/discuss.gradle.org/lance/32/787_2.png) [@Lance](https://discuss.gradle.org/u/Lance)\
**Post date:** [March 7, 2020, 2:47pm UTC](https://discuss.gradle.org/t/transparent-pluggable-input-output-checksums/35176/2 "2020-03-07T14:47:02Z")

</div>

For this purpose you can use [inputs.property(String name, Object value)](https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/TaskInputs.html#property-java.lang.String-java.lang.Object-) where the value can be a `Closure` or a `Serializable` object

---

<div class="post-metadata">

**Author:** ![bgeradz](https://avatars.discourse-cdn.com/v4/letter/b/ed655f/32.png) [@bgeradz](https://discuss.gradle.org/u/bgeradz)\
**Post date:** [March 8, 2020, 12:12pm UTC](https://discuss.gradle.org/t/transparent-pluggable-input-output-checksums/35176/3 "2020-03-08T12:12:30Z")

</div>

Of course I can apply something like this for 1% of all calculations for my custom tasks.

What I am after is a possibility from the top build script to say, hey gradle, do your usual thing with all the plethora of tasks but when it comes to file checksum calculation use this replacement _universallly_.

Wouldn’t it be useful?

---

<div class="post-metadata">

**Author:** ![Lance](https://sea1.discourse-cdn.com/gradle/user_avatar/discuss.gradle.org/lance/32/787_2.png) [@Lance](https://discuss.gradle.org/u/Lance)\
**Post date:** [March 8, 2020, 12:36pm UTC](https://discuss.gradle.org/t/transparent-pluggable-input-output-checksums/35176/4 "2020-03-08T12:36:33Z")

</div>

I think that if you’ve got a fantastic algorithm for calculating checksums that outperforms Gradle’s algorithm then you should open source it and let Gradle use that instead of its current mechanism

---

<div class="post-metadata">

**Author:** ![bgeradz](https://avatars.discourse-cdn.com/v4/letter/b/ed655f/32.png) [@bgeradz](https://discuss.gradle.org/u/bgeradz)\
**Post date:** [March 8, 2020, 3:14pm UTC](https://discuss.gradle.org/t/transparent-pluggable-input-output-checksums/35176/5 "2020-03-08T15:14:24Z")

</div>

I believe the algorithm is perfectly fine.  
It’s about _when_ checksums are calculated, not _how_.  
If I understand it right (gradle docs are not any clear on this) gradle calculates checksums right at the time when tasks are performed. What if these operations could be done in advance to save build time.

I am considering two options  
a) a daemon that tracks file system events and calculates checksums asynchronously as files are changed  
b) a fuse based file system that proxies file operations and calculates checksums asynchronously as files are changed

In both cases gradle just asks “give me the checksum of this file” and is likely to receive precalculated checksum almost always _instantaneously_.

---

<div class="post-metadata">

**Author:** ![Stefan\_Wolf](https://sea1.discourse-cdn.com/gradle/user_avatar/discuss.gradle.org/stefan_wolf/32/5519_2.png) [@Stefan\_Wolf](https://discuss.gradle.org/u/Stefan_Wolf)\
**Post date:** [June 4, 2020, 4:45pm UTC](https://discuss.gradle.org/t/transparent-pluggable-input-output-checksums/35176/6 "2020-06-04T16:45:20Z")

</div>

Hi Genadz,

we are not planning on making the checksum algorithm pluggable. What we are doing though is to watch the file-system for changes to re-use the checksums as much as we can - even between builds: [https://blog.gradle.org/introducing-file-system-watching](https://blog.gradle.org/introducing-file-system-watching). That should make calculating the checksums much faster.

We are also thinking about calculating the checksum in the background when the daemon is idle every time we detect a change. Though that may be more in the future when we see the need.

Maybe you could [give file-system watching a spin](https://blog.gradle.org/introducing-file-system-watching) and see if it helps for your use-case.

Cheers,  
Stefan
