Caching task input files that textually include other files

Help/Discuss
1

We have a OpenAPI/RAML code generation plugin that I’ve recently upgrade to 8.2 and enabled build-cache support for. There is one outstanding issue, “includes” or “references”.

These file types can contain textual references to other files. Parsing the file (with a parser understanding the domain), is the only way to generate a correct list of files referenced by each input file. How do I go report to the Gradle engine that the input files for the Task are more than just the list assigned by the build?

I’ve looked all through the code and aside from some undocumented & unstable features in (e.g. LanguageTransform) I cannot figure this out. This seems like a “standard” feature for most text processing format like templates, etc.

2

The task in question would have an @InputFiles property that calculates all those files, just probably in a cached way like a property on which you call finalizeValueOnRead().

3

This would have to run on every configure. I’m looking for a solution that allows Gradle to track these detected include files without requiring reconfiguration.

4

Why should that run on every configure?
It should only run when the task is going to be executed to determine whether it is up-to-date or can be taken vom cache.

5

Yes. To determine if it’s up-to-date, it will need the list of inputs, which using your method would require parsing the complete file.

6

Yes, but not on every configure, just right before that task is actually going to be executed eventually, which is exactly what you asked for isn’t it?

7

Maybe I used the wrong stage when I said configure. My point is that parsing a tree of files isn’t really a savings. In one of the cases we are currently using, the @InputFile pulls in 12 other files in a tree. So that would require parsing 13 files completely just to determine the includes. That’s a big task being done “before” the task is executed.

What I’m looking for is a way to parse the @InputFile, which naturally generates a list of includes, and have Gradle watch the source and the include files when performing its up-to-date check.

8

If I understood correctly, I don’t think there is something like that available.

Feel free to open a feature request. :slight_smile:

I guess in the meantime you need to maintain some cache yourself, for example within Task#temporaryDirectory, or within the GRADLE_USER_HOME where you then can store a mapping “checksum of file => list of included files” to only parse a given file once and otherwise take it from that cache.