for example, there is @Internalwhich indicates a property is not part of up to date checking, I intuitively thought this would also apply to caching but it does not. Any working approach to this problem?
For caching the @Internal is also ignored just like for up-to-dateness.
Build cache, that is.
For configuration cache this does not apply, because it is still configuration state of the task that is necessary to execute the task. You cannot make a property not CC relevant, because the whole task state is CC relevant.
You can mark a task as not configuration cache compatible using notCompatibleWithConfigurationCache(...), but that should only be a temporary measure for a transition period, as it completely disables configuration cache whenever that task is part of the task graph.
So it is a much better idea to make the task properly configuration-cache compatible fully.
yes i’ve been working my way through that though it hasn’t been worth the effort so far (cache hits seem super rare because the cache seems to be keyed by task so each different task you run (the norm in a large composite build IME) requires a new cache entry.
This Q was prompted by a task which contains a nested reference to an instance of HttpClientImpl, i.e. some not serializable instance. The configuration is of this does come from some cacheable data (i.e. a bunch of Property<XX> that gets exposed to the build) but, for various reasons, the resulting service is passed to the task itself via a Provider
I imagine I have to find a way to decouple this so that the configuration object(s) are passed to the task and that data is passed to some new BuildService which yields something that talks http. It appears that these are configuration cache friendly though the mechanics/limitations of that are unclear and I think going to lead to a chunk of duplicate code as the params api in that is quite static.
Is there any other option? or any other problem with using a BuildService here?
cache hits seem super rare because the cache seems to be keyed by task so each different task you run (the norm in a large composite build IME)
Yes, they are, everything else would not work, because if everyone behaves well and preserves task-configuration avoidance, only the tasks that are going to be executed are configured so only those can be persisted to CC and restored from it.
I cannot see however why you should constantly run different commands.
You typically run a small set of tasks like build or test or check repeatedly and there the CC then kicks in, saving a lot of time.
Is there any other option? or any other problem with using a BuildService here?
Not sure, as I did not fully understand the situation.
Maybe you should not store the http client in a field but just create it when you need it.
Or yes, maybe a build service is what you should use if you need to once create that client and reuse it in multiple tasks.
Of course, if your experience shows that CC is not worth it for your project, the question is why you try to enable it then at all. Just don’t enable it and you are good, but usually the benefits should be there.
From what I’ve seen (of a reasonably sized composite project) it’s common to run build or test on some particular included build or some subset of them rather than just running build on the whole thing. In our setup, the configuration of one included build is not dependent on another so caching at the task execution level means the same things are being cached repeatedly rather than reused. Why exactly this happens, and hence whether it is avoidable, is not something I’ve determined as yet.
well I assume it will get better and the only way to find out (whether it’s worth it) is to try it.
it’s basically like this
NamedDomainObjectContainer<MyHttpConfig> is exposed to the build, typically a few instances will be added and each one represents some instance of a particular service.
A task reads some dataset, each entry means 1 or more calls to a particular http service, i.e. it picks one of the configured MyHttpConfig to use, pass that to some other service which transforms that into an instance of MyHttpService which is the thing that contains the HttpClient. BuildService api has a few problems in this context
- parameters are static so I don’t think you can expose the
Paramsto the build and directly configure them, it seems I’d need to duplicateMyHttpConfigas aBuildServiceParametersimpl BuildServiceare individual named instances known at compile time so there’s no obvious way to allow them to be driven from build config
having written this down I suppose it’s more like a Transformer to apply to the NamedDomainObjectContainer and accept there’s no way to reuse a single HttpClient (there can be multiple tasks that need to make such calls)
From what I’ve seen (of a reasonably sized composite project) it’s common to run build or test on some particular included build or some subset of them rather than just running build on the whole thing.
Quite possible, yes.
But the point is, that you typically run that same task repeatedly while you work on a given topic.
Why exactly this happens, and hence whether it is avoidable, is not something I’ve determined as yet.
Imagine things like
val foo by tasks.existing
val bar by tasks.existing {
foo.dependsOn(this)
}
Ignoring the face that this is bad practice on multiple levels, but there are other similar constructs that are not, I’m just too lazy to come up with one right now without IDE.
If you execute foo, only foo is run.
If you run foo bar, foo depends on bar.
So the CC has to have different entries depending on the tasks requested.
it’s basically like this
Sounds like build service should help, yes, especially if you want to share the same instance between tasks.
You can then even configure that only one such task can run at the same time if that service should be used only by one task at the same time.
BuildService are individual named instances known at compile time so there’s no obvious way to allow them to be driven from build config
Why do you think it is a compile time constant? You can calculate the name from the config, no problem.
because I typically use org.gradle.api.services.ServiceReference so had that in my head when I commented, seems clear that that’s the approach to try so thanks for the guidance
that’s fine but I think it leads to more cache misses than necessary in the context of a composite project with a load of included builds as projects are invariably independent. Would improving this be a part of maturing Isolated Projects?
Most probably, yes.
Also, let me quote https://github.com/gradle/build-tool-roadmap/issues/76:
This allows configuration and tooling model creation for each project to safely run in parallel and the result cached and invalidated for each project independently.
using a BuildService here seems quite clumsy so perhaps I’m missing something?
problem is that the BuildService name depends on a Property<String> but there’s no way to get a registration from the BuildServiceRegistry without having the actual name. As far as I can see that means using afterEvaluate to wait until the value is known or supplying all possible instances to each task.
Is there an alternative or this is a gap in the API?
Maybe just have one BuildService which supports multiple configs and when you call it you give the config / config identifier to use?
it’s unclear how to make that work with the Params, I guess each one would be a MapProperty?
it’s just generally a much messier solution, e.g. the need to explicitly call useService on each task produces IllegalMutationException as it requires the result of a Provider<String> to know the service name which implies using an afterEvaluate but you can’t call that from a task.configureEach callback for some (not obvious) internal gradle reason.
You should practically never call afterEvaluate anyway. The main gain you get is timing problems, ordering problems, and race conditions.
as it requires the result of a
Provider<String>to know the service name
Not if you maybe follow my last suggestion where you only have one service that can deal with multiple configurations.
it’s unclear how to make that work with the Params, I guess each one would be a MapProperty?
I don’t know your code, and sorry, but I’m not designing your code either.
I give you help and hints how you could achieve things from the information you shared,
but you have to implement it yourself or hire someone that does it for you. 
You do have to be careful for sure but the alternatives can be very tricky to reason about
Quoting myself so I can correct myself. I can say that I have reached the point, ie all common operations are configuration cache compatible, that the benefits of a cache hit are v large and really quite transformative to the build experience.
You do have to be careful for sure but the alternatives can be very tricky to reason about
You cannot be careful.
You can only be slapped in the face square.
As soon as you use afterEvaluate anywhere, you do have race conditons and there is no way to avoid them.
The only “valid” reason to use afterEvaluate I have seen so far is if you use some misbehaving plugin that uses afterEvaluate and you need to do something after that plugin did its work, and it would be heavily advisable to at the same time make that plugin come in line.
I can say that I have reached the point, ie all common operations are configuration cache compatible, that the benefits of a cache hit are v large and really quite transformative to the build experience.
