Remote build cache node restarts

Help/Discuss
1

Hi gradle team,
I’m rolling out the build cache node (BCN) in a pod (k8s cluster). Everything works fine but I’m observing time-to-time (once a day or once in two days) BCN restarts itself and in logs, you can see the following:

​2024-02-19 00:10:58,364 INFO c.g.e.b.rw {eid=6dfifinqnquwq, service=rw} Current cache catalog size: 191 entries, consuming 12.8 MiB, with additional overhead of 96 KiB
​2024-02-19 00:40:58,365 INFO c.g.e.b.rw {eid=zdkl2sapvi3bw, service=rw} Current cache catalog size: 191 entries, consuming 12.8 MiB, with additional overhead of 96 KiB
​2024-02-19 01:10:58,365 INFO c.g.e.b.rw {eid=y45zr5r7bblgw, service=rw} Current cache catalog size: 191 entries, consuming 12.8 MiB, with additional overhead of 96 KiB
​2024-02-19 01:40:58,365 INFO c.g.e.b.rw {eid=n42ny5gda4aus, service=rw} Current cache catalog size: 191 entries, consuming 12.8 MiB, with additional overhead of 96 KiB
​2024-02-19 02:10:58,366 INFO c.g.e.b.rw {eid=5urtf2yapnlvg, service=rw} Current cache catalog size: 191 entries, consuming 12.8 MiB, with additional overhead of 96 KiB
​2024-02-19 02:40:58,366 INFO c.g.e.b.rw {eid=frl3fim2zn2s2, service=rw} Current cache catalog size: 191 entries, consuming 12.8 MiB, with additional overhead of 96 KiB
​2024-02-19 03:10:58,367 INFO c.g.e.b.rw {eid=p5jqy7igvk3mi, service=rw} Current cache catalog size: 191 entries, consuming 12.8 MiB, with additional overhead of 96 KiB
​2024-02-19 03:40:58,367 INFO c.g.e.b.rw {eid=jkt4masqzq7q2, service=rw} Current cache catalog size: 191 entries, consuming 12.8 MiB, with additional overhead of 96 KiB
​2024-02-19 04:00:00,214 INFO c.g.e.b.ud {eid=znltmzs5vhh2s, service=ty} Database compaction took 213 ms
​2024-02-19 04:10:58,368 INFO c.g.e.b.rw {eid=k6r6gfbb3kumi, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 04:40:58,368 INFO c.g.e.b.rw {eid=ie5pfp46sdexe, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 05:10:58,369 INFO c.g.e.b.rw {eid=7gsnlv22wgmhu, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 05:40:58,369 INFO c.g.e.b.rw {eid=jsbsxcz6vwwse, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 06:10:58,369 INFO c.g.e.b.rw {eid=iaeykzq6pvu3g, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 06:40:58,370 INFO c.g.e.b.rw {eid=ojoir4rfw73ei, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 07:10:58,370 INFO c.g.e.b.rw {eid=cxgk3whoedo64, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 07:40:58,370 INFO c.g.e.b.rw {eid=gf6m3doxunop6, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 08:10:58,371 INFO c.g.e.b.rw {eid=p77hh6va45jni, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 08:40:58,371 INFO c.g.e.b.rw {eid=nvvnc2nn3ap3i, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 09:10:58,371 INFO c.g.e.b.rw {eid=mieqplodyex5m, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 09:40:58,372 INFO c.g.e.b.rw {eid=hibzn6n53p3vo, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 10:10:58,372 INFO c.g.e.b.rw {eid=26q6irhwqywvk, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 76 KiB
​2024-02-19 10:40:18,029 INFO r.s.RatpackServer {} Stopping server...
​2024-02-19 10:40:18,030 INFO r.s.RatpackServer {} Stopping 44 services...
​2024-02-19 10:40:18,035 INFO c.g.r.a.v.c {eid=5wi5pan7tnhyy, peid=utdwto263i3dy} Stopping server version 18.0
​2024-02-19 10:40:18,035 INFO c.g.e.b.tz {eid=etmzjugx2ch2a, peid=p3rmfbg2tnjj2} Shutting down the database
​2024-02-19 10:40:18,251 INFO c.g.e.b.tz {eid=etmzjugx2ch2a, peid=p3rmfbg2tnjj2} The database was shut down in 215 ms
​2024-02-19 10:40:18,253 INFO r.s.RatpackServer {} Server stopped.
​2024-02-19 10:40:35,875 INFO c.g.e.b.fe {} Node dir: /data
​2024-02-19 10:40:35,875 INFO c.g.e.b.fe {} Config dir: /data/conf
​2024-02-19 10:40:35,929 INFO c.g.e.b.fe {} Using exclusive volume with size: 10 GiB
​2024-02-19 10:40:35,931 INFO c.g.e.b.fe$a {} No app plugins loaded
​2024-02-19 10:40:35,994 INFO r.s.RatpackServer {} Starting server...
​2024-02-19 10:40:36,008 INFO r.s.RatpackServer {} Building registry...
​2024-02-19 10:40:37,463 INFO r.s.RatpackServer {} Initializing 44 services...
​2024-02-19 10:40:37,466 INFO c.g.r.a.v.c {eid=7gy22fhxth6ba} Starting server version 18.0
​2024-02-19 10:40:37,476 INFO c.g.e.b.sw {eid=4k26wl4vk2wek} Directory structure initialized in PT0.006562109S
​2024-02-19 10:40:37,564 INFO c.g.e.b.uj {eid=gwnhdsutbc67e, peid=vokhkwi34viyk} Catalog migration completed in PT0.005916397S
​2024-02-19 10:40:37,584 INFO c.g.e.b.sz {eid=outrnlbtbjrxq, peid=46tca55mywzpe} Started file store reconciliation
​2024-02-19 10:40:37,584 INFO c.g.e.b.rw {eid=kx6qszbvivo3o, peid=bk3ekyk2axs2m, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 0 B
​2024-02-19 10:40:37,673 INFO c.g.a.g.s.h {eid=46ya6yesk2dqs, peid=upr3ojyt4nvxa} Started gRPC server on port 6011 with services [grpc.reflection.v1alpha.ServerReflection, grpc.health.v1.Health, build.bazel.remote.execution.v2.ActionCache, google.bytestream.ByteStream, build.bazel.remote.execution.v2.ContentAddressableStorage, build.bazel.remote.execution.v2.Capabilities] and interceptors [com.gradle.app.grpc.server.b, com.gradle.enterprise.buildcachenode.kn, com.gradle.buildcache.bazel.m, com.gradle.app.grpc.server.l]
​2024-02-19 10:40:37,683 INFO r.s.RatpackServer {} Ratpack started for http://localhost:5071
​2024-02-19 10:40:37,686 INFO c.g.e.b.fs {} Build cache node started (port: 5071).
​2024-02-19 10:40:37,714 INFO c.g.e.b.up {eid=23uh7khm5enrs, peid=n2p5q3n4x53ys} Configured storage settings: target cache size = maximum available, overridden volume size = 10 GiB, free space buffer size = 1 GiB, incoming buffer entry size = 300 MiB
​2024-02-19 10:40:37,715 INFO c.g.e.b.up {eid=23uh7khm5enrs, peid=n2p5q3n4x53ys} Configuration disabled age based eviction
​2024-02-19 10:40:37,715 INFO c.g.e.b.ah {eid=23uh7khm5enrs, peid=n2p5q3n4x53ys} Configuration loaded
​2024-02-19 10:40:37,769 INFO c.g.e.b.so {eid=jrdwkfnrfjooq, peid=outrnlbtbjrxq} Reconciled 50 bucket(s) out of a total of 130.
​2024-02-19 10:40:37,879 INFO c.g.e.b.so {eid=jrdwkfnrfjooq, peid=outrnlbtbjrxq} Reconciled 100 bucket(s) out of a total of 130.
​2024-02-19 10:40:37,921 INFO c.g.e.b.so {eid=jrdwkfnrfjooq, peid=outrnlbtbjrxq} Reconciled 130 bucket(s) out of a total of 130.
​2024-02-19 10:40:38,164 INFO c.g.e.b.so {eid=jrdwkfnrfjooq, peid=outrnlbtbjrxq} Reconciliation recovered 0 lost writes. Additionally, 0 lost writes were found, but were concurrently overwritten.
​2024-02-19 10:40:38,164 INFO c.g.e.b.so {eid=jrdwkfnrfjooq, peid=outrnlbtbjrxq} Reconciliation deleted 0 lost removals. Additionally, 0 lost removals were found, but were concurrently overwritten.
​2024-02-19 10:40:38,164 INFO c.g.e.b.sz {eid=jrdwkfnrfjooq, peid=outrnlbtbjrxq} Finished file store reconciliation
​2024-02-19 10:40:42,593 INFO c.g.e.b.vc {eid=r7ajn7ua4d5a4, service=MonitorFileStoreStorageStateTask} Using target cache size of 9 GiB (requested free space buffer: 1 GiB, free space: 9.9 GiB)
​2024-02-19 11:10:37,584 INFO c.g.e.b.rw {eid=2fn3bowcphyvi, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 128 KiB
​2024-02-19 11:40:37,585 INFO c.g.e.b.rw {eid=57yxdmofsu32m, service=rw} Current cache catalog size: 193 entries, consuming 12.8 MiB, with additional overhead of 128 KiB

I excluded the possibility that k8s restarts the pod. Maybe there is a chance to change the logging setting.

BCN: version 18.0
k8s: version 1.23
Gradle clients: version 8.5

2

Hi, did you fix this problem?