Configuring remote Gradle caching

Trying to setup remote Gradle caching, as part of poc did all the required steps mentioned in Gradle cache documentation and also created local docker image setup Docker Hub for remote caching.
Did notice a considerable time difference for first build between (–no-build-cache (9 mins.) & --build-cache (4 mins.)) when agent has just started.
But didn’t see a good time difference for further builds compared for --no-build-cache and --build-cache configurations.
Also while looking at build scan report for remote cache configuration, can see Remote Hit count but task requested from cache is still 0. Attaching screenshot for reference.
Not sure if doing something wrong, please help