I have felt many of the same frustrations as you have. I am still learning about Gradle, but part of the answer I have learned is that unlike with most build systems, the build configuration is not declarative. Instead, the build configuration is itself a program (the “build script”) that Gradle executes in order to discover what to build. So Gradle has to run this program every time before it can even look to see what has or hasn’t changed. Unfortunately, for most projects, this build script is quite complicated and takes a while to run.
To be fair, the fact that the build configuration is a program gives Gradle a tremendous amount of power. But it is a shame that it is so slow.