Attach java agent to test runner JVM from CLI?

I am looking to instrument test suites. As you know, the java agents that will apply the instrumentation need to be specified with the -javaagent:<path/to/agent.jar> syntax for the JVM.

I want to understand how I can configure this argument from the CLI invocation of gradle, something like

$ ./gradlew test

such that the JVM arguments will be pushed to the JVM that runs the tests. This is such that I do not need to modify the gradle build configuration manually at all and test packages as they come from the repository. The packages to be instrumented I do not (generally) own so I would like to avoid modifying them.