How to add additional elements to classpath when executing gradle tasks

Dear Gradle community.

I have a task that I use to debug a library in Java. I would like to add additional JARs to the Gradle client VM which are necessary to properly do the debugging (in our case this is because we have a IntellIJ plugin that needs these libraries on the executing VM in order to function properly).
I have tried using the GRADLE_OPTS environment variables which was not successful.

Pointing to the right documentation / way to do this would be immensely useful.