How to debug a Gradle plugin?

I use a 3rd party Gradle plugin in my build script which seems to have a bug. Now I want to debug it by setting breakpoints in the source code of the plugin.
I start my application (which uses the plugin) with GRADLE_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005" && ./gradlew jar. I have created a remote configuration in Intellij to connect to the application. The problem is that after I have ran this configuration my app just finished. How can I set the breakpoints? I have tried several things like importing the plugin jar in Intellij and set the breakpoints but they are always getting ignored… ;(