How to debug a groovy based plugin and step into its source?

I have custom gradle plugin (groovy based) as a separate project. My project that uses the plugin downloads the plugin from nexus repo where I publish my plugin.

Now when I debug some task in my project with IntelliJ, stepping in to plugin related calls lead to doCall() etc instead of original groovy source of the plugin. How do I debug gradle plugin with IntelliJ in a way I can step into it’s groovy source while debugging?