Having Both Java and Android Subprojects on Same Project

I am developing a project which includes an Android application and a Java backend (along with some other common APIs).
As an IDE I am using IntelliJ Idea.
I would like to have a single Gradle project (which will be opened on a single Idea instance), that contains all the applications as subprojects.

My problem is that in order to allow the Android plugin, I need to set it in the buildScript section in build.gradle file:

dependencies {
    classpath 'com.android.tools.build:gradle:2.2.3'
}

This forces the android plugin in the entire Gradle build process, and causes errors on the pure Java subprojects (they are automatically set to be built with the Android SDK instead of the Java SDK).

A workaround I have thought of is creating two separate projects (one for Android and one for Java), export the common JARs into a local Maven repository and import it from there into the Android project.

Is there a better solution that will allow me to have all the code base in the same place?