Javadoc fails because of strange default generates sources output directory when using an annotation processor

I have an annotation processor that is automatically picked up by the Java compiler at build time (using SPI). During a gradle build, the generated java sources of this annotation processor are put in build/classes as Gradle tells the annotation processor that this is the place to output generated source files. I use Filer.createSourceFile(…).

When the standard javadoc task is run, it tries to create javadoc for all files in build/classes, including *.java. This failes because javadoc only expects *.class files, making the whole build fail. Error: javadoc: error - No public or protected classes found to document.

So my question is:

Is this a Gradle bug/feature? How do I fix it/make it work?

My remarks: It looks to me that generated source files should not be put in a binary build output directory. Ideally my gradle build script should not be adjusted to give the java compiler a custom generated source files directory.