after the addition of compileOnly configuration into Gradle I would expect that MavenPublication publishing type would respect compileOnly configuration and generate proper Maven provided scope in the generated pom file.
compileOnly dependencies are only for compilation. They must never leak into clients, so there would be no point putting them into a published POM (no matter which scope).
If you have a JAR file that was compiled against a specific library using compileOnly configuration, it still needs such dependency at runtime.
So once the information is missing in the generated pom.xml, you simply cannot easily use such JAR file as you would experience ClassNotFoundExceptions at runtime.
I think the way we want to solve this in the future is making it simpler to define variants of a library (both on the producer and consumer side), so that you can model this at a higher level than “optional” and “exclude”. I.e. one problem with “optional” is that there might be multiple optional dependencies, but you need them all together if you take one of them. That gets lost in a simple “optional” flag.
Until we have that my recommendation would go in the other direction: Make them normal api/implementation dependencies and let the consumer decide if she wants to exclude or not. Or even better - make it a separate module that is discovered through Java’s ServiceLoader mechanism or similar.