Antlr 4 behavior change between Gradle 2.4 and 2.7

Help/Discuss
,
1

I recently switched locally to use Gradle 2.7 in Hibernate to resolve the regression in package generation with Antlr 3 introduced in 2.4.

However, we are developing a new query parser that uses Antlr 4 and that same change torpedos that build. With Gradle 2.7 and Antlr 4 how should one (now) structure the sources? In this new development we still keep the grammar source in a “package structure”. But running the Antlr plugin/task against this fails to work properly; I get errors.

This is the repo containing the new parser: https://github.com/hibernate/hibernate-semantic-query/tree/34f556b0e465c584b922e3cfcf4e51ff9b960203

The wrapper here is defined for 2.4. So if you run the build using the wrapper, it works fine. However, when I try to use 2.7 it fails:

error(160): org/hibernate/query/parser/internal/hql/antlr/HqlParser.g4:4:12: cannot find tokens file /home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/org/hibernate/query/parser/internal/hql/antlr/HqlLexer.tokens

The “problem” is that the plugin creates the package dir structure twice. The output is generated into /home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/org/hibernate/query/parser/internal/hql/antlr/org/hibernate/query/parser/internal/hql/antlr/ rather than /home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/org/hibernate/query/parser/internal/hql/antlr.

I figure part of the problem is the override of generateGrammarSource.outputDirectory (which was needed with 2.4). So I removed that, but that leads to a similar error:

error(160): org/hibernate/query/parser/internal/hql/antlr/HqlParser.g4:4:12: cannot find tokens file /home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/HqlLexer.tokens

Now the generated files are put into the proper directory (/home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/org/hibernate/query/parser/internal/hql/antlr). However, Antlr is not able to find things.

So how am I supposed to get Antlr 4 and Gradle 2.7 playing together nicely (using the Gradle AntlrPlugin)? I realize that there are other Antlr plugins for Gradle, is that my best option?

2

At Gradle2.7, ANTLR generated sources and packages structure is changed.

Gradle 2.7 release notes describes it as potential breaking changes.

https://docs.gradle.org/2.7/release-notes#changes-to-antlr-generated-sources-and-package-structure

3

Thanks for the link. So this seems a new regressions then. The problem is not where things are generated, the problem is that Antlr is later not finding them. So the plugin is not passing something correctly to Antlr tool.

4

Hey Steve,

seems to be indeed a regression. I might be able to look into it the
next days but from a brief look the only (inconvenient) workaround I see
at the moment is to move the antlr sources in the root folder without
the package structure. That might solve the problem.

cheers,
René

5

Thanks, Rene.

I’ll try to verify that (inconvenient) workaround for you. Obviously that’s not ideal long term, as you understand.

Any chance you can post the Jira you create for this so we can watch it? As for feedback, this is definitely a big heart-ache with the current forum->Jira system y’all have set up… we never get notified when things we report get resolved.

6

Moving the grammar sources to the src/main/antlr dir does “work”.

However, even then I still need to use generateGrammarSource.outputDirectory = packageDirStructure in order for the generated files to be put in the correct spot.

If I leave it off setting outputDirectory the generated files are put in /build/generated-src/antlr/main. The build “works” but IntelliJ complains and Eclipse just throws up its hands.

7

Hey Steve,

I raised GRADLE-3345 for this.

8

I saw you updated the output directory of the antlr task to the package folder. This causes the weired output path of your generated sources, as gradle appends the path calculated from the package declaration to the output directory. However, setting the outputdirectory to the root output results in problems with the tokenVocab statement.

I’ll update the linked issue accordingly. I think ideally gradle would calculate the dependencies created via tokenVocab and create the files in the accordant order. Furthermore it could use the -lib folder to point to the output (package) folder to get importVocab working within package structure.

We need to talk to the antlr team if they are willing to provide a 1st class gradle plugin for different antlr versions as they do for maven. I think in the long run that is the best way to get the best user experience and would take off some workload from the Gradle developers :wink:

cheers,
René

9

I think the other thing that would help (and pushing this to the Antlr developers would result in the same) is to split out the support for the different versions. At least some of the Antlr4-specific plugins out there handle this correctly.

The Antlr devs just would not care (most likely) about the Antlr 2/3 support. I cannot speak for them obviously, just my gut feel.

10

I totally agree with you that splitting the versions up is a good
strategy to provide a better, deeper integration. I reached out to the
antlr guys and asked them on their input on that. I’ll keep you updated.

cheers,
René

11

Rene, any word on this? Its come up again - Gradle 3 and Antlr