Gradle 2.7 & 2.8 causing ANTLR Panic Cannot find importVocab file

Hi everyone,

I know there has been a change in ANTLR package generation in Gradle 2.7, but this looks like a bug nevertheless. Our generation worked with Gradle 2.6 but fails with Gradle 2.7. I’m an antlr newbie so I can’t dive deep into it, but the very same antlr *.g files used to work with ant back in the days. When I run with --info, I get the following:

All input files are considered out-of-date for incremental task ':mymodule:generateGrammarSource'.
Starting process 'Gradle ANTLR Worker 1'. Working directory: C:\myproject\mymodule Command: C:\Programme\Java\jdk1.6
\bin\java.exe -DANTLR_DO_NOT_EXIT=true -Dfile.encoding=windows-1252 -Duser.country=DE -Duser.language=de -Duser.vari
ant -cp C:\Program Files\gradle\gradle\lib\gradle-base-services-2.7.jar;C:\Program Files\gradle\gradle\lib\gradle-co
re-2.7.jar;C:\Program Files\gradle\gradle\lib\gradle-cli-2.7.jar;C:\Program Files\gradle\gradle\lib\gradle-native-2.
7.jar;C:\Program Files\gradle\gradle\lib\gradle-messaging-2.7.jar;C:\Program Files\gradle\gradle\lib\slf4j-api-1.7.1
0.jar;C:\Program Files\gradle\gradle\lib\jul-to-slf4j-1.7.10.jar;C:\Program Files\gradle\gradle\lib\guava-jdk5-17.0.
jar org.gradle.process.internal.launcher.IsolatedGradleWorkerMain 'Gradle ANTLR Worker 1'
Successfully started process 'Gradle ANTLR Worker 1'
Processing with ANTLR 2
C:\myproject\mymodule\build\generated-src\antlr\main\ame : qualifiedName\InterfaceLexer.java.antlr.tmp (Die Syntax 
für den Dateinamen, Verzeichnisnamen oder die Datentrõgerbezeichnung ist falsch)
panic: Cannot find importVocab file 'InterfaceTokenTypes.txt'
:mymodule:generateGrammarSource FAILED
:mymodule:generateGrammarSource (Thread[main,5,main]) completed. Took 0.98 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':mymodule:generateGrammarSource'.
> There was 1 error during grammar generation
   > ANTLR Panic: panic: Cannot find importVocab file 'InterfaceTokenTypes.txt'

The weird part seems to be the path: C:\myproject\mymodule\build\generated-src\antlr\main\ame : qualifiedName\InterfaceLexer.java.antlr.tmp, especially the ame : qualifiedName part. Searching our Antlr grammar files, I found the following line:

packageName : qualifiedName;

My guess is that Gradle tries to extract the package declaration from the grammar file but that extraction is broken. Can anyone confirm this?

Hey Mike,
can you provide a small self contained reproducible example. We have
quite a test coverage for our antlr support.

cheers,
René

Hi Rene, could you try the example I creted here:

https://issues.gradle.org/browse/GRADLE-3262?focusedCommentId=20164&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-20164

Our company now blocks cloud storage providers, so I cannot either check with that link, nor upload a new one. :frowning:

Any updates on that issue? Have you been able to reproduce it?

This issue still persists with Gradle 2.8.

Has there been further analysis meanwhile?
We’re stuck with Gradle 2.6 for our Antlr builds :confused:

I have to express my feelings here.

I’m part of the Gradle community since 0.7 or so. I always had a positive experience with the Gradle forums and the core-team feedback and support. I reported several Gradle issues and submitted PRs.

I’m a bit disappointed about how this forum is developing lately. It seems that with the higher focus Gradle is getting, the core team members are busy doing other things instead of helping us solve the issues their regressions cause. It’s not the first one, but this thread is a good example.

That being said, as I don’t seem to get any more attention, I went ahead and replaced our Gradle Antlr plugin usage with good old ant. In case you’re also getting the broken behavior we are seeing, you might consider doing the same. Here are all elements you’ll need:

//apply plugin: 'antlr' // disable that.

configurations {
    antlr
}

ext.antlrOutputDir = "${buildDir}/generated-src/antlr/main"
ext.antlrInputDir = "${projectDir}/src/main/antlr"
sourceSets.main.java.srcDir antlrOutputDir

dependencies {
    antlr (group: 'antlr', name: 'antlr', version: '2.7.7')
    antlr (group: 'org.apache.ant', name: 'ant-antlr', version: '1.9.6')
}

task generateGrammarSource(dependsOn: configurations.antlr) {
    inputs.dir antlrInputDir
    outputs.dir antlrOutputDir

    doLast {
        def inputDir = file("${antlrInputDir}/com/foo")
        def outputDir = file("${antlrOutputDir}/com/foo")
        outputDir.mkdirs()

        ant.taskdef( name: 'antlr',
                     classname: 'org.apache.tools.ant.taskdefs.optional.ANTLR',
                     classpath: configurations.antlr.asPath)
        ant.antlr( target: "${inputDir}/file.g",
                   outputdirectory: outputDir,
                   tracetreewalker: 'false',
                   tracelexer: 'false',
                   debug: 'false',
                   html: 'false',
                   traceparser: 'false' );
    }
}

compileJava.dependsOn generateGrammarSource

Of course, you’ll have to repeat the ant.antlr call for all of your *.g files.

Hey Mike,

Mea culpa! Sorry for my late response on this. I had marked me a todo for that issue but somehow it slipped through. I wasn’t able to reproduce the issue with the sample project you provided. I don’t see any panic error message. Neither with gradle 2.2 nor with gradle 2.8. The only issue I see across multiple versions is that the sample project doesn’t compile (more likely because of different issues). I’m glad you found a workaround here but I’d like to reproduce this issue in detail.

On more comment on the ANTLR plugin we ship with Gradle. Since we’re not ANTLR experts and it requires a lot of effort to maintain that plugin we would really like the antlr community to step in and provide a first class third party ANTLR plugin for gradle that can be maintained in the gradle plugin portal. As a starting point we would definitely support this and provide the initial implementation (which could be copied more or less from the existing plugin) I reached out to the antlr mailinglist for this but unfortunately havn’t received any response so far. At the moment the antlr plugin supports ANTLR 2, 3, 4.
Maybe for supporting more version specific features it the ANTLR plugin needs to be splitup in antlr2 plugin etc.
Again, could you provide a small example that fully shows the issue you ran into here?

cheers,
René

Hi René,

Glad to hear you’ve not given up on this :smile:

I feel your pain: I’m an ANTLR novice as well. I “inherited” the project from a former colleague and I don’t unterstand much of the ANTLR part other than “it works” or “it doesn’t”. That’s also the reason why I’m having a hard time trying to build a minimalistic reproducible piece of ANTLR file (either it’s the whole thing, including a bazillion dependencies, or I’m facing ANTLR errors as soon as I try to cut down the *.g files). Let me try to recap my findings again.

From the error I get:

C:\myproject\mymodule\build\generated-src\antlr\main\ame : qualifiedName\InterfaceLexer.java.antlr.tmp (Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datentrõgerbezeichnung ist falsch)

I’m guessing that it must relate to the package-to-filesystem-path handling introduced in Gradle 2.7 ( https://issues.gradle.org/browse/GRADLE-3323 ), because of the ame : qualifiedName part in it. I have a line in one of my *.g files reading something like:

packageName : qualifiedName;

We’re generating code with it, so the term package appears in different places throughout the files and I guess the Gradle ANTLR plugin must be picking up the wrong thing here.

I hope this can help you reproducing the error.

Best regards,
Mike

most likely it is related to the ANTLR package handling introduced in Gradle 2.7. but still not sure how to reproduce this. Do you see the error with the sample you provided as attachment to GRADLE-3262?

That is the latest news on this bug?