Using Gradle 2.10's ANTLR plugin to import an ANTLR 4 lexer grammar into another grammar


(Jonathan Bluett-Duncan) #1

Hi everyone. I’m experiencing a problem with Gradle 2.10’s built-in ANTLR plugin, but I’m not sure if it’s due to a bug or a misunderstanding on my part, so I’m posting my question here in Help/Discuss in case it’s the latter. (I posted this question originally on StackOverflow, but I wasn’t able to get a useful answer, so I’m reposting it here in the hope someone can help me.)

I’ve been learning about ANTLR 4 with Terence Parr’s The Definitive ANTLR 4 Reference, which I’ve been following so far using Gradle 2.10 and its built-in ANTLR plugin. However I’m having some trouble getting some code, which I adapted from the book, to work properly with my Gradle build script. (The reason I’m using Gradle, rather than ANTLR directly, is because I want to eventually integrate ANTLR into a Java web application which I’m making for my university dissertation, and I’d strongly prefer to use a build tool so I can automate the ANTLR-to-Java code generation process and easily manage my dependencies.)

I’ve created two ANTLR 4 grammars (see the end of this question): src/main/antlr/org/jbduncan/Expr.g4 (a standard grammar) and src/main/antlr/org/jbduncan/CommonLexerRules.g4 (a lexer grammar), where Expr.g4 depends on CommonLexerRules.g4 via an import CommonLexerRules statement. But when I try to run gradlew generateGrammarSource at the command-line with my build.gradle and gradle.properties (see the end of this question), I get the following error:

12:45:21: Executing external task 'generateGrammarSource'...
:generateGrammarSource
error(110): org\jbduncan\Expr.g4:3:7: can't find or load grammar CommonLexerRules
warning(125): org\jbduncan\Expr.g4:12:11: implicit definition of token NEWLINE in parser
warning(125): org\jbduncan\Expr.g4:13:6: implicit definition of token ID in parser
warning(125): org\jbduncan\Expr.g4:19:6: implicit definition of token INT in parser
:generateGrammarSource FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':generateGrammarSource'.
> There was 1 error during grammar generation

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 3.166 secs
There was 1 error during grammar generation
12:45:24: External task execution finished 'generateGrammarSource'.

Looking at this error message, it seems to me that Gradle’s ANTLR plugin is able to find Expr.g4 but is somehow not able to find CommonLexerRules.g4.

I’ve made an attempt to resolve this error using a couple of external Gradle plugins (https://github.com/melix/antlr4-gradle-plugin and https://github.com/Abnaxos/gradle-antlr4-plugin) instead of the built-in one, but when I tried each of them, they introduced their own problems which I wasn’t able to resolve.

Using the ANTLR 4.5.2 jar downloaded straight from the ANTLR website does let me to compile my two grammars without problems, but as I said earlier this is undesirable for me.

My question is this: How can I resolve the error above and get Gradle to successfully compile my main grammar with my lexer grammar imported?

Any input would be welcome.


build.gradle

apply plugin: 'java'
apply plugin: 'application'
apply plugin: 'antlr'

sourceCompatibility = 1.8
targetCompatibility = 1.8

[compileJava, compileTestJava]*.options*.encoding = 'UTF-8'

group = 'org.jbduncan'
version = '1.0-SNAPSHOT'
mainClassName = 'org.jbduncan.Application'

repositories {
    jcenter()
}

dependencies {
    antlr "org.antlr:antlr4:$antlrVersion"
    compile "com.google.guava:guava:$guavaVersion"
    testCompile "com.google.guava:guava-testlib:$guavaVersion"
    testCompile "com.google.truth:truth:$truthVersion"
    testCompile "junit:junit:$junitVersion"
}

// Send all generated source code to a directory other than build/, to workaround an issue in
// IntelliJ IDEA where it fails to recognise source files in build/.
def generatedJavaSourcesDir = 'src/generated/java'

generateGrammarSource {
    arguments += ['-visitor']
    outputDirectory = file(generatedJavaSourcesDir)
}

sourceSets {
    main {
        java {
            srcDir generatedJavaSourcesDir
        }
    }
}

clean {
    delete generatedJavaSourcesDir
}

task wrapper(type: Wrapper) {
    distributionUrl = "http://services.gradle.org/distributions/gradle-$gradleVersion-bin.zip"
}

gradle.properties

gradleVersion=2.10

# Dependency versions
antlrVersion=4.5
guavaVersion=19.0
junitVersion=4.12
truthVersion=0.28

src/main/antlr/org/jbduncan/Expr.g4

grammar Expr;

import CommonLexerRules; // includes all rules from CommonLexerRules.g4

@header {
    package org.jbduncan;
}

/** The start rule; begin parsing here. */
prog: stat+;

stat: expr NEWLINE # printExpr
    | ID '=' expr NEWLINE # assign
    | NEWLINE # blank
    ;

expr: expr op=('*'|'/') expr # MulDiv
    | expr op=('+'|'-') expr # AddSub
    | INT # int
    | ID # id
    | '(' expr ')' # parens
    ;

MUL: '*'; // assigns token name to '*' used above in grammar
DIV: '/';
ADD: '+';
SUB: '-';

src/main/antlr/org/jbduncan/CommonLexerRules.g4

lexer grammar CommonLexerRules; // note "lexer grammar"

ID: [a-zA-Z]+ ; // match identifiers
INT: [0-9]+ ; // match integers
NEWLINE: '\r'? '\n' ; // return newlines to parser (is end-statement signal)
WS: [ \t] -> skip ; // toss out whitespace

src/main/java/org/jbduncan/Application.java

package org.jbduncan;

import org.antlr.v4.runtime.ANTLRInputStream;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.tree.ParseTree;

import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;

public final class Application {
  public static void main(String[] args) throws IOException {
    String inputFile = (args.length > 0) ? args[0] : null;

    InputStream inputStream = (inputFile == null) ? System.in : new FileInputStream(inputFile);
    ANTLRInputStream input = new ANTLRInputStream(inputStream);
    ExprLexer lexer = new ExprLexer(input);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    ExprParser parser = new ExprParser(tokens);
    ParseTree tree = parser.prog(); // parse; start at prog

    System.out.println(tree.toStringTree(parser)); // print tree as text
  }
}

Using ANTLR plugin to process separate lexer and parser grammars
(Stefan Oehme) #2

It would be much easier to help you with this problem, if you attached a minimal example project that people can try out. Just looking at walls of text, it is hard to figure out what might be wrong :slight_smile:


(Jonathan Bluett-Duncan) #3

Ah, oops! That make sense. :slightly_smiling:

Interestingly, when I was creating a minimal sample project just now, I discovered that my ANTLR files compiled successfully if they were stored under src/main/antlr rather than src/main/antlr/org/jbduncan (where org.jbduncan is my main Java package).

I was going to upload two projects just now, one for the case where the ANTLR files were stored in a Java package, and another where they weren’t, but apparently as I’m a new user I can’t upload any files?

Do you have any suggestions, @st_oehme?


(Stefan Oehme) #4

Good news that you already found a probable cause :slight_smile:

I upgraded you to basic user, so you can attach your example :wink:


(Jonathan Bluett-Duncan) #5

Great, thank you! Here are my two sample projects. antlr-example-with-java-package.zip (79.9 KB)
antlr-example-without-java-package.zip (80.0 KB)

From my (admittedly limited) understanding of ANTLR 4, I’m under the impression that both projects should compile (that is, it shouldn’t matter whether ANTLR files are stored under a Java package or not). My experiment with using ANTLR directly rather than Gradle seems to confirm this. But as I described earlier, when I use Gradle, only the project where the ANTLR files are not put in a Java package compiles successfully. Could this be a bug in Gradle’s ANTLR plugin?


(Heinrich Nirschl) #6

I had the same problem a few weeks ago. Putting everything into the root (src/main/antlr) did not work for me because I really wanted the generated files to be in a package and the IDE did not pick up the generated files when they were in the wrong place.

It seems, antlr is not able to find the lexer.tokens file when compiling the parser. I also tried to specify the location with the lib parameter similar to this

generateGrammarSource {
arguments = ["-lib", “…/…/…/build/generated-src/antlr/main/my/package”]
}

but this does not work either because antlr checks if the directory exists and the plugin removes it before antlr is called.


(Stefan Oehme) #8

Found the solution in our own integration tests :slightly_smiling:

generateGrammarSource {
  arguments << "-lib" << "src/main/antlr/org/jbduncan" 
}

(Heinrich Nirschl) #9

Thanks, but this only works if some other tool (eg. an IDE plugin) generates the lexer.tokens file in the source directory.

Note that the lexer.tokens file is normally generated by antlr in the destination directory when it processes the lexer grammar. This also means that the lexer must be processed before the parser which I guess depends on the file names (lexical order).


(Stefan Oehme) #10

Hi Heinrich,

my suggestion fixes Jonathan’s project (and there is no lexer tokens file
in the source sirectory).

Can you please open another topic to discuss your problem? It seems to be a
different issue.

Cheers,
Stefan


(Jonathan Bluett-Duncan) #11

Many thanks @st_oehme, your solution works for me! :grinning:

I wonder if you can explain to me how your solution fixes my problem? It’s not obvious to me how it works.

Could you also clarify if the problem was due to a bug or just a simple misunderstanding on my part?


(Cerbenus) #12

For those that try to make ANTLR4 work with Gradle:

I was now trying for 4 evenings to find out why my gradle build would fail. In the end I upgraded from version 2.8 to 2.12 and volia, the whole magic worked out of the blue… Hope it helps somebody else.