I have a problem while building AST in ANTLR (I'm using ANTLR 3.2, ANTLRWorks 1.4).
This is my grammar:
classDeclaration
:
(
'class' n=IDENTIFIER ('extends' e=IDENTIFIER)?
'{'
…
'}'
)
-> ^(CLASSDECLARATION ^(NAME $n) ^(EXTENDS $e)
;
The problem occurs with optional part of the class — ('extends' e=IDENTIFIER)?.
So the grammar works good with this class declaration:
class Test1 extends AbstractTest1 {
…
}
And fails when I exclude extends part, as follows:
class Test2 {
…
}
ANTLR just stops before this fragment and gives this exception in console:
javax.swing.text.BadLocationException: Position not represented by view
How can I point to ANTLR to handle rewrite rule ^(EXTENDS $e) as optional?
Got the problem solved. Nothing tricky, just had to use common RegExp syntax:
^(EXTENDS $e)?
Related
I have a simple grammar like so:
grammar Test;
generator : expression;
expression
: NUMBER # Number
| ID # String
| expression '+' expression # Add
;
NUMBER: [0-9]+ [0-9]*;
ID : [a-zA-Z_]+ [a-zA-Z0-9_]* ;
I want the expression 5xx to be considered an error (since it should be 5+xx or 5 or xx). With Antlr 4.6 this would happen, but with antlr 4.7 this doesn't happen.
Here's my full test:
#Test()
public void doATest() {
TestLexer lexer = new TestLexer(new ANTLRInputStream("5xx"));
TestParser parser = new TestParser(new CommonTokenStream(lexer));
//Walk the tree and throw if there are any error nodes.
ParseTreeWalker.DEFAULT.walk(new TestBaseListener() {
#Override public void visitErrorNode(ErrorNode node) {
//Throws with 4.6, not with 4.7
throw new RuntimeException("Hit error node: " + node);
}
}, parser.generator());
}
The other odd observation I have is that including the expression '+' expression rule is important, without this 4.6 won't generate an error either.
Is there some special flag that I need to set somewhere to indicate that an input stream should be exactly one generator and not have any trailing tokens?
Is there some special flag that I need to set somewhere to indicate that an input stream should be exactly one generator and not have any trailing tokens?
Yes, that's exactly what the EOF token does:
generator : expression EOF;
This way you'll always get an error on extra tokens, regardless of the version of ANTLR or whether or not you include the expression '+' expression rule.
I'm trying to implement a lexer rule for an oracle Q quoted string mechanism where we have something like q'$some string$'
Here you can have any character in place of $ other than whitespace, (, {, [, <, but the string must start and end with the same character. Some examples of accepted tokens would be:
q'!some string!'
q'ssome strings'
Notice how s is the custom delimiter but it is fine to have that in the string as well because we would only end at s'
Here's how I was trying to implement the rule:
Q_QUOTED_LITERAL: Q_QUOTED_LITERAL_NON_TERMINATED . QUOTE-> type(QUOTED_LITERAL);
Q_QUOTED_LITERAL_NON_TERMINATED:
Q QUOTE ~[ ({[<'"\t\n\r] { setDelimChar( (char)_input.LA(-1) ); }
( . { !isValidEndDelimChar() }? )*
;
I have already checked the value I get from !isValidEndDelimChar() and I'm getting a false predicate here at the right place so everything should work, but antlr simply ignores this predicate. I've also tried moving the predicate around, putting that part in a separate rule, and a bunch of other stuff, after a day and a half of research on the same I'm finally raising this issue.
I have also tried to implement it in other ways but there doesn't seem to be a way to implement a custom char delimited string in antlr4 (The antlr3 version used to work).
Not sure why the { ... } action isn't invoked, but it's not needed. The following grammar worked for me (put the predicate in front of the .!):
grammar Test;
#lexer::members {
boolean isValidEndDelimChar() {
return (_input.LA(1) == getText().charAt(2)) && (_input.LA(2) == '\'');
}
}
parse
: .*? EOF
;
Q_QUOTED_LITERAL
: 'q\'' ~[ ({[<'"\t\n\r] ( {!isValidEndDelimChar()}? . )* . '\''
;
SPACE
: [ \t\f\r\n] -> skip
;
If you run the class:
import org.antlr.v4.runtime.*;
public class Main {
public static void main(String[] args) {
Lexer lexer = new TestLexer(CharStreams.fromString("q'ssome strings' q'!foo!'"));
CommonTokenStream tokens = new CommonTokenStream(lexer);
tokens.fill();
for (Token t : tokens.getTokens()) {
System.out.printf("%-20s %s\n", TestLexer.VOCABULARY.getSymbolicName(t.getType()), t.getText());
}
}
}
the following output will be printed:
Q_QUOTED_LITERAL q'ssome strings'
Q_QUOTED_LITERAL q'!foo!'
EOF <EOF>
According to the antlr4 book (page 159), and using the grammar Ambig.g4, grammar ambiguity can be reported by:
grun Ambig stat -diagnostics
or equivalently, in code form:
parser.removeErrorListeners();
parser.addErrorListener(new DiagnosticErrorListener());
parser.getInterpreter().setPredictionMode(PredictionMode.LL_EXACT_AMBIG_DETECTION);
The grun command reports the ambiguity properly for me, using antlr-4.5.3. But when I use the code form, I dont get the ambiguity report. Here is the command trace:
$ antlr4 Ambig.g4 # see the book's page.159 for the grammar
$ javac Ambig*.java
$ grun Ambig stat -diagnostics < in1.txt # in1.txt is as shown on page.159
line 1:3 reportAttemptingFullContext d=0 (stat), input='f();'
line 1:3 reportAmbiguity d=0 (stat): ambigAlts={1, 2}, input='f();'
$ javac TestA_Listener.java
$ java TestA_Listener < in1.txt # exits silently
The TestA_Listener.java code is the following:
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.atn.*; // for PredictionMode
import java.util.*;
public class TestA_Listener {
public static void main(String[] args) throws Exception {
ANTLRInputStream input = new ANTLRInputStream(System.in);
AmbigLexer lexer = new AmbigLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
AmbigParser parser = new AmbigParser(tokens);
parser.removeErrorListeners(); // remove ConsoleErrorListener
parser.addErrorListener(new DiagnosticErrorListener());
parser.getInterpreter().setPredictionMode(PredictionMode.LL_EXACT_AMBIG_DETECTION);
parser.stat();
}
}
Can somebody please point out how the above java code should be modified, to print the ambiguity report?
For completeness, here is the code Ambig.g4 :
grammar Ambig;
stat: expr ';' // expression statement
| ID '(' ')' ';' // function call statement
;
expr: ID '(' ')'
| INT
;
INT : [0-9]+ ;
ID : [a-zA-Z]+ ;
WS : [ \t\r\n]+ -> skip ;
And here is the input file in1.txt :
f();
Antlr4 is a top-down parser, so for the given input, the parse match is unambiguously:
stat -> expr -> ID -> ( -> ) -> stat(cnt'd) -> ;
The second stat alt is redundant and never reached, not ambiguous.
To resolve the apparent redundancy, a predicate might be used:
stat: e=expr {isValidExpr($e)}? ';' #exprStmt
| ID '(' ')' ';' #funcStmt
;
When isValidExpr is false, the function statement alternative will be evaluated.
I waited for several days for other people to post their answers. Finally after several rounds of experimenting, I found an answer:
The following line should be deleted from the above code. Then we get the same ambiguity report as given by grun.
parser.removeErrorListeners(); // remove ConsoleErrorListener
The following code will be work
public static void main(String[] args) throws IOException {
CharStream input = CharStreams.fromStream(System.in);
AmbigLexer lexer = new AmbigLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
AmbigParser parser = new AmbigParser(tokens);
//parser.removeErrorListeners(); // remove ConsoleErrorListener
parser.addErrorListener(new org.antlr.v4.runtime.DiagnosticErrorListener()); // add ours
parser.getInterpreter().setPredictionMode(PredictionMode.LL_EXACT_AMBIG_DETECTION);
parser.stat(); // parse as usual
}
This does not compile in ANTLR 4:
Number options { backtrack=true; }
: (IntegerLiteral Range)=> IntegerLiteral { $type = IntegerLiteral; }
| (FloatLiteral)=> FloatLiteral { $type = FloatLiteral; }
| IntegerLiteral { $type = IntegerLiteral; }
;
because of backtrace= true... What happened to it?
WHat should I use in ANTLR 4 instread of it?
At the moment, there are no rule-level options in ANTLR v4. Note that backtrack=true is no longer needed since the new parsing algorithm has no need for backtracking. Also note that in ANTLR v3, backtrack=true was not valid inside lexer rules, only parser rules.
I examine heterogeneous trees in ANTLR (using ANTLRWorks 1.4.2).
Here is the example of what I have already done in ANTLR.
grammar test;
options {
language = java;
output = AST;
}
tokens {
PROGRAM;
VAR;
}
#members {
class Program extends CommonTree {
public Program(int ttype) {
token = new CommonToken(ttype, "<start>");
}
}
}
start
: program var function
// Works fine:
//-> ^(PROGRAM program var function)
// Does not work (described below):
-> ^(PROGRAM<Program> program var function)
;
program
: 'program'! ID ';'!
;
var
: TYPE^ ID ';'!
;
function
: ID '('! ')'! ';'!
;
TYPE
: 'int'
| 'string'
;
ID
: ('a'..'z' | 'A'..'Z')+
;
WHITESPACE
: (' ' | '\t' '\n'| '\r' | '\f')+ {$channel = HIDDEN;}
;
Sample input:
program foobar;
int foo;
bar();
When I use rewrite rule ^(PROGRAM<Program> program var function), ANTLR stumbles over and I get AST like this:
Whereas when I use this rewrite rule ^(PROGRAM program var function) it works:
Could anyone explain where am I wrong, please? Frankly, I do not really get the idea of heterogeneous trees and how do I use <…> syntax in ANTLR.
What do r0 and r1 mean (first picture)?
I have no idea what these r0 and r1 mean: I don't use ANTLRWorks for debugging, so can't comment on that.
Also, language = java; causes ANTLR 3.2 to produce the error:
error(10): internal error: no such group file java.stg
error(20): cannot find code generation templates java.stg
error(10): internal error: no such group file java.stg
error(20): cannot find code generation templates java.stg
ANTLR 3.2 expects it to be language = Java; (capital "J"). But, by default the target is Java, so, mind as well remove the language = ... entirely.
Now, as to you problem: I cannot reproduce it. As I mentioned, I tested it with ANTLR 3.2, and removed the language = java; part from your grammar, after which everything went as (I) expected.
Enabling the rewrite rule -> ^(PROGRAM<Program> program var function) produces the following ATS:
and when enabling the rewrite rule -> ^(PROGRAM program var function) instead, the following AST is created:
I tested both rewrite rules this with the following class:
import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;
public class Main {
public static void main(String[] args) throws Exception {
ANTLRStringStream in = new ANTLRStringStream("program foobar; int foo; bar();");
testLexer lexer = new testLexer(in);
CommonTokenStream tokens = new CommonTokenStream(lexer);
testParser parser = new testParser(tokens);
testParser.start_return returnValue = parser.start();
CommonTree tree = (CommonTree)returnValue.getTree();
DOTTreeGenerator gen = new DOTTreeGenerator();
StringTemplate st = gen.toDOT(tree);
System.out.println(st);
}
}
And the images are produced using graph.gafol.net (and the output of the Main class, of course).