I am new to ANTLR , and walking through existing grammar(got at Internet). See the given rule , i am not able to understand it for what is it all about?
Specially $model_expr inside Tree construct and initial (unary_expr -> unary_expr). Please help me understanding the same.
model_expr
: (unary_expr -> unary_expr)
(LEFT_BRACKET model_expr_element RIGHT_BRACKET
-> ^(MODEL_EXPR[$LEFT_BRACKET] $model_expr model_expr_element))?
;
Thanks
See the detailed explanation of above syntax with example (copied from book)
Referencing Previous Rule ASTs in Rewrite Rules
Sometimes you can’t build the proper AST in a purely declarative manner. In other words, executing a single rewrite after the parser has matched everything in a rule is insufficient. Sometimes you need to iteratively build up the AST . To iteratively build an AST, you need to be able to reference the previous value of the current rule’s AST. You can reference the previous value by using $r within a rewrite rule where r
is the enclosing rule. For example, the following rule matches either a single integer or a series of integers addedtogether:
expr : (INT -> INT) ( '+' i=INT -> ^( '+' $expr $i) ) * ;
The (INT->INT) subrule looks odd but makes sense. It says to match INT
and then make its AST node the result of expr . This sets a result AST
in case the (...)* subrule that follows matches nothing. To add another
integer to an existing AST, you need to make a new ’+’ root node that
has the previous expression as the left child and the new integer as the
right child.
That grammar with embedded rewrite rules recognizes the same input and generates the same tree as the following version that uses the construction operators:
expr : INT ('+'^ INT)*;
Related
I am looking for a solution to a simple problem.
The example :
SELECT date, date(date)
FROM date;
This is a rather stupid example where a table, its column, and a function all have the name "date".
The snippet of my grammar (very simplified) :
simple_select
: SELECT selected_element (',' selected_element) FROM from_element ';'
;
selected_element
: function
| REGULAR_WORD
;
function
: REGULAR_WORD '(' function_argument ')'
;
function_argument
: REGULAR_WORD
;
from_element
: REGULAR_WORD
;
DATE: D A T E;
FROM: F R O M;
SELECT: S E L E C T;
REGULAR_WORD
: (SIMPLE_LETTER) (SIMPLE_LETTER | '0'..'9')*
;
fragment SIMPLE_LETTER
: 'a'..'z'
| 'A'..'Z'
;
DATE is a keyword (it is used somewhere else in the grammar).
If I want it to be recognised by my grammar as a normal word, here are my solutions :
1) I add it everywhere I used REGULAR_WORD, next to it.
Example :
selected_element
: function
| REGULAR_WORD
| DATE
;
=> I don't want this solution. I don't have only "DATE" as a keyword, and I have many rules using REGULAR_WORD, so I would need to add a list of many (50+) keywords like DATE to many (20+) parser rules : it would be absolutely ugly.
PROS: make a clean tree
CONS: make a dirty grammar
2) I use a parser rule in between to get all those keywords, and then, I replace every occurrence of REGULAR_WORD by that parser rule.
Example :
word
: REGULAR_WORD
| DATE
;
selected_element
: function
| word
;
=> I do not want this solution either, as it adds one more parser rule in the tree and polluting the informations (I do not want to know that "date" is a word, I want to know that it's a selected_element, a function, a function_argument or a from_element ...
PROS: make a clean grammar
CONS: make a dirty tree
Either way, I have a dirty tree or a dirty grammar. Isn't there a way to have both clean ?
I looked for aliases, parser fragment equivalent, but it doesn't seem like ANTLR4 has any ?
Thank you, have a nice day !
There are four different grammars for SQL dialects in the Antlr4 grammar repository and all four of them use your second strategy. So it seems like there is a consensus among Antlr4 sql grammar writers. I don't believe there is a better solution given the design of the Antlr4 lexer.
As you say, that leads to a bit of noise in the full parse tree, but the relevant non-terminal (function, selected_element, etc.) is certainly present and it does not seem to me to be very difficult to collapse the unit productions out of the parse tree.
As I understand it, when Antlr4 was being designed, a decision was made to only automatically produce full parse trees, because the design of condensed ("abstract") syntax trees is too idiosyncratic to fit into a grammar DSL. So if you find an AST more convenient, you have the responsibility to generate one yourself. That's generally straight-forward although it involves a lot of boilerplate.
Other parser generators do have mechanisms which can handle "semireserved keywords". In particular, the Lemon parser generator, which is part of the Sqlite project, includes a %fallback declaration which allows you to specify that one or more tokens should be automatically reclassified in a context in which no grammar rule allows them to be used. Unfortunately, Lemon does not generate Java parsers.
Another similar option would be to use a parser generator which supports "scannerless" parsing. Such parsers typically use algorithms like Earley/GLL/GLR, capable of parsing arbitrary CFGs, to get around the need for more lookahead than can conveniently be supported in fixed-lookahead algorithms such as LALR(1).
This is the socalled keywords-as-identifiers problem and has been discussed many times before. For instance I asked a similar question already 6 years ago in the ANTLR mailing list. But also here at Stackoverflow there are questions touching this area, for instance Trying to use keywords as identifiers in ANTLR4; not working.
Terence Parr wrote a wiki article for ANTLR3 in 2008 that shortly describes 2 possible solutions:
This grammar allows "if if call call;" and "call if;".
grammar Pred;
prog: stat+ ;
stat: keyIF expr stat
| keyCALL ID ';'
| ';'
;
expr: ID
;
keyIF : {input.LT(1).getText().equals("if")}? ID ;
keyCALL : {input.LT(1).getText().equals("call")}? ID ;
ID : 'a'..'z'+ ;
WS : (' '|'\n')+ {$channel=HIDDEN;} ;
You can make those semantic predicates more efficient by intern'ing those strings so that you can do integer comparisons instead of string compares.
The other alternative is to do something like this
identifier : KEY1 | KEY2 | ... | ID ;
which is a set comparison and should be faster.
Normally, as #rici already mentioned, people prefer the solution where you keep all keywords in an own rule and add that to your normal identifier rule (where such a keyword is allowed).
The other solution in the wiki can be generalized for any keyword, by using a lookup table/list in an action in the ID lexer rule, which is used to check if a given string is a keyword. This solution is not only slower, but also sacrifies clarity in your parser grammar, since you can no longer use keyword tokens in your parser rules.
I have been working with Antlr, trying to parse and store the (grammars in antlr ) .g4 files in a data structure, so that i can be able to mutate over the rules, and then run antlr on garmmars with mutated rules. I have ANTLRv4Parser grammars, I am trying to write a listener that walks down the tree storing the tokens. However, doing that worked but for rules with alternatives the pipe "|" symbol appears to be off. This comes from the following rule in the antlrv4parser grammar, ruleAltList : alternative (OR alternative)* . So it seems i'm struggling to get tokens from child nodes of alternative before the pipe and then after the pipe in enterRuleAltList in my listener, it seems antlr does the preorder traversal so it gets the pipe before heading down to alternative.
so what i want is maybe using the same listener patterns in antlr with some sort of an inorder traversal.
here's snippet from antlrv4parser grammar
ruleAltList: labeledAlt (OR labeledAlt)* ;
the anltrv4parser grammar and other grammars can be found on this link https://github.com/antlr/grammars-v4/tree/master/antlr4
For example if i have the following grammar
grammar c;
A : B | C;
I want to be able to store in a data structure as
["A", ":", "B","|","C",";"]
what i get is
["A", ":", "|","B", "C",";"]
So any ideas on how to override the enterRuleAltList method in my listener to have tokens from alternative child node before the OR, which is "|"?
Reduced representation of the grammar:
parserRuleSpec
: RULE_REF COLON ruleBlock SEMI
;
ruleBlock
: ruleAltList
;
ruleAltList
: labeledAlt (OR labeledAlt)*
;
labeledAlt
: terminal
;
Collecting all terminals of the nodes as encountered during the walk should result in an ordering ["A", ":", ";", "|", "B", "C"]. (Post the actual full listener code if what was originally given was not a typo.)
enterParserRuleSpec -> A : ;
enterRuleBlock
enterRuleAltList -> |
enterLabeledAlt
enterTerminal -> B
enterLabeledAlt
enterTerminal -> C
When collecting the terminals, attention must be paid to their order in the list of context children relative to their sibling non-terminals.
Or, possibly, just collect the terminals into a list sorted by token index.
I have an AvailableCommands hash-set member. It's loaded with names before lexing/parsing starts from external source. How can I check if current CMD token exists in the set?
Something like:
CMD : [-_.0-9a-zA-Z]+ {AvailableCommands.Contains(/* what goes here? */);}?
-> mode(IN_CMD)
;
I want to do it in the lexer because it's context-sensitive: command is lexed/parsed differently than, say, variables.
Matching a lexer rule constructs a Token that is available in Lexer._token.
CMD : [-_.0-9a-zA-Z]+ {AvailableCommands.Contains(_token.getText());}?
-> mode(IN_CMD)
;
The Antlr documentation is fairly clear on point.
Is there any means to get ANTLR4 to automatically remove redundant nodes in generated parse trees?
More specifically, I've been experimenting with a grammar for GLSL and you end up with long linear sequences of "expressions" in the parse tree due to the rule forwarding needed to give the automatic handling of operator precedence.
Most of the generated tree nodes are simply "forward to the next level of precedence", so don't provide any useful syntactic information - you only really need the last expression node in each sequence (i.e. the point at which the rule forwarding stopped), or the point where it becomes an actual tree node with more than one child (i.e. an actual expression was encountered in the source) ...
I was hoping there would be an easy way to kill off the dummy intermediate expression nodes - this type of structure must be common in any grammar with operator precedence.
The basic structure of the grammar is a fairly direct clone taken from the Khronos specification for the language:
https://www.khronos.org/registry/gles/specs/3.1/es_spec_3.1.pdf
ANTLR v4 is able to generate code from a single recursive rule dealing with different precedence levels, if you use a grammar like this (example for basic math):
expr : '(' expr ')'
| '-' expr
| expr ('*'|'/') expr
| expr ('+'|'-') expr
| INT
;
ANTLR v3 was unable to do so and basically required you to write one rule per precedence level. So I'd advise you to rewrite your grammar to avoid these boilerplate rules.
Then, I think you're confusing the parse tree (aka concrete syntax tree) with the AST (abstract syntax tree). The AST is like a simplified version of the parse tree, which keeps only what's needed for your purpose. For instance, with the expr rule above, the AST wouldn't contain any node for parentheses, since the precedence is encoded in the tree itself and you usually don't need to know whether a part of a given expression was parenthesized or not.
Your program should build an AST from the parse tree and then go from there. Don't deal with parse trees directly, even if it seems convenient at first sight because the tool generates them for you. It'll quickly become cumbersome. Build your own tree structure (AST), tailored for the task at hand.
Use the Visitor implementation to access each node in sequence. Build your own tree by adding nodes to parents as they are visited. Decide at the time the node is visited whether to add it to your new tree or not. For example:
public T visitExpression(#NotNull AcParser.ExpressionContext ctx) {
// Expressionable parent = getParent(Expressionable.class, ctx);
// Class<? extends AcExpression> expClass = AcExpression.class;
AcExpression obj = null;
String text = ctx.getText();
//do something with text or children
for (int i=0; i<ctx.getChildCount(); i++){
printnl(ctx.getChild(i).getText()+"/");
}
return visitChildren(ctx);
}
I have a grammar Foo.xtext (too complex to include it here). Xtext generates InternalFoo.g from it. After some tweaking it also generates DebugInternalFoo.g which claims to be the same thing without actions. Now, I strip off actions with ANTLR directly
java -cp antlr-3.4.jar org.antlr.tool.Strip Internal.g > Stripped.g
I'd expect the three grammars to behave the same way when I check them. But here is what I experienced
InternalFoo.g - error, rule assignment has non-LL(*) decision
DebugInternalFoo.g - no problem, parses fine
Stripped.g - warnings at rule assignment, decision can match using multiple alternatives. It fails to parse properly.
Is it possible that a grammar parses a text differently with or without actions? Or is it a bug in any of the action-remover tools? (The rule in question has syntactic predicates, and without them, it would really have a non-LL(*) decision.)
UPDATE:
I partly found what caused the problem. The rule in question was like this
trickyRule:
({ some complex action})
(expression '=')=>...
Stripping with Antlr removed the action, but left an empty group there:
// Stripped.g
trickyRule:
() (expression '=')=>...
The generation of the debug grammar removes both the action, and the now empty group around it:
// DebugInternalFoo.g
trickyRule:
(expression '=')=>...
So the lesson learned is: an empty group before a syntactic predicate is not the same as nothing at all.
Is it possible that a grammar parses a text differently with or without actions?
Yes, that is possible. org.antlr.tool.Strip leaves syntactic predicates1, but removes validating2- and gated3 semantic predicates (and member sections that these semantic predicates might use).
For example, the following rules would only match an A_TOKEN:
parser_rule1
: (parser_rule2)=> parser_rule2
;
parser_rule2
: {input.LT(1).getType() == A_TOKEN}? .
;
but if you use the Strip tool on it, it leaves the following:
parser_rule1
: (parser_rule2)=> parser_rule2
;
parser_rule2
: /*{input.LT(1).getType() == A_TOKEN}?*/ .
;
making it match any token.
In other words, Strip could change the behavior of the generated lexer or parser.
1 syntactic predicate: ( ... )=>
2 validating semantic predicate { ... }?
3 gated semantic predicate { ... }?=>