Generating AST from ANTLR Grammar: arrays inside structs - antlr

I'm trying to parser and generate an AST from an ANTLR grammar. I'm finding some problems when I try and parse arrays inside structs and array of structs.
This is a declaration example:
TYPE MY_ARRAY :
ARRAY [ 0..2 ] OF INT;
END_TYPE
TYPE est :
STRUCT
c1 : INT;
c : MY_ARRAY;
END_STRUCT;
END_TYPE
TYPE MSA :
ARRAY [ 0..2 ] OF est;
END_TYPE
VAR
MA : MY_ARRAY;
STR : est;
STR2 : MSA;
END_VAR
I have no problems with declarations. I'm not able to write a grammar to parse expressions as next:
STR.c[1]
STR2[2].c[1]
Next code shows an extract of my ANTLR grammar:
operand
: variable_simbolic
| DIRECT_VAR<Localization>
| CTE_INT<ConstantINT>
| CTE_BOOL<ConstantBOOL>
| CTE_REAL<ConstantREAL>
;
variable_simbolic
: (ID -> ID<Identificador>) ( (('[' operand (',' operand)* ']') -> ^(ARRAY_ACCESS<ArrayAccess> ID<Identificador> operand+))
| (('.' operand ) -> ^(FIELD_ACCESS<FieldAccess> ID<Identificador> operand))
| (('#' operand ) -> ^(ENUM_ACCESS<EnumAccess> ID<Identificador> operand))
)?
;
This grammar allows me to parse expressions like STR.c1 or MA[1], but it doesn't parse expressions like STR.c[1]. If an array access is defined, it has to be the parent expression. I hope next diagram helps:
Is there any way to modify my grammar to accept that kind of expressions? Thank you in advance.

Something like this perhaps:
operand
: variable_simbolic
| CTE_INT
;
variable_simbolic
: (ID -> ID) (variable_simbolic_tail -> ^(ID variable_simbolic_tail))?
;
variable_simbolic_tail
: array variable_simbolic_tail? -> ^(ARRAY_ACCESS array variable_simbolic_tail?)
| '.' ID variable_simbolic_tail? -> ^(FIELD_ACCESS ID variable_simbolic_tail?)
| '#' ID variable_simbolic_tail? -> ^(ENUM_ACCESS ID variable_simbolic_tail?)
;
array
: '[' operand (',' operand)* ']' -> ^(ARRAY_OPS operand+)
;
?
EDIT
I propose a slightly different way. Instead of trying to construct a complex AST with many different levels, simply create a single LOOKUP AST and let the tails be added as child nodes to this ast.
An small example grammar:
grammar T;
options {
output=AST;
}
tokens {
LOOKUP;
ARRAY_ACCESS;
FIELD_ACCESS;
ENUM_ACCESS;
}
parse
: operand EOF
;
operand
: variable_simbolic
| CTE_INT
;
variable_simbolic
: ID variable_simbolic_tail* -> ^(LOOKUP ID variable_simbolic_tail*)
;
variable_simbolic_tail
: '[' operand (',' operand)* ']' -> ^(ARRAY_ACCESS operand+)
| '.' ID -> ^(FIELD_ACCESS ID)
| '#' ID -> ^(ENUM_ACCESS ID)
;
CTE_INT : '0'..'9'+;
ID : ('a'..'z' | 'A'..'Z') ('a'..'z' | 'A'..'Z' |'0'..'9')*;
SPACE : (' ' | '\t' | '\r' | '\n')+ {skip();};
If you parse the input: STR2[2].c[1], you will get the following AST:
which is easily evaluated by walking the children from left to right.

Related

antlr4 grammar errors when parsing

I have the following grammar:
grammar Token;
prog: (expr NL?)+ EOF;
expr: '[' type ']';
type : typeid ':' value;
typeid : 'TXT' | 'ENC' | 'USR';
value: Text | INT;
INT : '0' | [1-9] [0-9]*;
//WS : [ \t]+;
WS : [ \t\n\r]+ -> skip ;
NL: '\r'? '\n';
Text : ~[\]\[\n\r"]+ ;
and the text I need to parse is something like this below
[TXT:look at me!]
[USR:19700]
[TXT:, can I go there?]
[ENC:124124]
[TXT:this is needed for you to go...]
I need to split this text but I getting some errors when I run grun.bat Token prog -gui -trace -diagnostics
enter prog, LT(1)=[
enter expr, LT(1)=[
consume [#0,0:0='[',<3>,1:0] rule expr
enter type, LT(1)=TXT:look at me!
enter typeid, LT(1)=TXT:look at me!
line 1:1 mismatched input 'TXT:look at me!' expecting {'TXT', 'ENC', 'USR'}
... much more ...
what is wrong with my grammar? please, help me!
You must understand that the tokens are not created based on what the parser is trying to match. The lexer tries to match as much characters as possible (independently from that parser!): your Text token should be defined differently.
You could let the Text rule become a parser rule instead, and match single char tokens like this:
grammar Token;
prog : expr+ EOF;
expr : '[' type ']';
type : typeid ':' value;
typeid : 'TXT' | 'ENC' | 'USR';
value : text | INT;
text : CHAR+;
INT : '0' | [1-9] [0-9]*;
WS : [ \t\n\r]+ -> skip ;
CHAR : ~[\[\]\r\n];

Precedence in Antlr using parentheses

We are developing a DSL, and we're facing some problems:
Problem 1:
In our DSL, it's allowed to do this:
A + B + C
but not this:
A + B - C
If the user needs to use two or more different operators, he'll need to insert parentheses:
A + (B - C) or (A + B) - C.
Problem 2:
In our DSL, the most precedent operator must be surrounded by parentheses.
For example, instead of using this way:
A + B * C
The user needs to use this:
A + (B * C)
To solve the Problem 1 I've got a snippet of ANTLR that worked, but I'm not sure if it's the best way to solve it:
sumExpr
#init {boolean isSum=false;boolean isSub=false;}
: {isSum(input.LT(2).getText()) && !isSub}? multExpr('+'^{isSum=true;} sumExpr)+
| {isSub(input.LT(2).getText()) && !isSum}? multExpr('-'^{isSub=true;} sumExpr)+
| multExpr;
To solve the Problem 2, I have no idea where to start.
I appreciate your help to find out a better solution to the first problem and a direction to solve the seconde one. (Sorry for my bad english)
Below is the grammar that we have developed:
grammar TclGrammar;
options {
output=AST;
ASTLabelType=CommonTree;
}
#members {
public boolean isSum(String type) {
System.out.println("Tipo: " + type);
return "+".equals(type);
}
public boolean isSub(String type) {
System.out.println("Tipo: " + type);
return "-".equals(type);
}
}
prog
: exprMain ';' {System.out.println(
$exprMain.tree == null ? "null" : $exprMain.tree.toStringTree());}
;
exprMain
: exprQuando? (exprDeveSatis | exprDeveFalharCaso)
;
exprDeveSatis
: 'DEVE SATISFAZER' '{'! expr '}'!
;
exprDeveFalharCaso
: 'DEVE FALHAR CASO' '{'! expr '}'!
;
exprQuando
: 'QUANDO' '{'! expr '}'!
;
expr
: logicExpr
;
logicExpr
: boolExpr (('E'|'OU')^ boolExpr)*
;
boolExpr
: comparatorExpr
| emExpr
| 'VERDADE'
| 'FALSO'
;
emExpr
: FIELD 'EM' '[' (variable_lista | field_lista) comCruzamentoExpr? ']'
-> ^('EM' FIELD (variable_lista+)? (field_lista+)? comCruzamentoExpr?)
;
comCruzamentoExpr
: 'COM CRUZAMENTO' '(' FIELD ';' FIELD (';' FIELD)* ')' -> ^('COM CRUZAMENTO' FIELD+)
;
comparatorExpr
: sumExpr (('<'^|'<='^|'>'^|'>='^|'='^|'<>'^) sumExpr)?
| naoPreenchidoExpr
| preenchidoExpr
;
naoPreenchidoExpr
: FIELD 'NAO PREENCHIDO' -> ^('NAO PREENCHIDO' FIELD)
;
preenchidoExpr
: FIELD 'PREENCHIDO' -> ^('PREENCHIDO' FIELD)
;
sumExpr
#init {boolean isSum=false;boolean isSub=false;}
: {isSum(input.LT(2).getText()) && !isSub}? multExpr('+'^{isSum=true;} sumExpr)+
| {isSub(input.LT(2).getText()) && !isSum}? multExpr('-'^{isSub=true;} sumExpr)+
| multExpr
;
multExpr
: funcExpr(('*'^|'/'^) funcExpr)?
;
funcExpr
: 'QUANTIDADE'^ '('! FIELD ')'!
| 'EXTRAI_TEXTO'^ '('! FIELD ';' INTEGER ';' INTEGER ')'!
| cruzaExpr
| 'COMBINACAO_UNICA' '(' FIELD ';' FIELD (';' FIELD)* ')' -> ^('COMBINACAO_UNICA' FIELD+)
| 'EXISTE'^ '('! FIELD ')'!
| 'UNICO'^ '('! FIELD ')'!
| atom
;
cruzaExpr
: operadorCruzaExpr ('CRUZA COM'^|'CRUZA AMBOS'^) operadorCruzaExpr ondeExpr?
;
operadorCruzaExpr
: FIELD('('!field_lista')'!)?
;
ondeExpr
: ('ONDE'^ '('!expr')'!)
;
atom
: FIELD
| VARIABLE
| '('! expr ')'!
;
field_lista
: FIELD(';' field_lista)?
;
variable_lista
: VARIABLE(';' variable_lista)?
;
FIELD
: NONCONTROL_CHAR+
;
NUMBER
: INTEGER | FLOAT
;
VARIABLE
: '\'' NONCONTROL_CHAR+ '\''
;
fragment SIGN: '+' | '-';
fragment NONCONTROL_CHAR: LETTER | DIGIT | SYMBOL;
fragment LETTER: LOWER | UPPER;
fragment LOWER: 'a'..'z';
fragment UPPER: 'A'..'Z';
fragment DIGIT: '0'..'9';
fragment SYMBOL: '_' | '.' | ',';
fragment FLOAT: INTEGER '.' '0'..'9'*;
fragment INTEGER: '0' | SIGN? '1'..'9' '0'..'9'*;
WS : ( ' '
| '\t'
| '\r'
| '\n'
) {skip();}
;
This is similar to not having operator precedence at all.
expr
: funcExpr
( ('+' funcExpr)*
| ('-' funcExpr)*
| ('*' funcExpr)*
| ('/' funcExpr)*
)
;
I think the following should work. I'm assuming some lexer tokens with obvious names.
expr: sumExpr;
sumExpr: onlySum | subExpr;
onlySum: atom ( PLUS onlySum )?;
subExpr: onlySub | multExpr;
onlySub: atom ( MINUS onlySub )? ;
multExpr: atom ( STAR atomic )? ;
parenExpr: OPEN_PAREN expr CLOSE_PAREN;
atom: FIELD | VARIABLE | parenExpr
The only* rules match an expression if it only has one type of operator outside of parentheses. The *Expr rules match either a line with the appropriate type of operators or go to the next operator.
If you have multiple types of operators, then they are forced to be inside parentheses because the match will go through atom.

Solving antlr left recursion

I'm trying to parse a language using ANTLR which can contain the following syntax:
someVariable, somVariable.someMember, functionCall(param).someMember, foo.bar.baz(bjork).buffalo().xyzzy
This is the ANTLR grammar which i've come up with so far, and the access_operation throws the error
The following sets of rules are mutually left-recursive [access_operation, expression]:
grammar Test;
options {
output=AST;
ASTLabelType=CommonTree;
}
tokens {
LHS;
RHS;
CALL;
PARAMS;
}
start
: body? EOF
;
body
: expression (',' expression)*
;
expression
: function -> ^(CALL)
| access_operation
| atom
;
access_operation
: (expression -> ^(LHS)) '.'! (expression -> ^(RHS))
;
function
: (IDENT '(' body? ')') -> ^(IDENT PARAMS?)
;
atom
: IDENT
| NUMBER
;
fragment LETTER : ('a'..'z' | 'A'..'Z');
fragment DIGIT : '0'..'9';
IDENT : (LETTER)+ ;
NUMBER : (DIGIT)+ ;
SPACE : (' ' | '\t' | '\r' | '\n') { $channel=HIDDEN; };
What i could manage so far was to refactor the access_operation rule to '.' expression which generates an AST where the access_operation node only contains the right side of the operation.
What i'm looking for instead is something like this:
How can the left-recursion problem solved in this case?
By "wrong AST" I'll make a semi educated guess that, for input like "foo.bar.baz", you get an AST where foo is the root with bar as a child who in its turn has baz as a child, which is a leaf in the AST. You may want to have this reversed. But I'd not go for such an AST if I were you: I'd keep the AST as flat as possible:
foo
/ | \
/ | \
bar baz ...
That way, evaluating is far easier: you simply look up foo, and then walk from left to right through its children.
A quick demo:
grammar Test;
options {
output=AST;
ASTLabelType=CommonTree;
}
tokens {
BODY;
ACCESS;
CALL;
PARAMS;
}
start
: body EOF -> body
;
body
: expression (',' expression)* -> ^(BODY expression+)
;
expression
: atom
;
atom
: NUMBER
| (IDENT -> IDENT) ( tail -> ^(IDENT tail)
| call tail? -> ^(CALL IDENT call tail?)
)?
;
tail
: (access)+
;
access
: ('.' IDENT -> ^(ACCESS IDENT)) (call -> ^(CALL IDENT call))?
;
call
: '(' (expression (',' expression)*)? ')' -> ^(PARAMS expression*)
;
IDENT : LETTER+;
NUMBER : DIGIT+;
SPACE : (' ' | '\t' | '\r' | '\n') {$channel=HIDDEN;};
fragment LETTER : ('a'..'z' | 'A'..'Z');
fragment DIGIT : '0'..'9';
which can be tested with:
import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;
public class Main {
public static void main(String[] args) throws Exception {
String src = "someVariable, somVariable.someMember, functionCall(param).someMember, " +
"foo.bar.baz(bjork).buffalo().xyzzy";
TestLexer lexer = new TestLexer(new ANTLRStringStream(src));
TestParser parser = new TestParser(new CommonTokenStream(lexer));
CommonTree tree = (CommonTree)parser.start().getTree();
DOTTreeGenerator gen = new DOTTreeGenerator();
StringTemplate st = gen.toDOT(tree);
System.out.println(st);
}
}
The output of Main corresponds to the following AST:
EDIT
And since you indicated your ultimate goal is not evaluating the input, but that you rather need to conform the structure of the AST to some 3rd party API, here's a grammar that will create an AST like you indicated in your edited question:
grammar Test;
options {
output=AST;
ASTLabelType=CommonTree;
}
tokens {
BODY;
ACCESS_OP;
CALL;
PARAMS;
LHS;
RHS;
}
start
: body EOF -> body
;
body
: expression (',' expression)* -> ^(BODY expression+)
;
expression
: atom
;
atom
: NUMBER
| (ID -> ID) ( ('(' params ')' -> ^(CALL ID params))
('.' expression -> ^(ACCESS_OP ^(LHS ^(CALL ID params)) ^(RHS expression)))?
| '.' expression -> ^(ACCESS_OP ^(LHS ID) ^(RHS expression))
)?
;
params
: (expression (',' expression)*)? -> ^(PARAMS expression*)
;
ID : LETTER+;
NUMBER : DIGIT+;
SPACE : (' ' | '\t' | '\r' | '\n') {$channel=HIDDEN;};
fragment LETTER : ('a'..'z' | 'A'..'Z');
fragment DIGIT : '0'..'9';
which creates the following AST if you run the Main class:
The atom rule may be a bit daunting, but you can't shorten it much since the left ID needs to be available to most of the alternatives. ANTLRWorks helps in visualizing the alternative paths this rule may take:
which means atom can be any of the 5 following alternatives (with their corresponding AST's):
+----------------------+--------------------------------------------------------+
| alternative | generated AST |
+----------------------+--------------------------------------------------------+
| NUMBER | NUMBER |
| ID | ID |
| ID params | ^(CALL ID params) |
| ID params expression | ^(ACCESS_OP ^(LHS ^(CALL ID params)) ^(RHS expression))|
| ID expression | ^(ACCESS_OP ^(LHS ID) ^(RHS expression) |
+----------------------+--------------------------------------------------------+

ANTLR Template translator match part of grammar

I wrote a grammar for a language and now I want to treat some syntactic sugar constructions, for that I was thinking of writing a template translator.
The problem is I want my template grammar to translate only some constructions of the language and leave the rest as it is.
For example:
I have this as input:
class Main {
int a[10];
}
and I want to translate that into something like:
class Main {
Array a = new Array(10);
}
Ideally I would like to do some think like this in ANTLR
grammer Translator
options { output=template;}
decl
: TYPE ID '[' INT ']' -> template(name = {$ID.text}, size ={$INT.text})
"Array <name> = new Array(<size>);
I would like it to leave the rest of the input that doesn't match rule decl as it is.
How can I achieve this in ANTLR without writing the full grammar for the language ?
I would simply handle such things in the parser grammar.
Assuming you're constructing an AST in your parser grammar, I guess you'll have a rule to parse input like Array a = new Array(10); similar to:
decl
: TYPE ID '=' expr ';' -> ^(DECL TYPE ID expr)
;
where expr eventually matches a term like this:
term
: NUMBER
| 'new' ID '(' (expr (',' expr)*)? ')' -> ^('new' ID expr*)
| ...
;
To account for your short-hand declaration int a[10];, all you have to do is expand decl like this:
decl
: TYPE ID '=' expr ';' -> ^(DECL TYPE ID expr)
| TYPE ID '[' expr ']' ';' -> ^(DECL 'Array' ID ^(NEW ARRAY expr))
;
which will rewrite the input int a[10]; into the following AST:
which is exactly the same as the AST created for input Array a = new Array(10);.
EDIT
Here's a small working demo:
grammar T;
options {
output=AST;
}
tokens {
ROOT;
DECL;
NEW='new';
INT='int';
ARRAY='Array';
}
parse
: decl+ EOF -> ^(ROOT decl+)
;
decl
: type ID '=' expr ';' -> ^(DECL type ID expr)
| type ID '[' expr ']' ';' -> ^(DECL ARRAY ID ^(NEW ARRAY expr))
;
expr
: Number
| NEW type '(' (expr (',' expr)*)? ')' -> ^(NEW ID expr*)
;
type
: INT
| ARRAY
| ID
;
ID : ('a'..'z' | 'A'..'Z')+;
Number : '0'..'9'+;
Space : (' ' | '\t' | '\r' | '\n') {skip();};
which can be tested with the class:
import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;
public class Main {
public static void main(String[] args) throws Exception {
String src = "Array a = new Array(10); int a[10];";
TLexer lexer = new TLexer(new ANTLRStringStream(src));
TParser parser = new TParser(new CommonTokenStream(lexer));
CommonTree tree = (CommonTree)parser.parse().getTree();
DOTTreeGenerator gen = new DOTTreeGenerator();
StringTemplate st = gen.toDOT(tree);
System.out.println(st);
}
}

ANTLR expression interpreter

I have created the following grammar: I would like some idea how to build an interpreter that returns a tree in java, which I can later use for printing in the screen, Im bit stack on how to start on it.
grammar myDSL;
options {
language = Java;
}
#header {
package DSL;
}
#lexer::header {
package DSL;
}
program
: IDENT '={' components* '}'
;
components
: IDENT '=('(shape)(shape|connectors)* ')'
;
shape
: 'Box' '(' (INTEGER ','?)* ')'
| 'Cylinder' '(' (INTEGER ','?)* ')'
| 'Sphere' '(' (INTEGER ','?)* ')'
;
connectors
: type '(' (INTEGER ','?)* ')'
;
type
: 'MG'
| 'EL'
;
IDENT: ('a'..'z' | 'A'..'Z')('a'..'z' | 'A'..'Z' | '0'..'0')*;
INTEGER: '0'..'9'+;
// This if for the empty spaces between tokens and avoids them in the parser
WS: (' ' | '\t' | '\n' | '\r' | '\f')+ {$channel=HIDDEN;};
COMMENT: '//' .* ('\n' | '\r') {$channel=HIDDEN;};
A couple of remarks:
There's no need to set the language for Java, which is the default target language. So you can remove this:
options {
language = Java;
}
Your IDENT contains an error:
IDENT: ('a'..'z' | 'A'..'Z')('a'..'z' | 'A'..'Z' | '0'..'0')*;
the '0'..'0') should most probably be '0'..'9').
The sub rule (INTEGER ','?)* also matches source like 1 2 3 4 (no comma's at all!). Perhaps you meant to do: (INTEGER (',' INTEGER)*)?
Now, as to your question: how to let ANTLR construct a proper AST? This can be done by adding output = AST; in your options block:
options {
//language = Java;
output = AST;
}
And then either adding the "tree operators" ^ and ! in your parser rules, or by using tree rewrite rules: rule: a b c -> ^(c b a).
The "tree operator" ^ is used to define the root of the (sub) tree and ! is used to exclude a token from the (sub) tree.
Rewrite rules have ^( /* tokens here */ ) where the first token (right after ^() is the root of the (sub) tree, and all following tokens are child nodes of the root.
An example might be in order. Let's take your first rule:
program
: IDENT '={' components* '}'
;
and you want to let IDENT be the root, components* the children and you want to exclude ={ and } from the tree. You can do that by doing:
program
: IDENT^ '={'! components* '}'!
;
or by doing:
program
: IDENT '={' components* '}' -> ^(IDENT components*)
;