simple math expression parser - antlr

I have a simple math expression parser and I want to build the AST by myself (means no ast parser). But every node can just hold two operands. So a 2+3+4 will result in a tree like this:
+
/ \
2 +
/ \
3 4
The problem is, that I am not able to get my grammer doing the recursion, here ist just the "add" part:
add returns [Expression e]
: op1=multiply { $e = $op1.e; Print.ln($op1.text); }
( '+' op2=multiply { $e = new AddOperator($op1.e, $op2.e); Print.ln($op1.e.getClass(), $op1.text, "+", $op2.e.getClass(), $op2.text); }
| '-' op2=multiply { $e = null; } // new MinusOperator
)*
;
But at the end of the day this will produce a single tree like:
+
/ \
2 4
I know where the problem is, it is because a "add" can occour never or infinitly (*) but I do not know how to solve this. I thought of something like:
"add" part:
add returns [Expression e]
: op1=multiply { $e = $op1.e; Print.ln($op1.text); }
( '+' op2=(multiply|add) { $e = new AddOperator($op1.e, $op2.e); Print.ln($op1.e.getClass(), $op1.text, "+", $op2.e.getClass(), $op2.text); }
| '-' op2=multiply { $e = null; } // new MinusOperator
)?
;
But this will give me a recoursion error. Any ideas?

I don't have the full grammar to test this solution, but consider replacing this (from the first add rule in the question):
$e = new AddOperator($op1.e, $op2.e);
With this:
$e = new AddOperator($e, $op2.e); //$e instead of $op1.e
This way each iteration over ('+' multiply)* extends e rather than replaces it.
It may require a little playing around to get it right, or you may need a temporary Expression in the rule to keep things managed. Just make sure that the last expression created by the loop is somewhere on the right-hand side of the = operator, as in $e = new XYZ($e, $rhs.e);.

Related

how to tell which non-terminal is missing within an expansion in ANTLR

oC_RangeLiteral
: '*' SP? ( oC_IntegerLiteral SP? )? ( '..' SP? ( oC_IntegerLiteral SP? )? )? ;
Given a parser tree with ctx->oC_IntegerLiteral().size() == 1, How can I tell whether the first one is missing or the second one is missing?
Maybe the question title can be improved based on this concrete question.
You can label rule elements with name = like this:
oC_RangeLiteral
: '*' SP?
( first=oC_IntegerLiteral SP? )?
( '..' SP? ( second=oC_IntegerLiteral SP? )? )?
;
Not sure how to use them with the C++ runtime, but for, say, Java, that'd look like this:
if (ctx.first != null) {
// 'first' exists
}
if (ctx.second != null) {
// 'second' exists
}
EDIT
Is it possible to achieve similar without modifying the grammar?
Sure, but that makes it a lot messier. You'd need to figure out if there is a .. among the children of oC_RangeLiteral and then check if the oC_IntegerLiteral comes before or after this .. token. Something like this:
// Only need to check if 1 child is present: in case of 2 or 0 children, it is clear
if (ctx.oC_IntegerLiteral().size() == 1) {
int indexOfIntegerLiteral = ctx.children.indexOf(ctx.oC_IntegerLiteral().get(0));
OptionalInt indexOfDotDot = java.util.stream.IntStream
.range(0, ctx.children.size())
.filter(i -> ctx.children.get(i).getText().equals(".."))
.findFirst();
System.out.printf("indexOfIntegerLiteral=%d\n", indexOfIntegerLiteral);
System.out.printf("indexOfDotDot=%s\n", indexOfDotDot);
if (indexOfDotDot.isPresent() && indexOfIntegerLiteral > indexOfDotDot.getAsInt()) {
// If there is a ".." and the single `oC_IntegerLiteral` comes after it: it's the second one
}
else {
// otherwise, it's the first `oC_IntegerLiteral`
}
}
Another way is this:
Give your terminals own token names, say
STAR: '*';
DOTDOT: '..';
With that your rules becomes:
oC_RangeLiteral
: STAR SP? ( oC_IntegerLiteral SP? )? ( DOTDOT SP? ( oC_IntegerLiteral SP? )? )? ;
Now in your code you just check for DOTDOT:
if (ctx->oC_IntegerLiteral().size() == 1) {
if (ctx->DOTDOT()) {
// ctx->oC_IntegerLiteral(0) is the second integer literal
} else {
// ctx->oC_IntegerLiteral(0) is the first integer literal
}
}
As a side note: ctx->oC_IntegerLiteral().size() can be 0, because the it's optional in both cases where it appears in the rule. So, just testing the existence of DOTDOT does not tell you there's at least one integer literal.

How to preserve whitespace when we use text attribute in Antlr4

I want to keep white space when I call text attribute of token, is there any way to do it?
Here is the situation:
We have the following code
IF L > 40 THEN;
ELSE
IF A = 20 THEN
PUT "HELLO";
In this case, I want to transform it into:
if (!(L>40){
if (A=20)
put "hello";
}
The rule in Antlr is that:
stmt_if_block: IF expr
THEN x=stmt
(ELSE y=stmt)?
{
if ($x.text.equalsIgnoreCase(";"))
{
WriteLn("if(!(" + $expr.text +")){");
WriteLn($stmt.text);
Writeln("}");
}
}
But the result looks like:
if(!(L>40))
{
ifA=20put"hello";
}
The reason is that the white space in $stmt was removed. I was wondering if there is anyway to keep these white space
Thank you so much
Update: If I add
SPACE: [ ] -> channel(HIDDEN);
The space will be preserved, and the result would look like below, many spaces between tokens:
IF SUBSTR(WNAME3,M-1,1) = ')' THEN M = L; ELSE M = L - 1;
This is the C# extension method I use for exactly this purpose:
public static string GetFullText(this ParserRuleContext context)
{
if (context.Start == null || context.Stop == null || context.Start.StartIndex < 0 || context.Stop.StopIndex < 0)
return context.GetText(); // Fallback
return context.Start.InputStream.GetText(Interval.Of(context.Start.StartIndex, context.Stop.StopIndex));
}
Since you're using java, you'll have to translate it, but it should be straightforward - the API is the same.
Explanation: Get the first token, get the last token, and get the text from the input stream between the first char of the first token and the last char of the last token.
#Lucas solution, but in java in case you have troubles in translating:
private String getFullText(ParserRuleContext context) {
if (context.start == null || context.stop == null || context.start.getStartIndex() < 0 || context.stop.getStopIndex() < 0)
return context.getText();
return context.start.getInputStream().getText(Interval.of(context.start.getStartIndex(), context.stop.getStopIndex()));
}
Looks like InputStream is not always updated after removeLastChild/addChild operations. This solution helped me for one grammar, but it doesn't work for another.
Works for this grammar.
Doesn't work for modern groovy grammar (for some reason inputStream.getText contains old text).
I am trying to implement function name replacement like this:
enterPostfixExpression(ctx: PostfixExpressionContext) {
// Get identifierContext from ctx
...
const token = CommonTokenFactory.DEFAULT.createSimple(GroovyParser.Identifier, 'someNewFnName');
const node = new TerminalNode(token);
identifierContext.removeLastChild();
identifierContext.addChild(node);
UPD: I used visitor pattern for the first implementation

Null Pointer Exception - ANTLR TreeWalker

I keep getting a NullPoiterException in my TreeWalker but I can't seem to find out why.
I can't post the whole grammar, cause it's far too long.
This is the rule in the treeWalker where antlrWorks says the problem is:
collection_name returns [MyType value]
: ID { $value = (MyType) database.get($collection_name.text); }
;
Note that database is a HashMap.
Thank you!
I can't post the whole grammar, cause it's far too long.
The following is more "readable" and does exactly the same as your original rule:
collection_name returns [MyType value]
: ID { $value = (MyType) database.get($ID.text); }
;
Perhaps do some sanity checks:
collection_name returns [MyType value]
: ID
{
Object v = database.get($ID.text);
if(v == null) {
throw new RuntimeException($ID.text + " unknown in database!");
}
$value = (MyType) v;
}
;
EDIT
As you already found out, accessing the .text attribute of a rule is not possible in a tree grammar (only in a parser grammar). In tree grammars, every rule is of type Tree and knows a .start and .end attributes instead. Tokens can be accessed the same in both parser- and tree-grammars. So $ID.text works okay.

Is there way to detect if an optional (? operator) tree grammar rule executed in an action?

path[Scope sc] returns [Path p]
#init{
List<String> parts = new ArrayList<String>();
}
: ^(PATH (id=IDENT{parts.add($id.text);})+ pathIndex? )
{// ACTION CODE
// need to check if pathIndex has executed before running this code.
if ($pathIndex.index >=0 ){
p = new Path($sc, parts, $pathIndex.index);
}else if($pathIndex.pathKey != ""){
p = new Path($sc, parts, $pathIndex.pathKey);
}
;
Is there a way to detect if pathIndex was executed? In my action code, I tried testing $pathIndex == null, but ANTLR doesn't let you do that. ANTLRWorks gives a syntax error which saying "Missing attribute access on rule scope: pathIndex."
The reason why I need to do this is because in my action code I do:
$pathIndex.index
which returns 0 if the variable $pathIndex is translated to is null. When you are accessing an attribute, ANTLR generates pathIndex7!=null?pathIndex7.index:0 This causes a problem with an object because it changes a value I have preset to -1 as an error flag to 0.
There are a couple of options:
1
Put your code inside the optional pathIndex:
rule
: ^(PATH (id=IDENT{parts.add($id.text);})+ (pathIndex {/*pathIndex cannot be null here!*/} )? )
;
2
Use a boolean flag to denote the presence (or absence) of pathIndex:
rule
#init{boolean flag = false;}
: ^(PATH (id=IDENT{parts.add($id.text);})+ (pathIndex {flag = true;} )? )
{
if(flag) {
// ...
}
}
;
EDIT
You could also make pathIndex match nothing so that you don't need to make it optional inside path:
path[Scope sc] returns [Path p]
: ^(PATH (id=IDENT{parts.add($id.text);})+ pathIndex)
{
// code
}
;
pathIndex returns [int index, String pathKey]
#init {
$index = -1;
$pathKey = "";
}
: ( /* some rules here */ )?
;
PS. Realize that the expression $pathIndex.pathKey != "" will most likely evaluate to false. To compare the contents of strings in Java, use their equals(...) method instead:
!$pathIndex.pathKey.equals("")
or if $pathIndex.pathKey can be null, you can circumvent a NPE by doing:
!"".equals($pathIndex.pathKey)
More information would have been helpful. However, if I understand correctly, when a value for the index is not present in the input you want to test for $pathIndex.index == null. This code does that using the pathIndex rule to return the Integer $index to the path rule:
path
: ^(PATH IDENT+ pathIndex?)
{ if ($pathIndex.index == null)
System.out.println("path index is null");
else
System.out.println("path index = " + $pathIndex.index); }
;
pathIndex returns [Integer index]
: DIGIT
{ $index = Integer.parseInt($DIGIT.getText()); }
;
For testing, I created these simple parser and lexer rules:
path : 'path' IDENT+ pathIndex? -> ^(PATH IDENT+ pathIndex?)
;
pathIndex : DIGIT
;
/** lexer rules **/
DIGIT : '0'..'9' ;
IDENT : LETTER+ ;
fragment LETTER : ('a'..'z' | 'A'..'Z') ;
When the index is present in the input, as in path a b c 5, the output is:
Tree = (PATH a b c 5)
path index = 5
When the index is not present in the input, as in path a b c, the output is:
Tree = (PATH a b c)
path index is null

How can I build an ANTLR Works style parse tree?

I've read that you need to use the '^' and '!' operators in order to build a parse tree similar to the ones displayed in ANTLR Works (even though you don't need to use them to get a nice tree in ANTLR Works). My question then is how can I build such a tree? I've seen a few pages on tree construction using the two operators and rewrites, and yet say I have an input string abc abc123 and a grammar:
grammar test;
program : idList;
idList : id* ;
id : ID ;
ID : LETTER (LETTER | NUMBER)* ;
LETTER : 'a' .. 'z' | 'A' .. 'Z' ;
NUMBER : '0' .. '9' ;
ANTLR Works will output:
What I dont understand is how you can get the 'idList' node on top of this tree (as well as the grammar one as a matter of fact). How can I reproduce this tree using rewrites and those operators?
What I dont understand is how you can get the 'idList' node on top of this tree (as well as the grammar one as a matter of fact). How can I reproduce this tree using rewrites and those operators?
You can't use ^ and ! alone. These operators only operate on existing tokens, while you want to create extra tokens (and make these the root of your sub trees). You can do that using rewrite rules and defining some imaginary tokens.
A quick demo:
grammar test;
options {
output=AST;
ASTLabelType=CommonTree;
}
tokens {
IdList;
Id;
}
#parser::members {
private static void walk(CommonTree tree, int indent) {
if(tree == null) return;
for(int i = 0; i < indent; i++, System.out.print(" "));
System.out.println(tree.getText());
for(int i = 0; i < tree.getChildCount(); i++) {
walk((CommonTree)tree.getChild(i), indent + 1);
}
}
public static void main(String[] args) throws Exception {
testLexer lexer = new testLexer(new ANTLRStringStream("abc abc123"));
testParser parser = new testParser(new CommonTokenStream(lexer));
walk((CommonTree)parser.program().getTree(), 0);
}
}
program : idList EOF -> idList;
idList : id* -> ^(IdList id*);
id : ID -> ^(Id ID);
ID : LETTER (LETTER | DIGIT)*;
SPACE : ' ' {skip();};
fragment LETTER : 'a' .. 'z' | 'A' .. 'Z';
fragment DIGIT : '0' .. '9';
If you run the demo above, you will see the following being printed to the console:
IdList
Id
abc
Id
abc123
As you can see, imaginary tokens must also start with an upper case letter, just like lexer rules. If you want to give the imaginary tokens the same text as the parser rule they represent, do something like this instead:
idList : id* -> ^(IdList["idList"] id*);
id : ID -> ^(Id["id"] ID);
which will print:
idList
id
abc
id
abc123