ANTLR not matching unicode escaped character - antlr

I'm writing a parser/interpreter for a C-like language and I need to interpret escaped characters. One of them is the unicode-escaped sequence with this pattern "\uXXXX" where X is some hex number.
My ANTLR rules look like this:
public char returns [char c]
: '\\"' { $c = '"'; }
| '\\\\' { $c = '\\'; }
| '\\/' { $c = '/'; }
| '\\b' { $c = '\b'; }
| '\\f' { $c = '\f'; }
| '\\n' { $c = '\n'; }
| '\\r' { $c = '\r'; }
| '\\t' { $c = '\t'; }
| '\\u' HEXDIGIT HEXDIGIT HEXDIGIT HEXDIGIT { $c = 'e'; }
| ~('\\' | '"') { $c = '/'; }
;
fragment HEXDIGIT
: ('0'..'9'|'a'..'f'|'A'..'F')
I'm feeding it this string "\u1234" for which I expect an 'e' but I'm getting a '/' instead which is the fallback rule for everything else.
Is there some magic juju going on with fragments and rules or something that I'm not aware of?

As mentioned by Adam, char is a parser rule at the moment, but should be made a lexer rule instead, in which case you can't let it return a char (lexer rules always return an instance of a Token!).
You can adjust the inner-text of a token using its setText(...) method like this (assuming Java is the target language):
// lexer rules start with a capital!
Char
: '\\"' { setText("\""); }
| '\\\\' { setText("\\"); }
| '\\/' { setText("/"); }
| '\\b' { setText("\b"); }
| '\\f' { setText("\f"); }
| '\\n' { setText("\n"); }
| '\\r' { setText("\r"); }
| '\\t' { setText("\t"); }
| '\\u' HEXDIGIT HEXDIGIT HEXDIGIT HEXDIGIT
{
String hex = getText();
int i = Integer.parseInt(hex.substring(2), 16);
setText(hex + " base 10 = " + i);
}
| ~('\\' | '"')
;
fragment HEXDIGIT
: ('0'..'9'|'a'..'f'|'A'..'F')
;

Related

ANTLR doesnt complain about unclosed comment

I'm trying to define a comment rule that allows == bla bla bla == but not == on its own.
FYI, originally, I had added F_COMMENT!? between everything and still was getting the same problem. I've left one F_COMMENT!? in the expression rule in some vain attempt at getting it working.
When I debug == in ANTLRworks (1.5.2) it just ignores the == and returns the EOF token.
Heres some of the grammar...
expression
: F_COMMENT!? condition? EOF;
WS
: ( '\t' | ' ' | '\r' | '\n' | F_COMMENT)+ { $channel = HIDDEN; } ;
F_COMMENT
: '==' ( options {greedy=false;} : . )* '==';
UPDATE:
I've created a concise grammar for this problem which seems to be working, for the == case at least...
#members {
#Override
public void displayRecognitionError(String[] tokenNames, RecognitionException e) {
String hdr = getErrorHeader(e);
String msg = getErrorMessage(e, tokenNames);
throw new RuntimeException(hdr + ":" + msg);
}
}
expression
: condition? EOF;
condition
: (F_COMMENT!)* cnd_word;
cnd_word
: ( CND_WORD );
WS
: ( '\t' | ' ' | '\r' | '\n' )+ { $channel = HIDDEN; } ;
CND_WORD
: ('=' | '*'? (F_QUOTEDWORD+ | F_WORDCHARS+) '*'?) | '*';
fragment
F_COMMENT
: '==' ~('=')* '==';
fragment
F_QUOTEDWORD
: '"' ( ~('\\'|'"') | ('\\' '"') )* '"';
fragment
F_WORDCHARS
: ('a'..'z'|'A'..'Z'|'0'..'9')+;

ANTLR v3 Treewalker class. How to evaluate right associative function such as Factorial

I'm trying to build an expression evaluator with ANTLR v3 but I can't get the factorial function because it is right associative.
This is the code:
class ExpressionParser extends Parser;
options { buildAST=true; }
imaginaryTokenDefinitions :
SIGN_MINUS
SIGN_PLUS;
expr : LPAREN^ sumExpr RPAREN! ;
sumExpr : prodExpr ((PLUS^|MINUS^) prodExpr)* ;
prodExpr : powExpr ((MUL^|DIV^|MOD^) powExpr)* ;
powExpr : runary (POW^ runary)? ;
runary : unary (FAT)?;
unary : (SIN^|COS^|TAN^|LOG^|LN^|RAD^)* signExpr;
signExpr : (
m:MINUS^ {#m.setType(SIGN_MINUS);}
| p:PLUS^ {#p.setType(SIGN_PLUS);}
)? atom ;
atom : NUMBER | expr ;
class ExpressionLexer extends Lexer;
PLUS : '+' ;
MINUS : '-' ;
MUL : '*' ;
DIV : '/' ;
MOD : '%' ;
POW : '^' ;
SIN : 's' ;
COS : 'c' ;
TAN : 't' ;
LOG : 'l' ;
LN : 'n' ;
RAD : 'r' ;
FAT : 'f' ;
LPAREN: '(' ;
RPAREN: ')' ;
SEMI : ';' ;
protected DIGIT : '0'..'9' ;
NUMBER : (DIGIT)+ ('.' (DIGIT)+)?;
{import java.lang.Math;}
class ExpressionTreeWalker extends TreeParser;
expr returns [double r]
{ double a,b; int i,f=1; r=0; }
: #(PLUS a=expr b=expr) { r=a+b; }
| #(MINUS a=expr b=expr) { r=a-b; }
| #(MUL a=expr b=expr) { r=a*b; }
| #(DIV a=expr b=expr) { r=a/b; }
| #(MOD a=expr b=expr) { r=a%b; }
| #(POW a=expr b=expr) { r=Math.pow(a,b); }
| #(SIN a=expr ) { r=Math.sin(a); }
| #(COS a=expr ) { r=Math.cos(a); }
| #(TAN a=expr ) { r=Math.tan(a); }
| #(LOG a=expr ) { r=Math.log10(a); }
| #(LN a=expr ) { r=Math.log(a); }
| #(RAD a=expr ) { r=Math.sqrt(a); }
| #(FAT a=expr ) { for(i=1; i<=a; i++){f=f*i;}; r=(double)f;}
| #(LPAREN a=expr) { r=a; }
| #(SIGN_MINUS a=expr) { r=-1*a; }
| #(SIGN_PLUS a=expr) { if(a<0)r=0-a; else r=a; }
| d:NUMBER { r=Double.parseDouble(d.getText()); } ;
if I change FAT matching case in class TreeWalker with something like this:
| #(a=expr FAT ) { for(i=1; i<=a; i++){f=f*i;}; r=(double)f;}
I get this errors:
Expression.g:56:7: rule classDef trapped:
Expression.g:56:7: unexpected token: a
error: aborting grammar 'ExpressionTreeWalker' due to errors
Exiting due to errors.
Your tree walker (the original one) is fine, as far as I can see.
However, you probably need to mark FAT in the grammar:
runary : unary (FAT^)?;
(Note the hat ^, as in all the other productions.)
Edit:
As explained in the Antlr3 wiki, the hat operator is needed to make the node the "root of subtree created for entire enclosing rule even if nested in a subrule". In this case, the ! operator is nested in a conditional subrule ((FAT)?). That's independent of whether the operator is prefix or postfix.
Note that in your grammar the ! operator is not right-associative since a!! is not valid at all. But I would say that associativity is only meaningful for infix operators.

Antlr grammar with EBNF

I'm trying to generate a Lexer/Parser through a simple grammar using ANTLR. What I've done right now :
grammar ExprV2;
#header {
package mypack.parte2;
}
#lexer::header {
package mypack.parte2;
}
start
: expr EOF { System.out.println($expr.val);}
;
expr returns [int val]
: term e=exprP[$term.val] { $val = $e.val; }
;
exprP[int i] returns [int val]
: { $val = $i; }
| '+' term e=exprP[$i + $term.val] { $val = $e.val; }
| '-' term e=exprP[$i - $term.val] { $val = $e.val; }
;
term returns [int val]
: fact e=termP[$fact.val] { $val = $e.val; }
;
termP[int i] returns [int val]
: {$val = $i;}
| '*' fact e=termP[$i * $fact.val] {$val = $e.val; }
| '/' fact e=termP[$i / $fact.val] {$val = $e.val; }
;
fact returns [int val]
: '(' expr ')' { $val = $expr.val; }
| NUM { $val=Integer.parseInt($NUM.text); }
;
NUM : '0'..'9'+ ;
WS : (' ' | '\t' |'\r' | '\n')+ { skip(); };
What I would like to obtain is to generate a Lexer/Parser with a grammar written using EBNF but I'm stucked and I don't know how to go ahead. I looked on the internet but I did not succeed in finding anything clear. Thanks to all!

Simplified smalltalk grammar using antlr - unary minus and message chaining

I am writing simple smalltalk-like grammar using antlr. It is simplified version of smalltalk, but basic ideas are the same (message passing for example).
Here is my grammar so far:
grammar GAL;
options {
//k=2;
backtrack=true;
}
ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*
;
INT : '0'..'9'+
;
FLOAT
: ('0'..'9')+ '.' ('0'..'9')* EXPONENT?
| '.' ('0'..'9')+ EXPONENT?
| ('0'..'9')+ EXPONENT
;
COMMENT
: '"' ( options {greedy=false;} : . )* '"' {$channel=HIDDEN;}
;
WS : ( ' '
| '\t'
) {$channel=HIDDEN;}
;
NEW_LINE
: ('\r'?'\n')
;
STRING
: '\'' ( ESC_SEQ | ~('\\'|'\'') )* '\''
;
fragment
EXPONENT : ('e'|'E') ('+'|'-')? ('0'..'9')+ ;
fragment
HEX_DIGIT : ('0'..'9'|'a'..'f'|'A'..'F') ;
fragment
ESC_SEQ
: '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\')
| UNICODE_ESC
| OCTAL_ESC
;
fragment
OCTAL_ESC
: '\\' ('0'..'3') ('0'..'7') ('0'..'7')
| '\\' ('0'..'7') ('0'..'7')
| '\\' ('0'..'7')
;
fragment
UNICODE_ESC
: '\\' 'u' HEX_DIGIT HEX_DIGIT HEX_DIGIT HEX_DIGIT
;
BINARY_MESSAGE_CHAR
: ('~' | '!' | '#' | '%' | '&' | '*' | '-' | '+' | '=' | '|' | '\\' | '<' | '>' | ',' | '?' | '/')
('~' | '!' | '#' | '%' | '&' | '*' | '-' | '+' | '=' | '|' | '\\' | '<' | '>' | ',' | '?' | '/')?
;
// parser
program
: NEW_LINE* (statement (NEW_LINE+ | EOF))*
;
statement
: message_sending
| return_statement
| assignment
| temp_variables
;
return_statement
: '^' statement
;
assignment
: identifier ':=' statement
;
temp_variables
: '|' identifier+ '|'
;
object
: raw_object
;
raw_object
: number
| string
| identifier
| literal
| block
| '(' message_sending ')'
;
message_sending
: keyword_message_sending
;
keyword_message_sending
: binary_message_sending keyword_message?
;
binary_message_sending
: unary_message_sending binary_message*
;
unary_message_sending
: object (unary_message)*
;
unary_message
: unary_message_selector
;
binary_message
: binary_message_selector unary_message_sending
;
keyword_message
: (NEW_LINE? single_keyword_message_selector NEW_LINE? binary_message_sending)+
;
block
:
'[' (block_signiture
)? NEW_LINE*
block_body
NEW_LINE* ']'
;
block_body
: (statement
)?
(NEW_LINE+ statement
)*
;
block_signiture
:
(':' identifier
)+ '|'
;
unary_message_selector
: identifier
;
binary_message_selector
: BINARY_MESSAGE_CHAR
;
single_keyword_message_selector
: identifier ':'
;
keyword_message_selector
: single_keyword_message_selector+
;
symbol
: '#' (string | identifier | binary_message_selector | keyword_message_selector)
;
literal
: symbol block? // if there is block then this is method
;
number
: /*'-'?*/
( INT | FLOAT )
;
string
: STRING
;
identifier
: ID
;
1. Unary Minus
I have a problem with unary minus for numbers (commented part for rule number). The problem is that minus is valid binary message. To make things worse two minus signs are also valid binary message. What I need is unary minus in case where there is no object to send binary message to (for example, -3+4 should be unary minus because there is nothing in frot of -3). Also, (-3) should be binary minus too. It would be great if 1 -- -2 would be binary message '--' with parameter -2, but I can live without that. How can I do this?
If I uncomment unary minus I get error MismatchedSetException(0!=null) when parsing something like 1-2.
2. Message chaining
What would be best way to implement message chainging like in smalltalk? What I mean by this is something like this:
obj message1 + 3;
message2;
+ 3;
keyword: 2+3
where every message would be sent to the same object, in this case obj. Message precedence should be kept (unary > binary > keyword).
3. Backtrack
Most of this grammar can be parsed with k=2, but when input is something like this:
1 + 2
Obj message:
1 + 2
message2: 'string'
parser tries to match Obj as single_keyword_message_selector and raises UnwantedTokenExcaption on token message. If remove k=2 and set backtrack=true (as I did) everything works as it should. How can I remove backtrack and get desired behaviour?
Also, most of the grammar can be parsed using k=1, so I tried to set k=2 only for rules that require it, but that is ignored. I did something like this:
rule
options { k = 2; }
: // rule definition
;
but it doesn't work until I set k in global options. What am I missing here?
Update:
It is not ideal solution to write grammar from scratch, because I have a lot of code that depends on it. Also, some features of smalltalk that are missing - are missing by design. This is not intended to be another smalltalk implementation, smalltalk was just an inspiration.
I would be more then happy to have unary minus working in cases like this: -1+2 or 2+(-1). Cases like 2 -- -1 are just not so important.
Also, message chaining is something that should be done as simple as posible. That means that I don't like idea of changeing AST I am generating.
About backtrack - I can live with it, just asked here out of personal curiosity.
This is little modified grammar that generates AST - maybe it will help to better understand what I don't want to change. (temp_variables are probably going to be deleted, I havent made that decision).
grammar GAL;
options {
//k=2;
backtrack=true;
language=CSharp3;
output=AST;
}
tokens {
HASH = '#';
COLON = ':';
DOT = '.';
CARET = '^';
PIPE = '|';
LBRACKET = '[';
RBRACKET = ']';
LPAREN = '(';
RPAREN = ')';
ASSIGN = ':=';
}
// generated files options
#namespace { GAL.Compiler }
#lexer::namespace { GAL.Compiler}
// this will disable CLSComplaint warning in ANTLR generated code
#parser::header {
// Do not bug me about [System.CLSCompliant(false)]
#pragma warning disable 3021
}
#lexer::header {
// Do not bug me about [System.CLSCompliant(false)]
#pragma warning disable 3021
}
ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*
;
INT : '0'..'9'+
;
FLOAT
: ('0'..'9')+ '.' ('0'..'9')* EXPONENT?
| '.' ('0'..'9')+ EXPONENT?
| ('0'..'9')+ EXPONENT
;
COMMENT
: '"' ( options {greedy=false;} : . )* '"' {$channel=Hidden;}
;
WS : ( ' '
| '\t'
) {$channel=Hidden;}
;
NEW_LINE
: ('\r'?'\n')
;
STRING
: '\'' ( ESC_SEQ | ~('\\'|'\'') )* '\''
;
fragment
EXPONENT : ('e'|'E') ('+'|'-')? ('0'..'9')+ ;
fragment
HEX_DIGIT : ('0'..'9'|'a'..'f'|'A'..'F') ;
fragment
ESC_SEQ
: '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\')
| UNICODE_ESC
| OCTAL_ESC
;
fragment
OCTAL_ESC
: '\\' ('0'..'3') ('0'..'7') ('0'..'7')
| '\\' ('0'..'7') ('0'..'7')
| '\\' ('0'..'7')
;
fragment
UNICODE_ESC
: '\\' 'u' HEX_DIGIT HEX_DIGIT HEX_DIGIT HEX_DIGIT
;
BINARY_MESSAGE_CHAR
: ('~' | '!' | '#' | '%' | '&' | '*' | '-' | '+' | '=' | '|' | '\\' | '<' | '>' | ',' | '?' | '/')
('~' | '!' | '#' | '%' | '&' | '*' | '-' | '+' | '=' | '|' | '\\' | '<' | '>' | ',' | '?' | '/')?
;
// parser
public program returns [ AstProgram program ]
: { $program = new AstProgram(); }
NEW_LINE*
( statement (NEW_LINE+ | EOF)
{ $program.AddStatement($statement.stmt); }
)*
;
statement returns [ AstNode stmt ]
: message_sending
{ $stmt = $message_sending.messageSending; }
| return_statement
{ $stmt = $return_statement.ret; }
| assignment
{ $stmt = $assignment.assignment; }
| temp_variables
{ $stmt = $temp_variables.tempVars; }
;
return_statement returns [ AstReturn ret ]
: CARET statement
{ $ret = new AstReturn($CARET, $statement.stmt); }
;
assignment returns [ AstAssignment assignment ]
: dotted_expression ASSIGN statement
{ $assignment = new AstAssignment($dotted_expression.dottedExpression, $ASSIGN, $statement.stmt); }
;
temp_variables returns [ AstTempVariables tempVars ]
: p1=PIPE
{ $tempVars = new AstTempVariables($p1); }
( identifier
{ $tempVars.AddVar($identifier.identifier); }
)+
p2=PIPE
{ $tempVars.EndToken = $p2; }
;
object returns [ AstNode obj ]
: number
{ $obj = $number.number; }
| string
{ $obj = $string.str; }
| dotted_expression
{ $obj = $dotted_expression.dottedExpression; }
| literal
{ $obj = $literal.literal; }
| block
{ $obj = $block.block; }
| LPAREN message_sending RPAREN
{ $obj = $message_sending.messageSending; }
;
message_sending returns [ AstKeywordMessageSending messageSending ]
: keyword_message_sending
{ $messageSending = $keyword_message_sending.keywordMessageSending; }
;
keyword_message_sending returns [ AstKeywordMessageSending keywordMessageSending ]
: binary_message_sending
{ $keywordMessageSending = new AstKeywordMessageSending($binary_message_sending.binaryMessageSending); }
( keyword_message
{ $keywordMessageSending = $keywordMessageSending.NewMessage($keyword_message.keywordMessage); }
)?
;
binary_message_sending returns [ AstBinaryMessageSending binaryMessageSending ]
: unary_message_sending
{ $binaryMessageSending = new AstBinaryMessageSending($unary_message_sending.unaryMessageSending); }
( binary_message
{ $binaryMessageSending = $binaryMessageSending.NewMessage($binary_message.binaryMessage); }
)*
;
unary_message_sending returns [ AstUnaryMessageSending unaryMessageSending ]
: object
{ $unaryMessageSending = new AstUnaryMessageSending($object.obj); }
(
unary_message
{ $unaryMessageSending = $unaryMessageSending.NewMessage($unary_message.unaryMessage); }
)*
;
unary_message returns [ AstUnaryMessage unaryMessage ]
: unary_message_selector
{ $unaryMessage = new AstUnaryMessage($unary_message_selector.unarySelector); }
;
binary_message returns [ AstBinaryMessage binaryMessage ]
: binary_message_selector unary_message_sending
{ $binaryMessage = new AstBinaryMessage($binary_message_selector.binarySelector, $unary_message_sending.unaryMessageSending); }
;
keyword_message returns [ AstKeywordMessage keywordMessage ]
:
{ $keywordMessage = new AstKeywordMessage(); }
(
NEW_LINE?
single_keyword_message_selector
NEW_LINE?
binary_message_sending
{ $keywordMessage.AddMessagePart($single_keyword_message_selector.singleKwSelector, $binary_message_sending.binaryMessageSending); }
)+
;
block returns [ AstBlock block ]
: LBRACKET
{ $block = new AstBlock($LBRACKET); }
(
block_signiture
{ $block.Signiture = $block_signiture.blkSigniture; }
)? NEW_LINE*
block_body
{ $block.Body = $block_body.blkBody; }
NEW_LINE*
RBRACKET
{ $block.SetEndToken($RBRACKET); }
;
block_body returns [ IList<AstNode> blkBody ]
#init { $blkBody = new List<AstNode>(); }
:
( s1=statement
{ $blkBody.Add($s1.stmt); }
)?
( NEW_LINE+ s2=statement
{ $blkBody.Add($s2.stmt); }
)*
;
block_signiture returns [ AstBlockSigniture blkSigniture ]
#init { $blkSigniture = new AstBlockSigniture(); }
:
( COLON identifier
{ $blkSigniture.AddIdentifier($COLON, $identifier.identifier); }
)+ PIPE
{ $blkSigniture.SetEndToken($PIPE); }
;
unary_message_selector returns [ AstUnaryMessageSelector unarySelector ]
: identifier
{ $unarySelector = new AstUnaryMessageSelector($identifier.identifier); }
;
binary_message_selector returns [ AstBinaryMessageSelector binarySelector ]
: BINARY_MESSAGE_CHAR
{ $binarySelector = new AstBinaryMessageSelector($BINARY_MESSAGE_CHAR); }
;
single_keyword_message_selector returns [ AstIdentifier singleKwSelector ]
: identifier COLON
{ $singleKwSelector = $identifier.identifier; }
;
keyword_message_selector returns [ AstKeywordMessageSelector keywordSelector ]
#init { $keywordSelector = new AstKeywordMessageSelector(); }
:
( single_keyword_message_selector
{ $keywordSelector.AddIdentifier($single_keyword_message_selector.singleKwSelector); }
)+
;
symbol returns [ AstSymbol symbol ]
: HASH
( string
{ $symbol = new AstSymbol($HASH, $string.str); }
| identifier
{ $symbol = new AstSymbol($HASH, $identifier.identifier); }
| binary_message_selector
{ $symbol = new AstSymbol($HASH, $binary_message_selector.binarySelector); }
| keyword_message_selector
{ $symbol = new AstSymbol($HASH, $keyword_message_selector.keywordSelector); }
)
;
literal returns [ AstNode literal ]
: symbol
{ $literal = $symbol.symbol; }
( block
{ $literal = new AstMethod($symbol.symbol, $block.block); }
)? // if there is block then this is method
;
number returns [ AstNode number ]
: /*'-'?*/
( INT
{ $number = new AstInt($INT); }
| FLOAT
{ $number = new AstInt($FLOAT); }
)
;
string returns [ AstString str ]
: STRING
{ $str = new AstString($STRING); }
;
dotted_expression returns [ AstDottedExpression dottedExpression ]
: i1=identifier
{ $dottedExpression = new AstDottedExpression($i1.identifier); }
(DOT i2=identifier
{ $dottedExpression.AddIdentifier($i2.identifier); }
)*
;
identifier returns [ AstIdentifier identifier ]
: ID
{ $identifier = new AstIdentifier($ID); }
;
Hi Smalltalk Grammar writer,
Firstly, to get a smalltalk grammar to parse properly (1 -- -2) and to support the optional '.' on the last statement, etc., you should treat whitespace as significant. Don't put it on the hidden channel.
The grammar so far is not breaking down the rules into small enough fragments. This will be a problem like you have seen with K=2 and backtracking.
I suggest you check out a working Smalltalk grammar in ANTLR as defined by the Redline Smalltalk project http://redline.st & https://github.com/redline-smalltalk/redline-smalltalk
Rgs, James.

Why is Antlr going into an infinite loop when processing my grammar

I created an ANTLR grammar for parsing mathematical expressions and a second one for evaluating them. As I thought building an AST and then re-parsing in order to actually evaluate it is sort of one operation too much, I wanted to refactor my grammar to produce a hierarchy of "Term" objects representing the expression including the logic to perform that particular operation. The root Term object can then be simply evaluated to a concrete result.
I had to rewrite quite a lot of my grammar and finally got rid of the last error message. Unfortunately now ANTLR seems to sort of go into an infinite loop.
Could someone here please help me sort out the problem? I think the grammar should be pretty interesting for some, therefore I am posting it. (It is based upon a garmmar I found with google, I should admit, but I have altered it quite a lot to suite my needs).
grammar SecurityRulesNew;
options {
language = Java;
output=AST;
backtrack = true;
ASTLabelType=CommonTree;
k=2;
}
tokens {
POS;
NEG;
CALL;
}
#header {package de.cware.cweb.services.evaluator.parser;}
#lexer::header{package de.cware.cweb.services.evaluator.parser;}
formula returns [Term term]
: a=expression EOF { $term = a; }
;
expression returns [Term term]
: a=boolExpr { $term = a; }
;
boolExpr returns [Term term]
: a=sumExpr { $term = a; }
| a=sumExpr AND b=boolExpr { $term = new AndTerm(a, b); }
| a=sumExpr OR b=boolExpr { $term = new OrTerm(a, b); }
| a=sumExpr LT b=boolExpr { $term = new LessThanTerm(a, b); }
| a=sumExpr LTEQ b=boolExpr { $term = new LessThanOrEqualTerm(a, b); }
| a=sumExpr GT b=boolExpr { $term = new GreaterThanTerm(a, b); }
| a=sumExpr GTEQ b=boolExpr { $term = new GreaterThanTermOrEqual(a, b); }
| a=sumExpr EQ b=boolExpr { $term = new EqualsTerm(a, b); }
| a=sumExpr NOTEQ b=boolExpr { $term = new NotEqualsTerm(a, b); }
;
sumExpr returns [Term term]
: a=productExpr { $term = a; }
| a=productExpr SUB b=sumExpr { $term = new SubTerm(a, b); }
| a=productExpr ADD b=sumExpr { $term = new AddTerm(a, b); }
;
productExpr returns [Term term]
: a=expExpr { $term = a; }
| a=expExpr DIV productExpr { $term = new DivTerm(a, b); }
| a=expExpr MULT productExpr { $term = new MultTerm(a, b); }
;
expExpr returns [Term term]
: a=unaryOperation { $term = a; }
| a=unaryOperation EXP expExpr { $term = new ExpTerm(a, b); }
;
unaryOperation returns [Term term]
: a=operand { $term = a; }
| NOT a=operand { $term = new NotTerm(a); }
| SUB a=operand { $term = new NegateTerm(a); }
;
operand returns [Term term]
: l=literal { $term = l; }
| f=functionExpr { $term = f; }
| v=VARIABLE { $term = new VariableTerm(v); }
| LPAREN e=expression RPAREN { $term = e; }
;
functionExpr returns [Term term]
: f=FUNCNAME LPAREN! RPAREN! { $term = new CallFunctionTerm(f, null); }
| f=FUNCNAME LPAREN! a=arguments RPAREN! { $term = new CallFunctionTerm(f, a); }
;
arguments returns [List<Term> terms]
: a=expression
{
$terms = new ArrayList<Term>();
$terms.add(a);
}
| a=expression COMMA b=arguments
{
$terms = new ArrayList<Term>();
$terms.add(a);
$terms.addAll(b);
}
;
literal returns [Term term]
: n=NUMBER { $term = new NumberLiteral(n); }
| s=STRING { $term = new StringLiteral(s); }
| t=TRUE { $term = new TrueLiteral(t); }
| f=FALSE { $term = new FalseLiteral(f); }
;
STRING
:
'\"'
( options {greedy=false;}
: ESCAPE_SEQUENCE
| ~'\\'
)*
'\"'
|
'\''
( options {greedy=false;}
: ESCAPE_SEQUENCE
| ~'\\'
)*
'\''
;
WHITESPACE
: (' ' | '\n' | '\t' | '\r')+ {skip();};
TRUE
: ('t'|'T')('r'|'R')('u'|'U')('e'|'E')
;
FALSE
: ('f'|'F')('a'|'A')('l'|'L')('s'|'S')('e'|'E')
;
NOTEQ : '!=';
LTEQ : '<=';
GTEQ : '>=';
AND : '&&';
OR : '||';
NOT : '!';
EQ : '=';
LT : '<';
GT : '>';
EXP : '^';
MULT : '*';
DIV : '/';
ADD : '+';
SUB : '-';
LPAREN : '(';
RPAREN : ')';
COMMA : ',';
PERCENT : '%';
VARIABLE
: '[' ~('[' | ']')+ ']'
;
FUNCNAME
: (LETTER)+
;
NUMBER
: (DIGIT)+ ('.' (DIGIT)+)?
;
fragment
LETTER
: ('a'..'z') | ('A'..'Z')
;
fragment
DIGIT
: ('0'..'9')
;
fragment
ESCAPE_SEQUENCE
: '\\' 't'
| '\\' 'n'
| '\\' '\"'
| '\\' '\''
| '\\' '\\'
;
Help is greatly appreciated.
Chris
Because your grammar is so incredibly ambiguous, ANTLR has a problem creating a parser. Apparently ANTLR 3.3+ chokes on it, but ANTLR 3.2 (with less time than 3.3+) produces the following error:
error(10): internal error: org.antlr.tool.Grammar.createLookaheadDFA(Grammar.java:1279):
could not even do k=1 for decision 1; reason: timed out (>1000ms)
For a simple expression parser, you really shouldn't use backtrack=true.
Besides the fact your grammar is ambiguous, much of your embedded code contains errors.
Let's have a look at your formula rule:
formula returns [Term term]
: a=expression EOF { $term = $a; }
;
Also, the return type of a rule should be explicitly defined. The a in { $term = a; } should have a $ in front of it:
formula returns [Term term]
: a=expression EOF { $term = $a; }
;
but then $a refers to the entire "thing" expression returns. You then have to "tell" ANTLR you want the Term this expression creates. This can be done like this:
formula returns [Term term]
: a=expression EOF { $term = $a.term; }
;
expression returns [Term term]
: a=boolExpr { $term = $a.term; }
;
It looks like you've converted some LR grammar into an ANTLR grammar (note that although ANTLR ends with LR, ANTLR 3.x is an LL parser generator) and without testing in between, you had hoped it should all work: unfortunately, it doesn't. There's too much wrong with it to produce a small working example based on your grammar: I'd have a look at an existing expression parser based on an ANTLR grammar and try again. Have a look at these Q&A's:
ANTLR: Is there a simple example?
ANTLR: From CommonTree to useful object graph
First of all, thank you for that detailed explanation. That really helps :-) ... All of the "$a.term" and similar stuff is sorted out now and code is generated that actually compiles (I simply hacked in that code wanting to fix the issues with that as soon as something is generated at all). I simply commented out a lot of options and keept on generating until I came to the one fragment that seems to break the build. I turned on that backtrack feature, because some errors I got, suggested that I turn it on.
EDIT:
Well I actually refactored the grammar to get rid of the errors without activating backtrack and now my parser is generated really fast and it seems to do it's job nicely. Here comes the current version:
grammar SecurityRulesNew;
options {
language = Java;
output=AST;
ASTLabelType=CommonTree;
/* backtrack = true;*/
}
tokens {
POS;
NEG;
CALL;
}
#header {package de.cware.cweb.services.evaluator.parser;
import de.cware.cweb.services.evaluator.terms.*;}
#lexer::header{package de.cware.cweb.services.evaluator.parser;}
formula returns [Term term]
: a=expression EOF { $term = $a.term; }
;
expression returns [Term term]
: a=boolExpr { $term = $a.term; }
;
boolExpr returns [Term term]
: a=sumExpr (AND! b=boolExpr | OR! c=boolExpr | LT! d=boolExpr | LTEQ! e=boolExpr | GT! f=boolExpr | GTEQ! g=boolExpr | EQ! h=boolExpr | NOTEQ! i=boolExpr)? {
if(b != null) {
$term = new AndTerm($a.term, $b.term);
} else if(c != null) {
$term = new OrTerm($a.term, $c.term);
} else if(d != null) {
$term = new LessThanTerm($a.term, $d.term);
} else if(e != null) {
$term = new LessThanOrEqualTerm($a.term, $e.term);
} else if(f != null) {
$term = new GreaterThanTerm($a.term, $f.term);
} else if(g != null) {
$term = new GreaterThanOrEqualTerm($a.term, $g.term);
} else if(h != null) {
$term = new EqualsTerm($a.term, $h.term);
} else if(i != null) {
$term = new NotEqualsTerm($a.term, $i.term);
} else {
$term = $a.term;
}
}
;
sumExpr returns [Term term]
: a=productExpr (SUB! b=sumExpr | ADD! c=sumExpr)?
{
if(b != null) {
$term = new SubTerm($a.term, $b.term);
} else if(c != null) {
$term = new AddTerm($a.term, $c.term);
} else {
$term = $a.term;
}
}
;
productExpr returns [Term term]
: a=expExpr (DIV! b=productExpr | MULT! c=productExpr)?
{
if(b != null) {
$term = new DivTerm($a.term, $b.term);
} else if(c != null) {
$term = new MultTerm($a.term, $c.term);
} else {
$term = $a.term;
}
}
;
expExpr returns [Term term]
: a=unaryOperation (EXP! b=expExpr)?
{
if(b != null) {
$term = new ExpTerm($a.term, $b.term);
} else {
$term = $a.term;
}
}
;
unaryOperation returns [Term term]
: a=operand { $term = $a.term; }
| NOT! a=operand { $term = new NotTerm($a.term); }
| SUB! a=operand { $term = new NegateTerm($a.term); }
| LPAREN! e=expression RPAREN! { $term = $e.term; }
;
operand returns [Term term]
: l=literal { $term = $l.term; }
| v=VARIABLE { $term = new VariableTerm($v.text); }
| f=functionExpr { $term = $f.term; }
;
functionExpr returns [Term term]
: f=FUNCNAME LPAREN! (a=arguments)? RPAREN! { $term = new CallFunctionTerm($f.text, $a.terms); }
;
arguments returns [List<Term> terms]
: a=expression (COMMA b=arguments)?
{
$terms = new ArrayList<Term>();
$terms.add($a.term);
if(b != null) {
$terms.addAll($b.terms);
}
}
;
literal returns [Term term]
: n=NUMBER { $term = new NumberLiteral(Double.valueOf($n.text)); }
| s=STRING { $term = new StringLiteral($s.text.substring(1, s.getText().length() - 1)); }
| TRUE! { $term = new TrueLiteral(); }
| FALSE! { $term = new FalseLiteral(); }
;
STRING
:
'\"'
( options {greedy=false;}
: ESCAPE_SEQUENCE
| ~'\\'
)*
'\"'
|
'\''
( options {greedy=false;}
: ESCAPE_SEQUENCE
| ~'\\'
)*
'\''
;
WHITESPACE
: (' ' | '\n' | '\t' | '\r')+ {skip();};
TRUE
: ('t'|'T')('r'|'R')('u'|'U')('e'|'E')
;
FALSE
: ('f'|'F')('a'|'A')('l'|'L')('s'|'S')('e'|'E')
;
NOTEQ : '!=';
LTEQ : '<=';
GTEQ : '>=';
AND : '&&';
OR : '||';
NOT : '!';
EQ : '=';
LT : '<';
GT : '>';
EXP : '^';
MULT : '*';
DIV : '/';
ADD : '+';
SUB : '-';
LPAREN : '(';
RPAREN : ')';
COMMA : ',';
PERCENT : '%';
VARIABLE
: '[' ~('[' | ']')+ ']'
;
FUNCNAME
: (LETTER)+
;
NUMBER
: (DIGIT)+ ('.' (DIGIT)+)?
;
fragment
LETTER
: ('a'..'z') | ('A'..'Z')
;
fragment
DIGIT
: ('0'..'9')
;
fragment
ESCAPE_SEQUENCE
: '\\' 't'
| '\\' 'n'
| '\\' '\"'
| '\\' '\''
| '\\' '\\'
;
Thanks again for your explanation ... it got me on the right track :-)
Chris