Antlr parsing numbers problem - antlr

I have a problem parsing integer & hex numbers. I want to parse C++ enums with the following rules:
grammar enum;
rule_enum
: 'enum' ID '{' enum_values+ '}'';';
enum_values
: enum_value (COMMA enum_value)+;
enum_value
: ID ('=' number)?;
number : hex_number | integer_number;
hex_number
: '0' 'x' HEX_DIGIT+;
integer_number
: DIGIT+;
fragment
HEX_DIGIT : ('0'..'9'|'a'..'f'|'A'..'F') ;
fragment
DIGIT : ('0'..'9');
COMMA : ',';
ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*;
The problem I have is the following - when parsing code like:
enum Enum
{
Option1 = 0,
Option2 = 1
};
it does not recognize the 0 as integer_number but tries to parse it as hex_number. How can I resolve this?
Thank you.
Tobias

First, fragment rules can only be "seen" by lexer rules, not parser rules. So, the following is invalid:
integer_number
: DIGIT+; // can't use DIGIT here!
fragment
DIGIT : ('0'..'9');
To fix your ambiguity with these numbers, it's IMO best to make these integer- and hex numbers lexer rules instead of parser rules.
An example:
grammar enum;
rule_enum
: 'enum' ID '{' enum_values+ '}'';';
enum_values
: enum_value (COMMA enum_value)+;
enum_value
: ID ('=' number)?;
number
: HEX_NUMBER
| INTEGER_NUMBER
;
HEX_NUMBER
: '0' 'x' HEX_DIGIT+;
INTEGER_NUMBER
: DIGIT+;
fragment
HEX_DIGIT : ('0'..'9'|'a'..'f'|'A'..'F') ;
fragment
DIGIT : ('0'..'9');
COMMA : ',';
ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*;
SPACE : (' ' | '\t' | '\r' | '\n') {skip();};
which produces the following parse tree of your example snippet:

The following ANTLR works for just the number bit of the enum.
(editted to include Bart's advice below)
grammar enum;
number :
integer_number | hex_number ;
hex_number
: HEX_NUMBER;
integer_number
: INT_NUMBER;
HEX_NUMBER
: HEX_INTRO HEX_DIGIT+;
INT_NUMBER
: DIGIT+;
HEX_INTRO
: '0x';
DIGIT : ('0'..'9');
HEX_DIGIT
: ('0'..'9'|'a'..'f'|'A'..'F') ;

Related

"The following sets of rules are mutually left-recursive"

I have tried to write a grammar to recognize expressions like:
(A + MAX(B) ) / ( C - AVERAGE(A) )
IF( A > AVERAGE(A), 0, 1 )
X / (MAX(X)
Unfortunately antlr3 fails with these errors:
error(210): The following sets of rules are mutually left-recursive [unaryExpression, additiveExpression, primaryExpression, formula, multiplicativeExpression]
error(211): DerivedKeywords.g:110:13: [fatal] rule booleanTerm has non-LL(*) decision due to recursive rule invocations reachable from alts 1,2. Resolve by left-factoring or using syntactic predicates or using backtrack=true option.
error(206): DerivedKeywords.g:110:13: Alternative 1: after matching input such as decision cannot predict what comes next due to recursion overflow to additiveExpression from formula
I have spent some hours trying to fix these, it would be great if anyone could at least help me fix the first problem. Thanks
Code:
grammar DerivedKeywords;
options {
output=AST;
//backtrack=true;
}
WS : ( ' ' | '\t' | '\n' | '\r' )
{ skip(); }
;
//for numbers
DIGIT
: '0'..'9'
;
//for both integer and real number
NUMBER
: (DIGIT)+ ( '.' (DIGIT)+ )?( ('E'|'e')('+'|'-')?(DIGIT)+ )?
;
// Boolean operatos
AND : 'AND';
OR : 'OR';
NOT : 'NOT';
EQ : '=';
NEQ : '!=';
GT : '>';
LT : '<';
GTE : '>=';
LTE : '<=';
COMMA : ',';
// Token for Functions
IF : 'IF';
MAX : 'MAX';
MIN : 'MIN';
AVERAGE : 'AVERAGE';
VARIABLE : 'A'..'Z' ('A'..'Z' | '0'..'9')*
;
// OPERATORS
LPAREN : '(' ;
RPAREN : ')' ;
DIV : '/' ;
PLUS : '+' ;
MINUS : '-' ;
STAR : '*' ;
expression : formula;
formula
: functionExpression
| additiveExpression
| LPAREN! a=formula RPAREN! // First Problem
;
additiveExpression
: a=multiplicativeExpression ( (MINUS^ | PLUS^ ) b=multiplicativeExpression )*
;
multiplicativeExpression
: a=unaryExpression ( (STAR^ | DIV^ ) b=unaryExpression )*
;
unaryExpression
: MINUS^ u=unaryExpression
| primaryExpression
;
functionExpression
: f=functionOperator LPAREN e=formula RPAREN
| IF LPAREN b=booleanExpression COMMA p=formula COMMA s=formula RPAREN
;
functionOperator :
MAX | MIN | AVERAGE;
primaryExpression
: NUMBER
// Used for scientific numbers
| DIGIT
| VARIABLE
| formula
;
// Boolean stuff
booleanExpression
: orExpression;
orExpression : a=andExpression (OR^ b=andExpression )*
;
andExpression
: a=notExpression (AND^ b=notExpression )*
;
notExpression
: NOT^ t=booleanTerm
| booleanTerm
;
booleanOperator :
GT | LT | EQ | GTE | LTE | NEQ;
booleanTerm : a=formula op=booleanOperator b=formula
| LPAREN! booleanTerm RPAREN! // Second problem
;
error(210): The following sets of rules are mutually left-recursive [unaryExpression, additiveExpression, primaryExpression, formula, multiplicativeExpression]
- this means that if the parser enters unaryExpression rule, it has the possibility to match additiveExpression, primaryExpression, formula, multiplicativeExpression and unaryExpression again without ever consuming a single token from input - so it cannot decide whether to use those rules or not, because even if it uses the rules, the input will be the same.
You're probably trying to allow subexpressions in expressions by this sequence of rules - you need to make sure that path will consume the left parenthesis of the subexpression. Probably the formula alternative in primaryExpression should be changed to LPAREN formula RPAREN, and the rest of grammar be adjusted accordingly.

ParserRule matching the wrong token

I'm trying to learn a bit ANTLR4 and define a grammar for some 4GL language.
This is what I've got:
compileUnit
:
typedeclaration EOF
;
typedeclaration
:
ID LPAREN DATATYPE INT RPAREN
;
DATATYPE
:
DATATYPE_ALPHANUMERIC
| DATATYPE_NUMERIC
;
DATATYPE_ALPHANUMERIC
:
'A'
;
DATATYPE_NUMERIC
:
'N'
;
fragment
DIGIT
:
[0-9]
;
fragment
LETTER
:
[a-zA-Z]
;
INT
:
DIGIT+
;
ID
:
LETTER
(
LETTER
| DIGIT
)*
;
LPAREN
:
'('
;
RPAREN
:
')'
;
WS
:
[ \t\f]+ -> skip
;
What I want to be able to parse:
TEST (A10)
what I get:
typedeclaration:1:6: mismatched input 'A10' expecting DATATYPE
I am however able to write:
TEST (A 10)
Why do I need to put a whitespace in here? The LPAREN DATATYPE in itself is working, so there is no need for a space inbetween. Also the INT RPAREN is working.
Why is a space needed between DATATYPE and INT? I'm a bit confused on that one.
I guess that it's matching ID because it's the "longest" match, but there must be some way to force to be lazier here, right?
You should ignore 'A' and 'N' chats at first position of ID. As #CoronA noticed ANTLR matches token as long as possible (length of ID 'A10' more than length of DATATYPE_ALPHANUMERIC 'A'). Also read this: Priority rules. Try to use the following grammar:
grammar expr;
compileUnit
: typedeclaration EOF
;
typedeclaration
: ID LPAREN datatype INT RPAREN
;
datatype
: DATATYPE_ALPHANUMERIC
| DATATYPE_NUMERIC
;
DATATYPE_ALPHANUMERIC
: 'A'
;
DATATYPE_NUMERIC
: 'N'
;
INT
: DIGIT+
;
ID
: [b-mo-zB-MO-Z] (LETTER | DIGIT)*
;
LPAREN
: '('
;
RPAREN
: ')'
;
WS
: [ \t\f]+ -> skip
;
fragment
DIGIT
: [0-9]
;
fragment
LETTER
: [a-zA-Z]
;
Also you can use the following grammar without id restriction. Data types will be recognized earlier than letters. it's not clear too:
grammar expr;
compileUnit
: typedeclaration EOF
;
typedeclaration
: id LPAREN datatype DIGIT+ RPAREN
;
id
: (datatype | LETTER) (datatype | LETTER | DIGIT)*
;
datatype
: DATATYPE_ALPHANUMERIC
| DATATYPE_NUMERIC
;
DATATYPE_ALPHANUMERIC: 'A';
DATATYPE_NUMERIC: 'N';
// List with another Data types.
LETTER: [a-zA-Z];
LPAREN
: '('
;
RPAREN
: ')'
;
WS
: [ \t\f]+ -> skip
;
DIGIT
: [0-9]
;

Guide or approval for ANTLR example

I have an AlgebraRelacional.g4 file with this. I need to read a file with a syntax like a CSV file, put the content in some memory tables and then resolve relational algebra operations with that. Can you tell me if I am doing it right?
Example data file to read:
cod_buy(char);name_suc(char);Import(int);date_buy(date)
“P-11”;”DC Med”;900;01/03/14
“P-14”;”Center”;1500;02/05/14
Current ANTLR grammar:
grammar AlgebraRelacional;
SEL : '\u03C3'
;
PRO : '\u220F'
;
UNI : '\u222A'
;
DIF : '\u002D'
;
PROC : '\u0058'
;
INT : '\u2229'
;
AND : 'AND'
;
OR : 'OR'
;
NOT : 'NOT'
;
EQ : '='
;
DIFERENTE : '!='
;
MAYOR : '>'
;
MENOR : '<'
;
SUMA : '+'
;
MULTI : '*'
;
IPAREN : '('
;
DPAREN : ')'
;
COMA : ','
;
PCOMA : ';'
;
Comillas: '"'
;
file : hdr row+ ;
hdr : row ;
row : field (',' field)* '\r'? '\n' ;
field : TEXT | STRING | ;
TEXT : ~[,\n\r"]+ ;
STRING : '"' ('""'|~'"')* '"' ;
I suggest you that read this document (http://is.muni.cz/th/208197/fi_b/bc_thesis.pdf), It contains usefull information about how to write a parser for relational algebra. That is not ANTLR, but you only has to translate the grammar in BNF to EBNF.

Antlr grammar won't work

My grammar won't accept inputs like "x = 4, xy = 1"
grammar Ex5;
prog : stat+;
stat : ID '=' NUMBER;
LETTER : ('a'..'z'|'A'..'Z');
DIGIT : ('0'..'9');
ID : LETTER+;
NUMBER : DIGIT+;
WS : (' '|'\t'| '\r'? '\n' )+ {skip();};
What do i have to do to make it accept a multitude of inputs like ID = NUMBER??
Thank you in advance.
You'll have to account for the comma, ',', inside your grammar. Also, since you (most probably) don't want LETTER and DIGIT tokens to be created since they are only used in other lexer rules, you should make these fragments:
grammar Ex5;
prog : stat (',' stat)*;
stat : ID '=' NUMBER;
ID : LETTER+;
NUMBER : DIGIT+;
fragment LETTER : 'a'..'z' | 'A'..'Z';
fragment DIGIT : '0'..'9';
WS : (' '|'\t'| '\r'? '\n')+ {skip();};

antlr grammar for [Foo]-[Bar]-[Baz][X]-[Y][Z]

I am trying to define a grammar that allows for
[Foo]-[Bar]-[Baz][X]-[Y][Z]
I want to parse this as one set.
I currently have
grammar Sample;
items : (item association? item?)*;
item : LBRACK ID RBRACK;
association : ASSOCIATION;
RBRACK : ']';
LBRACK : '[';
ASSOCIATION : '-';
ID : ('a'..'z' | 'A'..'Z' | '_')*;
But this gives an error
warning(200): Sample.g:3:30: Decision can match input such as
"LBRACK ID RBRACK" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
And as a diagram I get
And I get this railroad diagram
How can I fix this?
Given the input [Foo]-[Bar]-[Baz][X]-[Y][Z], the following:
chunk : items+ EOF;
items : item (ASSOCIATION item)* ;
item : '[' ID ']' ;
RBRACK : ']';
LBRACK : '[';
ASSOCIATION : '-';
ID : ('a'..'z' | 'A'..'Z' | '_')*;
SPACE : (' ' | '\t' | '\r' | '\n'){skip();};
produces the following parse tree:
Try something like
items : ( '-'? item )* ;
item : '[' ID ']' ;
ID : ('a'..'z'|'A'..'Z'|'_') ;
which will allow a leading '-' or empty input
or
items : item? ( '-'? item )* ;
item : '[' ID ']' ;
ID : ('a'..'z'|'A'..'Z'|'_') ;
which won't allow a leading '-', but also allows empty input
You could try something like this:
ID : ('a'..'z'|'A'..'Z'|'_')* ;
item : '[' ID ']' ;
dependency : item ( '-' item )+ ;
list : ( item | dependency )+
;