What's wrong with my ANTLR grammar file? - antlr

Question

I defined the following grammar, following Scott Stanchfield tutorial.
grammar SampleScript;
program
:
declaration+
;
declaration
: macrodeclaration
;
macrodeclaration
:
MACRO STRING (LEFTPAREN parameters RIGHTPAREN)?
statement*
ENDMACRO
;
statement
: assignmentStatement
| ifStatement
| iterationStatement
| jumpStatement
| procedureCallStatement
| dimStatement
| labeledStatement
;
actualParameters
: expression (',' expression?)*
;
parameters
: ID (',' ID)*
;
assignmentStatement
: ID ASSIGN expression
| ID MATRIXASSIGN expression
;
ifStatement
: IF expression THEN (statement|compoundStatement)
(ELSE expression (statement|compoundStatement))?
;
iterationStatement
: WHILE expression compoundStatement
| FOR ID '=' expression TO expression (STEP expression)? compoundStatement
;
jumpStatement
: BREAK
| CONTINUE
| GOTO ID
| RETURN LEFTPAREN expression RIGHTPAREN
;
procedureCallStatement //todo: expression statement
: ID LEFTPAREN actualParameters? RIGHTPAREN
;
dimStatement
: DIM ID LEFTBRACKET expression(',' expression)* RIGHTBRACKET (',' ID LEFTBRACKET expression(',' expression)* RIGHTBRACKET)*
;
labeledStatement
: ID ':' statement
;
compoundStatement
: DO statement* END
;
term
: NUMBER
| STRING
| ID
| LEFTPAREN expression RIGHTPAREN //( )
| ID LEFTPAREN actualParameters RIGHTPAREN //Procedure Call
| ID (LEFTBRACKET expression RIGHTBRACKET)+ //Array Arr[3]
| ID ('.' expression)+ //Array Arr.Length
| LEFTBRACE (expression)? (',' expression)* RIGHTBRACE //{"OK","False"}
;
negation
: 'not'* term
;
unary
: ('-')* negation
;
mult
: unary (('*' | '/') unary)*
;
add
: mult (('+' | '-') mult)*
;
relation
: add (('=' | '/=' | '<' | '<=' | '>=' | '>') add)*
;
expression
: relation (('and' | 'or') relation)*
;
//Keywords
DIM: D I M;
RETURN: R E T U R N;
FOR: F O R;
STEP: S T E P;
TO: T O;
WHILE: W H I L E;
DO: D O;
END: E N D;
GOTO: G O T O;
BREAK: B R E A K;
CONTINUE: C O N T I N U E;
IF: I F;
THEN: T H E N;
ELSE: E L S E;
MACRO :M A C R O;
ENDMACRO :E N D M A C R O;
ID : ('_'|LETTER) ('_'|LETTER|DIGIT)*;
ASSIGN: '=';
MATRIXASSIGN: ':=';
LEFTPAREN : '(';
RIGHTPAREN : ')';
LEFTBRACKET : '[';
RIGHTBRACKET : ']';
LEFTBRACE : '{';
RIGHTBRACE : '}';
//STRING : '"' .*? '"' ; // match anything in "..."
STRING
: '"' (STRING_ESCAPE_SEQ|~('\n'|'\r'))*? '"'
| '\'' (STRING_ESCAPE_SEQ|~('\n'|'\r'))*? '\''
;
/// stringescapeseq ::= "\" <any source character>
fragment STRING_ESCAPE_SEQ //'\\"'
: '\\' .
;
UNSIGNED_INT : DIGIT+; //('0' | '1'..'9' '0'..'9'*);
UNSIGNED_FLOAT: DIGIT+ '.' DIGIT* Exponent?
| '.' DIGIT+ Exponent?
| DIGIT+ Exponent
;
NUMBER
: UNSIGNED_INT
| UNSIGNED_FLOAT
;
fragment DIGIT : [0-9] ; // not a token by itself
fragment Exponent : ('e'|'E') ('+'|'-')? (DIGIT)+ ;
LINE_COMMENT : '//' .*? '\r'? '\n' -> skip ; // Match "//" stuff '\n'
COMMENT : '/*' .*? '*/' -> skip ; // Match "/*" stuff "*/"
fragment A:('a'|'A');
fragment B:('b'|'B');
fragment C:('c'|'C');
fragment D:('d'|'D');
fragment E:('e'|'E');
fragment F:('f'|'F');
fragment G:('g'|'G');
fragment H:('h'|'H');
fragment I:('i'|'I');
fragment J:('j'|'J');
fragment K:('k'|'K');
fragment L:('l'|'L');
fragment M:('m'|'M');
fragment N:('n'|'N');
fragment O:('o'|'O');
fragment P:('p'|'P');
fragment Q:('q'|'Q');
fragment R:('r'|'R');
fragment S:('s'|'S');
fragment T:('t'|'T');
fragment U:('u'|'U');
fragment V:('v'|'V');
fragment W:('w'|'W');
fragment X:('x'|'X');
fragment Y:('y'|'Y');
fragment Z:('z'|'Z');
fragment LETTER : [A-Za-z];
WS : [ \t\n\r]+ -> skip ; // skip spaces, tabs, newlines
I am trying to parse following code
Macro 'test' (x)
a=1
b=2
c={}
d = x(3,4)
matrixinfo_skim = GetMatrixInfo(m_skim)
showmessage (i2s(a))
showarray(c)
endmacro
and gets the error below, I spent over 2 days on it and couldn't figure out why it could not parse the assignment statements a=1 and later? someone please help me..
[#0,0:4='Macro',<30>,1:0]
[#1,6:11=''test'',<41>,1:6]
[#2,13:13='(',<35>,1:13]
[#3,14:14='x',<32>,1:14]
[#4,15:15=')',<36>,1:15]
[#5,20:20='a',<32>,3:0]
[#6,21:21='=',<33>,3:1]
[#7,22:22='1',<42>,3:2]
[#8,25:25='b',<32>,4:0]
[#9,26:26='=',<33>,4:1]
[#10,27:27='2',<42>,4:2]
[#11,30:30='c',<32>,5:0]
[#12,31:31='=',<33>,5:1]
[#13,32:32='{',<39>,5:2]
[#14,33:33='}',<40>,5:3]
[#15,36:36='d',<32>,6:0]
[#16,38:38='=',<33>,6:2]
[#17,40:40='x',<32>,6:4]
[#18,41:41='(',<35>,6:5]
[#19,42:42='3',<42>,6:6]
[#20,43:43=',',<2>,6:7]
[#21,44:44='4',<42>,6:8]
[#22,45:45=')',<36>,6:9]
[#23,48:62='matrixinfo_skim',<32>,7:0]
[#24,64:64='=',<33>,7:16]
[#25,66:78='GetMatrixInfo',<32>,7:18]
[#26,79:79='(',<35>,7:31]
[#27,80:85='m_skim',<32>,7:32]
[#28,86:86=')',<36>,7:38]
[#29,91:101='showmessage',<32>,9:0]
[#30,103:103='(',<35>,9:12]
[#31,104:106='i2s',<32>,9:13]
[#32,107:107='(',<35>,9:16]
[#33,108:108='a',<32>,9:17]
[#34,109:109=')',<36>,9:18]
[#35,110:110=')',<36>,9:19]
[#36,113:121='showarray',<32>,10:0]
[#37,122:122='(',<35>,10:9]
[#38,123:123='c',<32>,10:10]
[#39,124:124=')',<36>,10:11]
[#40,127:134='endmacro',<31>,11:0]
[#41,140:139='<EOF>',<-1>,13:0]
line 3:2 extraneous input '1' expecting {'-', 'not', ID, '(', '{', STRING, NUMBER}
line 4:2 extraneous input '2' expecting {'-', 'not', ID, '(', '{', STRING, NUMBER}
line 6:6 mismatched input '3' expecting {'-', 'not', ID, '(', '{', STRING, NUMBER}
line 6:8 extraneous input '4' expecting {',', ')'}
(program (declaration (macrodeclaration Macro 'test' ( (parameters x) ) (statement (assignmentStatement a = (expression (relation (add (mult (unary 1 (negation (term b))))) = (add (mult (unary 2 (negation (term c))))) = (add (mult (unary (negation (term { }))))))))) (statement (assignmentStatement d = (expression (relation (add (mult (unary (negation (term x ( (actualParameters (expression (relation (add (mult (unary 3))))) , 4) )))))))))) (statement (assignmentStatement matrixinfo_skim = (expression (relation (add (mult (unary (negation (term GetMatrixInfo ( (actualParameters (expression (relation (add (mult (unary (negation (term m_skim)))))))) )))))))))) (statement (procedureCallStatement showmessage ( (actualParameters (expression (relation (add (mult (unary (negation (term i2s ( (actualParameters (expression (relation (add (mult (unary (negation (term a)))))))) ))))))))) ))) (statement (procedureCallStatement showarray ( (actualParameters (expression (relation (add (mult (unary (negation (term c)))))))) ))) endmacro)))

Answer 1

As the error messages indicate, things go wrong with the numbers which is matched by the expression in the assignmentStatement rule, which ultimately is (or should be) matched as a NUMBER in the term rule.
Looking at the lexer rules responsible for the creation of a NUMBER token:
UNSIGNED_INT : DIGIT+;
UNSIGNED_FLOAT: DIGIT+ '.' DIGIT* Exponent?
| '.' DIGIT+ Exponent?
| DIGIT+ Exponent
;
NUMBER
: UNSIGNED_INT
| UNSIGNED_FLOAT
;
it appears that a NUMBER token is never created since a NUMBER matches either a UNSIGNED_INT or an UNSIGNED_FLOAT. But since these 2 tokens are defined before the NUMBER is defined, the lexer creates UNSIGNED_INT and UNSIGNED_FLOAT tokens instead of NUMBER tokens.
You need to change UNSIGNED_INT and UNSIGNED_FLOAT into fragment rules instead:
fragment UNSIGNED_INT : DIGIT+;
fragment UNSIGNED_FLOAT: DIGIT+ '.' DIGIT* Exponent?
| '.' DIGIT+ Exponent?
| DIGIT+ Exponent
;
NUMBER
: UNSIGNED_INT
| UNSIGNED_FLOAT
;
Be sure to understand what a fragment is: What does "fragment" mean in ANTLR?