I have the following grammar:
grammar Expr;
expr : '-' expr # unaryOpExpr
| expr ('*'|'/'|'%') expr # mulDivModuloExpr
| expr ('+'|'-') expr # addSubExpr
| '(' expr ')' # nestedExpr
| IDENTIFIER '(' fnArgs? ')' # functionExpr
| IDENTIFIER # identifierExpr
| DOUBLE # doubleExpr
| LONG # longExpr
| STRING # string
;
fnArgs : expr (',' expr)* # functionArgs
;
IDENTIFIER : [_$a-zA-Z][_$a-zA-Z0-9]* | '"' (ESC | ~ ["\\])* '"';
LONG : [0-9]+;
DOUBLE : [0-9]+ '.' [0-9]*;
WS : [ \t\r\n]+ -> skip ;
STRING: '"' (~["\\\r\n] | ESC)* '"';
fragment ESC : '\\' (['"\\/bfnrt] | UNICODE) ;
fragment UNICODE : 'u' HEX HEX HEX HEX ;
fragment HEX : [0-9a-fA-F] ;
MINUS : '-' ;
MUL : '*' ;
DIV : '/' ;
MODULO : '%' ;
PLUS : '+' ;
// math function
MAX: 'MAX';
when I enter following text,It should be effective
-1.1
bug when i enter following text:
-1.1ffff
I think it should report an error, bug antlr didn't do it, antlr captures the previous "-1.1", discard "ffff",
but i want to change this behavior, didn't discard invalid token, but throw exception,report
detection invalid token.
So what should i do, Thanks for your advice
Are you using expr as your main rule? if so make another rule, call it something like parse or program and simply write it like this:
parse: expr EOF;
This will make antlr not ignore trailing tokens that don't make sense, and actually throw an error.
Related
I have a simple grammar like :
grammar CellMath;
equation : expr EOF;
expr
: '-'expr #UnaryNegation // unary minus
| expr op=('*'|'/') expr #MultiplicativeOp // MultiplicativeOperation
| expr op=('+'|'-') expr #AdditiveOp // AdditiveOperation
| FLOAT #Float // Floating Point Number
| INT #Integer // Integer Number
| '(' expr ')' #ParenExpr // Parenthesized Expression
;
MUL : '*' ;
DIV : '/' ;
ADD : '+' ;
SUB : '-' ;
FLOAT
: DIGIT+ '.' DIGIT*
| '.' DIGIT+
;
INT : DIGIT+ ;
fragment
DIGIT : [0-9] ; // match single digit
//fragment
//ATSIGN : [#];
WS : [ \t\r\n]+ -> skip ;
ERRORCHAR : . ;
Not able to throw an exception in case of special char in between expression
[{Number}{SPLChar}{Chars}]
Ex:
"123#abc",
"123&abc".
I expecting an exception to throw
For Example:
Input stream : 123#abc Just like in ANTLR labs Image
But in my case Output : '123' without any errors
I'm using Listener pattern, Error nodes are just ignored not going through VisitTerminal([NotNull] ITerminalNode node) / VisitErrorNode([NotNull] IErrorNode node) in the BaseListener class. Also all the BaseErrorListener class methods has been overridden not even there.
Thanks in advance for your help.
I cannot seem to figure out what antlr is doing here in this grammar. I have a grammar that should match an input like:
i,j : bool;
setvar : set<bool>;
i > 5;
j < 10;
But I keep getting an error telling me that "line 3:13 mismatched input '<' expecting '<'". This tells me there is some ambiguity in the lexer, but I only use '<' in a single token.
Here is the grammar:
//// Parser Rules
grammar MLTL1;
start: block*;
block: var_list ';'
| expr ';'
;
var_list: IDENTIFIER (',' IDENTIFIER)* ':' type ;
type: BASE_TYPE
| KW_SET REL_LT BASE_TYPE REL_GT
;
expr: expr REL_OP expr
| '(' expr ')'
| IDENTIFIER
| INT
;
//// Lexical Spec
// Types
BASE_TYPE: 'bool'
| 'int'
| 'float'
;
// Keywords
KW_SET: 'set' ;
// Op groups for precedence
REL_OP: REL_EQ | REL_NEQ | REL_GT | REL_LT
| REL_GTE | REL_LTE ;
// Relational ops
REL_EQ: '==' ;
REL_NEQ: '!=' ;
REL_GT: '>' ;
REL_LT: '<' ;
REL_GTE: '>=' ;
REL_LTE: '<=' ;
IDENTIFIER
: LETTER (LETTER | DIGIT)*
;
INT
: SIGN? NONZERODIGIT DIGIT*
| '0'
;
fragment
SIGN
: [+-]
;
fragment
DIGIT
: [0-9]
;
fragment
NONZERODIGIT
: [1-9]
;
fragment
LETTER
: [a-zA-Z_]
;
COMMENT : '#' ~[\r\n]* -> skip;
WS : [ \t\r\n]+ -> channel(HIDDEN);
I tested the grammar to see what tokens it is generating for the test input above using this python:
from antlr4 import InputStream, CommonTokenStream
import MLTL1Lexer
import MLTL1Parser
input="""
i,j : bool;
setvar: set<bool>;
i > 5;
j < 10;
"""
lexer = MLTL1Lexer.MLTL1Lexer(InputStream(input))
stream = CommonTokenStream(lexer)
stream.fill()
tokens = stream.getTokens(0,100)
for t in tokens:
print(str(t.type) + " " + t.text)
parser = MLTL1Parser.MLTL1Parser(stream)
parse_tree = parser.start()
print(parse_tree.toStringTree(recog=parser))
And noticed that both '>' and '<' were assigned the same token value despite being two different tokens. Am I missing something here?
(There may be more than just these two instances, but...)
Change REL_OP and BASE_TYPE to parser rules (i.e. make them lowercase.
As you've used them, you're turning many of your intended Lexer rules, effectively into fragments.
I't important to understand that tokens are the "atoms" you have in your grammar, when you combine several of them into another Lexer rule, you just make that the token type.
(If you used grun to dump the tokens you would have seen them identified as REL_OP tokens.
With the changes below, your sample input works just fine.
grammar MLTL1
;
start: block*;
block: var_list ';' | expr ';';
var_list: IDENTIFIER (',' IDENTIFIER)* ':' type;
type: baseType | KW_SET REL_LT baseType REL_GT;
expr: expr rel_op expr | '(' expr ')' | IDENTIFIER | INT;
//// Lexical Spec
// Types
baseType: 'bool' | 'int' | 'float';
// Keywords
KW_SET: 'set';
// Op groups for precedence
rel_op: REL_EQ | REL_NEQ | REL_GT | REL_LT | REL_GTE | REL_LTE;
// Relational ops
REL_EQ: '==';
REL_NEQ: '!=';
REL_GT: '>';
REL_LT: '<';
REL_GTE: '>=';
REL_LTE: '<=';
IDENTIFIER: LETTER (LETTER | DIGIT)*;
INT: SIGN? NONZERODIGIT DIGIT* | '0';
fragment SIGN: [+-];
fragment DIGIT: [0-9];
fragment NONZERODIGIT: [1-9];
fragment LETTER: [a-zA-Z_];
COMMENT: '#' ~[\r\n]* -> skip;
WS: [ \t\r\n]+ -> channel(HIDDEN);
My ANTLR Grammar for simple expressions is as below:
This grammar works for most of the scenarios except when I try to use negative numbers.
abs(1.324) is valid
abs(-1.324) is being thrown as an error.
Or if the expression is just a negative number such as -1.344 I am having the following error in the console.
grammar ExpressionGrammar;
parse: expr EOF;
expr:
MIN expr
| expr ( MUL | DIV) expr
| expr ( ADD | MIN) expr
| expr ( MOD ) expr
| NUM
| ID
| STRING
| function
| '(' expr ')';
function: ID '(' arguments? ')';
arguments: expr ( ',' expr)*;
/* Tokens */
MUL: '*';
DIV: '/';
MIN: '-';
ADD: '+';
MOD: '%';
OPEN_PAR: '(';
CLOSE_PAR: ')';
NUM: ([0-9]*[.])?[0-9]+;
STRING: '"' ~ ["]* '"';
fragment ID_NODE: [a-zA-Z_$][a-zA-Z0-9_$]*;
ID: ID_NODE ('.' ID_NODE)*;
COMMENT: '/*' .*? '*/' -> skip;
LINE_COMMENT
: '//' ~[\r\n]* -> skip
;
WS: [ \r\t\n]+ -> skip;
The grammar seems fine to me. It could be a bug with the runtime you're using, but that seems odd to me, given you're not doing anything special.
With the Java runtime, this is what I get when parsing/lexing the input abs(-1.324):
The following tokens are produced:
ID `abs`
OPEN_PAR `(`
MIN `-`
NUM `1.324`
CLOSE_PAR `)`
EOF `<EOF>`
and the entry point parse gives:
I would like to create an Antlr parser for custom language and decided to pick a simple calculator as an example. In my new grammar it should be possible to define a string, like this:
s = "Hello, I am a string"
and handle string interpolation.
Text in double quotes enclosed in persent should be treated as interpolated, e.g.
s = "Hello, did you know that %2 + 2% is 4?"
Double percent sign should not be processed, e.g.
s = "He wants 50%% of this deal."
But at the same time my calculator should support modulus operation:
x = 5 % 2
So far, I was able to craft a Lexer/Grammar, which could switch mode and parse simple strings, here they are:
lexer grammar CalcLexer;
EQ: '=';
PLUS: '+';
MINUS: '-';
MULT: '*';
DIV: '/';
LPAREN : '(' ;
RPAREN : ')' ;
SINGLE_PERCENT_POP: '%' -> popMode;
ID : [a-zA-Z]+ ;
INT : [0-9]+ ;
OPEN_DOUBLE_QUOTE: '"' -> pushMode(STRING_MODE);
NEWLINE:'\r'? '\n' ;
WS : [ \t]+ -> skip;
mode STRING_MODE;
DOUBLE_PERCENT: '%%';
SINGLE_PERCENT: '%' -> pushMode(DEFAULT_MODE);
TEXT: ~('%'|'\n'|'"')+;
CLOSE_DOUBLE_QUOTE: '"' -> popMode;
and
parser grammar CalcGrammar;
options { tokenVocab=CalcLexer; } // use tokens from CalcLexer.g4
prog: stat+ ;
stat: expr NEWLINE
| ID EQ (expr|text) NEWLINE
| NEWLINE
;
text: OPEN_DOUBLE_QUOTE content* CLOSE_DOUBLE_QUOTE;
content: DOUBLE_PERCENT | TEXT | SINGLE_PERCENT expr SINGLE_PERCENT_POP;
expr: expr (MULT|DIV) expr
| expr (PLUS|MINUS) expr
| INT
| ID
| LPAREN expr RPAREN
;
But only thing doesn't work and I'm not sure if it ever possible to implement without custom code (members) is modulus operation:
x = 5 % 2
There is no way I can ask Anltr to check for previous mode and safely pop mode.
But I hope my understanding is wrong and there is some way to treat % sign as operator in default mode?
I have found several sources for inspiration, probably they would help you as well:
Parsing string interpolation in ANTLR
ANTLR String interpolation
Parsing String Interpolations with ANTLR4
String interpolation and lexer modes
Murphy's law for StackOverflow: you will find an answer to your own question after several minutes you post detailed question to SO.
Instead of switching to DEFAULT_MODE, I should create separate one - STRING_INTERPOLATION. This way I have to define separate tokens for this mode, which will let use % sign in normal mode (and prohibit in interpolated).
Here is Lexer and Grammar which works for me:
lexer grammar CalcLexer;
EQ: '=';
PLUS: '+';
MINUS: '-';
MULT: '*';
DIV: '/';
MOD: '%';
LPAREN : '(' ;
RPAREN : ')' ;
ID : F_ID;
INT : F_INT;
fragment F_ID: [a-zA-Z]+ ;
fragment F_INT: [0-9]+ ;
OPEN_DOUBLE_QUOTE: '"' -> pushMode(STRING_MODE);
NEWLINE:'\r'? '\n' ;
WS : [ \t]+ -> skip;
mode STRING_MODE;
DOUBLE_PERCENT: '%%';
SINGLE_PERCENT: '%' -> pushMode(STRING_INTERPOLATION);
TEXT: ~('%'|'\n'|'"')+;
CLOSE_DOUBLE_QUOTE: '"' -> popMode;
mode STRING_INTERPOLATION;
SINGLE_PERCENT_POP: '%' -> popMode;
I_PLUS: PLUS -> type(PLUS);
I_MINUS: MINUS -> type(MINUS);
I_MULT: MULT -> type(MULT);
I_DIV: DIV -> type(DIV);
I_MOD: MOD -> type(MOD);
I_LPAREN: LPAREN -> type(LPAREN);
I_RPAREN: RPAREN -> type(RPAREN);
I_ID : F_ID -> type(ID);
I_INT : F_INT -> type(INT);
WS1 : [ \t]+ -> skip;
and
parser grammar CalcGrammar;
options { tokenVocab=CalcLexer; } // use tokens from CalcLexer.g4
prog: stat+ ;
stat: expr NEWLINE
| ID EQ (expr|text) NEWLINE
| NEWLINE
;
text: OPEN_DOUBLE_QUOTE content* CLOSE_DOUBLE_QUOTE;
content: DOUBLE_PERCENT | TEXT | SINGLE_PERCENT expr SINGLE_PERCENT_POP;
expr: expr (MULT|DIV|MOD) expr
| expr (PLUS|MINUS) expr
| INT
| ID
| LPAREN expr RPAREN
;
I hope this would help someone. Probably, future me.
I am writing a parser for a scripting language, and using antlr 4.5.3 for the purpose.
grammar VSE;
chunk
: block* EOF
;
block
: var '=' exp
| functioncall
;
var
: NAME
| var '[' exp ']'
| var '.' var
;
exp
: number
| string
| var
| functioncall
| <assoc=right> exp exp //concat
;
functioncall
: NAME '(' (exp)? (',' exp)* ')'
| var '.' functioncall
;
string
: '"' (~('"' | '\\' | '\r' | '\n') | '\\' ('"' | '\\'))* '"'
;
NAME
: [a-zA-Z_][a-zA-Z_0-9]*
;
number
: INT | HEX | FLOAT
;
INT
: Digit+
;
HEX
: '0' [xX] [0-9a-fA-F]+
;
FLOAT
: Digit* '.' Digit+
;
Digit
: [0-9]
;
WS
: [ \t\u000C\r\n]+ -> skip
;
However, while testing it, I found a variable assignment like var = something followed by some function call in next line leads to a concat statement. (My concat statement is a variable followed by another like var = var1 var2) I understand that antlr is skipping ALL the new lines in favor of line continuation, but I'd like to add the condition that if there is a new line between two exps, it would treat them as two separate blocks instead of a concat statement. i.e.
var = var2
functioncall(var)
These should be two separate blocks instead of concat statement.
Is there any way to do this?
Does the following rule suitable for you?
block
: var '=' exp NEW_LINE
| functioncall NEW_LINE
;
NEW_LINE: '\r'? '\n'
WS
: [ \t]+ -> skip
;
In another case you should use Semantic Predicates or very unclear grammar.