antlr gated predicate - antlr

This is a follow-up question from Antlr superfluous Predicate required? where I stated my problem in a simplified way, however it could not be solved there.
I have the following grammar and when I delete the {true}?=> predicates, the text is not recognized anymore. The input string is MODULE main LTLSPEC H {} {} {o} FALSE;. Note that the trailing ; is not tokenized as EOC, but as IGNORE. When I add {true}?=> to the EOC rule ; is tokenized as EOC.
I tried this from command-line with antlr-v3.3 and v3.4 without difference. Thanks in advance, I appreciate your help.
grammar NusmvInput;
options {
language = Java;
}
#parser::members{
public static void main(String[] args) throws Exception {
NusmvInputLexer lexer = new NusmvInputLexer(new ANTLRStringStream("MODULE main LTLSPEC H {} {} {o} FALSE;"));
NusmvInputParser parser = new NusmvInputParser(new CommonTokenStream(lexer));
parser.specification();
}
}
#lexer::members{
private boolean inLTL = false;
}
specification :
module+ EOF
;
module :
MODULE module_decl
;
module_decl :
NAME parameter_list ;
parameter_list
: ( LP (parameter ( COMMA parameter )*)? RP )?
;
parameter
: (NAME | INTEGER )
;
/**************
*** LEXER
**************/
COMMA
:{!inLTL}?=> ','
;
OTHER
: {!inLTL}?=>( '&' | '|' | 'xor' | 'xnor' | '=' | '!' |
'<' | '>' | '-' | '+' | '*' | '/' |
'mod' | '[' | ']' | '?')
;
RCP
: {!inLTL}?=>'}'
;
LCP
: {!inLTL}?=>'{'
;
LP
: {!inLTL}?=>'('
;
RP
: {!inLTL}?=>')'
;
MODULE
: {true}?=> 'MODULE' {inLTL = false;}
;
LTLSPEC
: {true}?=> 'LTLSPEC'
{inLTL = true; skip(); }
;
EOC
: ';'
{
if (inLTL){
inLTL = false;
skip();
}
}
;
WS
: (' ' | '\t' | '\n' | '\r')+ {$channel = HIDDEN;}
;
COMMENT
: '--' .* ('\n' | '\r') {$channel = HIDDEN;}
;
INTEGER
: {!inLTL}?=> ('0'..'9')+
;
NAME
:{!inLTL}?=> ('A'..'Z' | 'a'..'z') ('a'..'z' | 'A'..'Z' | '0'..'9' | '_' | '$' | '#' | '-')*
;
IGNORE
: {inLTL}?=> . {skip();}
;

It seems that without a predicate before MODULE and LTLSPEC, the NAME gets precedence over them even if these tokens are defined before the NAME token. Whether this is by design or a bug, I don't know.
However, the way you're trying to solve it seems rather complicated. As far as I can see, you seem to want to ignore (or skip) input starting with LTLSPEC and ending with a semi colon. Why not do something like this instead:
specification : module+ EOF;
module : MODULE module_decl;
module_decl : NAME parameter_list;
parameter_list : (LP (parameter ( COMMA parameter )*)? RP)?;
parameter : (NAME | INTEGER);
MODULE : 'MODULE';
LTLSPEC : 'LTLSPEC' ~';'* ';' {skip();};
COMMA : ',';
OTHER : ( '&' | '|' | 'xor' | 'xnor' | '=' | '!' |
'<' | '>' | '-' | '+' | '*' | '/' |
'mod' | '[' | ']' | '?')
;
RCP : '}';
LCP : '{';
LP : '(';
RP : ')';
EOC : ';';
WS : (' ' | '\t' | '\n' | '\r')+ {$channel = HIDDEN;};
COMMENT : '--' .* ('\n' | '\r') {$channel = HIDDEN;};
INTEGER : ('0'..'9')+;
NAME : ('A'..'Z' | 'a'..'z') ('a'..'z' | 'A'..'Z' | '0'..'9' | '_' | '$' | '#' | '-')*;

Related

ANTLR arithmetic and comparison expressions grammer ANTLR

how to add relational operations to my code
Thanks
My code is
grammar denem1;
options {
output=AST;
}
tokens {
ROOT;
}
parse
: stat+ EOF -> ^(ROOT stat+)
;
stat
: expr ';'
;
expr
: Id Assign expr -> ^(Assign Id expr)
| add
;
add
: mult (('+' | '-')^ mult)*
;
mult
: atom (('*' | '/')^ atom)*
;
atom
: Id
| Num
| '('! expr ')' !
;
Assign : '=' ;
Comment : '//' ~('\r' | '\n')* {skip();};
Id : 'a'..'z'+;
Num : '0'..'9'+;
Space : (' ' | '\t' | '\r' | '\n')+ {skip();};
Like this:
...
expr
: Id Assign expr -> ^(Assign Id expr)
| rel
;
rel
: add (('<=' | '<' | '>=' | '>')^ add)?
;
add
: mult (('+' | '-')^ mult)*
;
...
If possible, use ANTLR v4 instead of the old v3. In v4, you can simply do this:
stat
: expr ';'
;
expr
: Id Assign expr
| '-' expr
| expr ('*' | '/') expr
| expr ('+' | '-') expr
| expr ('<=' | '<' | '>=' | '>') expr
| Id
| Num
| '(' expr ')'
;

My grammar identifiers keywords as identifiers

I'm trying to parse expressions from the Jakarta Expression Language. In summary, it is a simplified Java expressions, with addition of a few things:
Support for creating maps like: {"foo": "bar"}
Support for creating lists and sets like: [1,2,3,4] {1,2,3,4}
Use some identifiers instead of symbols, like: foo gt bar (foo > bar), foo mod bar(foo % bar), and so on.
I'm struggling in the last bit, where it always understands the "mod", "gt", "ge" as identifiers instead of using the expression that has the "%", ">", ">=".
I'm new to ANTLR. My grammar is based on the Java grammar in the https://github.com/antlr/grammars-v4/tree/master/java/java and the JavaCC provided by: https://jakarta.ee/specifications/expression-language/4.0/jakarta-expression-language-spec-4.0.html#collected-syntax
grammar ExpressionLanguageGrammar;
prog: compositeExpression;
compositeExpression: (dynamicExpression | deferredExpression | literalExpression)*;
dynamicExpression: '${' expression RCURL;
deferredExpression: '#{' expression RCURL;
literalExpression: literal;
literal: BOOL_LITERAL | FLOATING_POINT_LITERAL | INTEGER_LITERAL | StringLiteral | NULL;
mapData | listData | setData;
methodArguments: LPAREN expressionList? RPAREN;
expressionList: (expression ((COMMA expression)*));
lambdaExpressionOrCall: LPAREN lambdaExpression RPAREN methodArguments*;
lambdaExpression: lambdaParameters ARROW expression;
lambdaParameters: IDENTIFIER | (LPAREN (IDENTIFIER ((COMMA IDENTIFIER)*))? RPAREN);
mapEntry: expression COLON expression;
mapEntries: mapEntry (COMMA mapEntry)*;
expression
: primary
|'[' expressionList? ']'
| '{' expressionList? '}'
| '{' mapEntries? '}'
| expression bop='.' (IDENTIFIER | IDENTIFIER '(' expressionList? ')')
| expression ('[' expression ']')+
| prefix=('-' | '!' | NOT1 | EMPTY) expression
| expression bop=('*' | '/' | '%' | MOD1 | DIV1) expression
| expression bop=('+' | '-') expression
| expression bop=('<=' | '>=' | '>' | '<' | LE1 | GE1 | LT1 | GT1) expression
| expression bop=INSTANCEOF IDENTIFIER
| expression bop=('==' | '!=' | EQ1 | NE1) expression
| expression bop=('&&' | AND1) expression
| expression bop=('||' | OR1) expression
| <assoc=right> expression bop='?' expression bop=':' expression
| <assoc=right> expression
bop=('=' | '+=' | '-=' | '*=' | '/=')
expression
| lambdaExpression
| lambdaExpressionOrCall
;
primary
: '(' expression ')'
| literal
| IDENTIFIER
;
BOOL_LITERAL: TRUE | FALSE;
IDENTIFIER: LETTER (LETTER|DIGIT)*;
INTEGER_LITERAL: [0-9]+;
FLOATING_POINT_LITERAL: [0-9]+ '.' [0-9]* EXPONENT? | '.' [0-9]+ EXPONENT? | [0-9]+ EXPONENT?;
fragment EXPONENT: ('e'|'E') ('+'|'-')? [0-9]+;
StringLiteral: ('"' DoubleStringCharacter* '"'
| '\'' SingleStringCharacter* '\'') ;
fragment DoubleStringCharacter
: ~["\\\r\n]
| '\\' EscapeSequence
;
fragment SingleStringCharacter
: ~['\\\r\n]
| '\\' EscapeSequence
;
fragment EscapeSequence
: CharacterEscapeSequence
| '0'
| HexEscapeSequence
| UnicodeEscapeSequence
| ExtendedUnicodeEscapeSequence
;
fragment CharacterEscapeSequence
: SingleEscapeCharacter
| NonEscapeCharacter
;
fragment HexEscapeSequence
: 'x' HexDigit HexDigit
;
fragment UnicodeEscapeSequence
: 'u' HexDigit HexDigit HexDigit HexDigit
| 'u' '{' HexDigit HexDigit+ '}'
;
fragment ExtendedUnicodeEscapeSequence
: 'u' '{' HexDigit+ '}'
;
fragment SingleEscapeCharacter
: ['"\\bfnrtv]
;
fragment NonEscapeCharacter
: ~['"\\bfnrtv0-9xu\r\n]
;
fragment EscapeCharacter
: SingleEscapeCharacter
| [0-9]
| [xu]
;
fragment HexDigit
: [_0-9a-fA-F]
;
fragment DecimalIntegerLiteral
: '0'
| [1-9] [0-9_]*
;
fragment ExponentPart
: [eE] [+-]? [0-9_]+
;
fragment IdentifierPart
: IdentifierStart
| [\p{Mn}]
| [\p{Nd}]
| [\p{Pc}]
| '\u200C'
| '\u200D'
;
fragment IdentifierStart
: [\p{L}]
| [$_]
| '\\' UnicodeEscapeSequence
;
LCURL: '{';
RCURL: '}';
LETTER: '\u0024' |
'\u0041'..'\u005a' |
'\u005f' |
'\u0061'..'\u007a' |
'\u00c0'..'\u00d6' |
'\u00d8'..'\u00f6' |
'\u00f8'..'\u00ff' |
'\u0100'..'\u1fff' |
'\u3040'..'\u318f' |
'\u3300'..'\u337f' |
'\u3400'..'\u3d2d' |
'\u4e00'..'\u9fff' |
'\uf900'..'\ufaff';
DIGIT: '\u0030'..'\u0039'|
'\u0660'..'\u0669'|
'\u06f0'..'\u06f9'|
'\u0966'..'\u096f'|
'\u09e6'..'\u09ef'|
'\u0a66'..'\u0a6f'|
'\u0ae6'..'\u0aef'|
'\u0b66'..'\u0b6f'|
'\u0be7'..'\u0bef'|
'\u0c66'..'\u0c6f'|
'\u0ce6'..'\u0cef'|
'\u0d66'..'\u0d6f'|
'\u0e50'..'\u0e59'|
'\u0ed0'..'\u0ed9'|
'\u1040'..'\u1049';
TRUE: 'true';
FALSE: 'false';
NULL: 'null';
DOT: '.';
LPAREN: '(';
RPAREN: ')';
LBRACK: '[';
RBRACK: ']';
COLON: ':';
COMMA: ',';
SEMICOLON: ';';
GT0: '>';
GT1: 'gt';
LT0: '<';
LT1: 'lt';
GE0: '>=';
GE1: 'ge';
LE0: '<=';
LE1: 'le';
EQ0: '==';
EQ1: 'eq';
NE0: '!=';
NE1: 'ne';
NOT0: '!';
NOT1: 'not';
AND0: '&&';
AND1: 'and';
OR0: '||';
OR1: 'or';
EMPTY: 'empty';
INSTANCEOF: 'instanceof';
MULT: '*';
PLUS: '+';
MINUS: '-';
QUESTIONMARK: '?';
DIV0: '/';
DIV1: 'div';
MOD0: '%';
MOD1: 'mod';
CONCAT: '+=';
ASSIGN: '=';
ARROW: '->';
DOLLAR: '$';
HASH: '#';
WS: [ \t\r\n]+ -> skip;
Move the Lexer rules for them to be prior to the Lexer rule for Identifier.
If ANTLR has more than one Lexer rule that matches input of the same length it chooses the first rule in the grammar that matches.
For example “mod” is matched by Identifier and MOD1, but Identifier is 1st, so it chooses Identifier. Move the MOD1 rule to be before Identifier and it’ll match MOD1
———-
BTW, unless you care about having different token values for “%” and “mod”, you can just define a single rule:
MOD: ‘%’ | ‘mod’;
You’d can still get the token text if you need it but it will you can just specify MOD in your parser rules instead of (MOD0 | MOD1)

Parsing DECAF grammar in ANTLR

I am creating a the parser for DECAF with Antlr
grammar DECAF ;
//********* LEXER ******************
LETTER: ('a'..'z'|'A'..'Z') ;
DIGIT : '0'..'9' ;
ID : LETTER( LETTER | DIGIT)* ;
NUM: DIGIT(DIGIT)* ;
COMMENTS: '//' ~('\r' | '\n' )* -> channel(HIDDEN);
WS : [ \t\r\n\f | ' '| '\r' | '\n' | '\t']+ ->channel(HIDDEN);
CHAR: (LETTER|DIGIT|' '| '!' | '"' | '#' | '$' | '%' | '&' | '\'' | '(' | ')' | '*' | '+'
| ',' | '-' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '#' | '[' | '\\' | ']' | '^' | '_' | '`'| '{' | '|' | '}' | '~'
'\t'| '\n' | '\"' | '\'');
// ********** PARSER *****************
program : 'class' 'Program' '{' (declaration)* '}' ;
declaration: structDeclaration| varDeclaration | methodDeclaration ;
varDeclaration: varType ID ';' | varType ID '[' NUM ']' ';' ;
structDeclaration : 'struct' ID '{' (varDeclaration)* '}' ;
varType: 'int' | 'char' | 'boolean' | 'struct' ID | structDeclaration | 'void' ;
methodDeclaration : methodType ID '(' (parameter (',' parameter)*)* ')' block ;
methodType : 'int' | 'char' | 'boolean' | 'void' ;
parameter : parameterType ID | parameterType ID '[' ']' ;
parameterType: 'int' | 'char' | 'boolean' ;
block : '{' (varDeclaration)* (statement)* '}' ;
statement : 'if' '(' expression ')' block ( 'else' block )?
| 'while' '(' expression ')' block
|'return' expressionA ';'
| methodCall ';'
| block
| location '=' expression
| (expression)? ';' ;
expressionA: expression | ;
location : (ID|ID '[' expression ']') ('.' location)? ;
expression : location | methodCall | literal | expression op expression | '-' expression | '!' expression | '('expression')' ;
methodCall : ID '(' arg1 ')' ;
arg1 : arg2 | ;
arg2 : (arg) (',' arg)* ;
arg : expression;
op: arith_op | rel_op | eq_op | cond_op ;
arith_op : '+' | '-' | '*' | '/' | '%' ;
rel_op : '<' | '>' | '<=' | '>=' ;
eq_op : '==' | '!=' ;
cond_op : '&&' | '||' ;
literal : int_literal | char_literal | bool_literal ;
int_literal : NUM ;
char_literal : '\'' CHAR '\'' ;
bool_literal : 'true' | 'false' ;
When I give it the input:
class Program {
void main(){
return 3+5 ;
}
}
The parse tree is not building correctly since it is not recognizing the 3+5 as an expression. Is there anything wrong with my grammar that is causing the problem?
Lexer rules are matched from top to bottom. When 2 or more lexer rules match the same amount of characters, the one defined first will win. Because of that, a single digit integer will get matched as a DIGIT instead of a NUM.
Try parsing the following instead:
class Program {
void main(){
return 33 + 55 ;
}
}
which will be parsed just fine. This is because 33 and 55 are matched as NUMs, because NUM can now match 2 characters (DIGIT only 1, so NUM wins).
To fix it, make DIGIT a fragment (and LETTER as well):
fragment LETTER: ('a'..'z'|'A'..'Z') ;
fragment DIGIT : '0'..'9' ;
ID : LETTER( LETTER | DIGIT)* ;
NUM: DIGIT(DIGIT)* ;
Lexer fragments are only used internally by other lexer rules, and will never become tokens of their own.
A couple of other things: your WS rule matches way too much (it now also matches a | and a '), it should be:
WS : [ \t\r\n\f]+ ->channel(HIDDEN);
and you shouldn't match a char literal in your parser: do it in the lexer:
CHAR : '\'' ( ~['\r\n\\] | '\\' ['\\] ) '\'';
If you don't, the following will not get parsed properly:
class Program {
void main(){
return '1';
}
}
because the 1 wil be tokenized as a NUM and not as a CHAR.

ANTLR - Field that accept attributes with more than one word

My Grammar file (see below) parses queries of the type:
(name = Jon AND age != 16 OR city = NY);
However, it doesn't allow something like:
(name = 'Jon Smith' AND age != 16);
ie, it doesn't allow assign to a field values with more than one word, separated by White Spaces. How can I modify my grammar file to accept that?
options
{
language = Java;
output = AST;
}
tokens {
BLOCK;
RETURN;
QUERY;
ASSIGNMENT;
INDEXES;
}
#parser::header {
package pt.ptinovacao.agorang.antlr;
}
#lexer::header {
package pt.ptinovacao.agorang.antlr;
}
query
: expr ('ORDER BY' NAME AD)? ';' EOF
-> ^(QUERY expr ^('ORDER BY' NAME AD)?)
;
expr
: logical_expr
;
logical_expr
: equality_expr (logical_op^ equality_expr)*
;
equality_expr
: NAME equality_op atom -> ^(equality_op NAME atom)
| '(' expr ')' -> ^('(' expr)
;
atom
: ID
| id_list
| Int
| Number
;
id_list
: '(' ID (',' ID)* ')'
-> ID+
;
NAME
: 'equipType'
| 'equipment'
| 'IP'
| 'site'
| 'managedDomain'
| 'adminState'
| 'dataType'
;
AD : 'ASC' | 'DESC' ;
equality_op
: '='
| '!='
| 'IN'
| 'NOT IN'
;
logical_op
: 'AND'
| 'OR'
;
Number
: Int ('.' Digit*)?
;
ID
: ('a'..'z' | 'A'..'Z' | '_' | '.' | '-' | Digit)*
;
String
#after {
setText(getText().substring(1, getText().length()-1).replaceAll("\\\\(.)", "$1"));
}
: '"' (~('"' | '\\') | '\\' ('\\' | '"'))* '"'
| '\'' (~('\'' | '\\') | '\\' ('\\' | '\''))* '\''
;
Comment
: '//' ~('\r' | '\n')* {skip();}
| '/*' .* '*/' {skip();}
;
Space
: (' ' | '\t' | '\r' | '\n' | '\u000C') {skip();}
;
fragment Int
: '1'..'9' Digit*
| '0'
;
fragment Digit
: '0'..'9'
;
indexes
: ('[' expr ']')+ -> ^(INDEXES expr+)
;
Include the String token as an alternative in your atom rule:
atom
: ID
| id_list
| Int
| Number
| String
;

Small grammar that doesn't work; what am I missing (antlr4)

I have the following grammar. It's supposed to accept the string shown in the comments in the header. It does not. I must be missing something fundamental. Hints on how to debug this would also be appreciated.
/*
Should accept:
b
a:b
a:b^10
b^10
Should not accept:
:b
a:
a:^10
*/
grammar test;
filter:
boostedField EOF
;
boostedField
: qualifiedField (CARET NUMBER)?
;
qualifiedField
: (FIELDNAME COLON)? term
;
term
: TERM
;
FIELDNAME: (LETTER | UNDERSCORE) (ALPHANUM | UNDERSCORE)* ;
NUMBER : NUM_CHAR+ ('.' NUM_CHAR+)? ;
COLON : ':' ;
CARET : '^' ;
WS : (' ' | '\t' | '\n' | '\r' | '\u3000') -> skip ;
UNDERSCORE: '_' ;
// a term may not have a colon or a caret (unless escaped)
TERM : TERM_START_CHAR TERM_CHAR*;
fragment TERM_START_CHAR
: ~( ' ' | '\t' | '\n' | '\r' | '\u3000' | ':' | '^' ) ;
fragment TERM_CHAR : (TERM_START_CHAR | ESCAPED_CHAR) ;
fragment ESCAPED_CHAR : ( '\\' . ) ;
fragment NUM_CHAR: '0'..'9';
fragment LETTER: 'a'..'z' | 'A'..'Z' ;
fragment ALPHANUM: LETTER | NUM_CHAR;