I am new to writing Antlr grammars and I'm having some trouble. I have simplified my issue down to the following:
Here is my grammar:
grammar Test;
prog : stmt* ;
stmt : INT NEWLINE ;
INT : [0-9]+ ;
NEWLINE : '\r'? '\n' ;
My input.txt is simply 0\r\n (not the literals '\', 'r', '\', 'n', but the ASCII codes 0x0D 0x0A
I get the following error:
line 1:1 extraneous input '\r\n' expecting {<EOF>, INT}
Why does it expect another INT without first matching a NEWLINE when the stmt rule requires that the INT be followed by a NEWLINE?
If it's relevant, I'm using PyCharm and the Antlr4 plugin to do the grammar generation.
As it turns out, the issue was related to Python's odd behavior when running modules that import modules that have been altered (hint: they're not automatically reloaded as they should be). The solution was to simply restart the interpreter when you change the grammar.
I'm sorry that the original question didn't have enough information for anyone else to figure out that answer.
Related
Surprise, I am building an SQL like language parser for a project.
I had it mostly working, but when I started testing it against real requests it would be handling, I realized it was behaving differently on the inside than I thought.
The main issue in the following grammar is that I define a lexer rule PCT_WITHIN for the language keyword 'pct_within'. This works fine, but if I try to match a field like 'attributes.pct_vac', I get the field having text of 'attributes.ac' and a pretty ANTLR error of:
line 1:15 mismatched character u'v' expecting 'c'
GRAMMAR
grammar Select;
options {
language=Python;
}
eval returns [value]
: field EOF
;
field returns [value]
: fieldsegments {print $field.text}
;
fieldsegments
: fieldsegment (DOT (fieldsegment))*
;
fieldsegment
: ICHAR+ (USCORE ICHAR+)*
;
WS : ('\t' | ' ' | '\r' | '\n')+ {self.skip();};
ICHAR : ('a'..'z'|'A'..'Z');
PCT_CONTAINS : 'pct_contains';
USCORE : '_';
DOT : '.';
I have been reading everything I can find on the topic. How the Lexer consumes stuff as it finds it even if it is wrong. How you can use semantic predication to remove ambiguity/how to use lookahead. But everything I read hasn't helped me fix this issue.
Honestly I don't see how it even CAN be an issue. I must be missing something super obvious because other grammars I see have Lexer rules like EXISTS but that doesn't cause the parser to take a string like 'existsOrNot' and spit out and IDENTIFIER with the text of 'rNot'.
What am I missing or doing completely wrong?
Convert your fieldsegment parser rule into a lexer rule. As it stands now it will accept input like
"abc
_ abc"
which is probably not what you want. The keyword "pct_contains" won't be matched by this rule since it is defined separately. If you want to accept the keyword in certain sequences as regular identifier you will have to include it in the accepted identifier rule.
I am writing an ANTLR parser that attempts to recognize GDB backtrace output from a given input string.
I'm ignoring new lines with the following lexer grammar:
RETURN : ('\r' | '\n' | '\r\n') { skip(); };
However, when I run the parser against some input, ANTLR gives the following lexer error:
line 20:21 no viable alternative at character '\n'
line 23:14 no viable alternative at character '\n'
line 30:21 no viable alternative at character '\n'
line 33:31 no viable alternative at character '\n'
I am not sure why this would ever happen, since I have already specified '\n' in the lexer.
Does anybody has any ideas? Thanks.
It looks like the problem is elsewhere in your grammar: it is still lexing a different element that has not yet ended, and unexpectedly encountered the end of line while it was still expecting to finish the current element.
I have a grammar that accepts key / value pairs that appear one per line. The values may be multi-line.
The Eclipse plug-in ANTLR IDE works correctly and accepts a valid test string. However, the generated Java does not accept the same string.
Here is the grammar:
message: block4 ;
block4: STARTBLOCK '4' COLON expr4+ ENDBLOCK ;
expr4: NEWLINE (COLON key COLON expr | '-')+;
key: FIELDVALUE* ;
expr: FIELDVALUE* ;
NEWLINE : ('\n'|'\r') ;
FIELDVALUE : (~('-'|COLON|ENDBLOCK|STARTBLOCK))+;
COLON : ':' ;
STARTBLOCK : '{' ;
ENDBLOCK : '}' ;
ANTLR IDE parses this correctly:
Don't squint... It is dividing up key/expression pairs whether they are single-line values (like 23B / CRED) or multiline values (like 59 / /13212312\r\nRECEIVER NAME S.A\r\n).
Here is the input string:
{4:
:20:007505327853
:23B:CRED
:32A:050902JPY3520000,
:33B:JPY3520000,
:50K:EUROXXXEI
:52A:FEBXXXM1
:53A:MHCXXXJT
:54A:FOOBICXX
:59:/13212312
RECEIVER NAME S.A
:70:FUTURES
:71A:SHA
:71F:EUR12,00
:71F:EUR2,34
-}
When Eclipse runs anltr-3.4-complete.jar on the grammar, it generates SwiftTinyLexer.java and SwiftTinyParser.java. The lexer lexes them into 35 tokens, starting with:
STARTBLOCK
4
COLON
FIELDVALUE
COLON
I would like token 4 to be an expr4 rather than a FIELDVALUE (and the IDE seems to agree with me). But since it is a FIELDVALUE, the parser is choking on that token with line 1:3 required (...)+ loop did not match anything at input '\r\n'.
Why is there a difference between the way that anltr 3.4 and ANTLR IDE 2.1.2.201108281759 lex the same string?
Is there a way to fix the grammar so that it matches expr4 before it matches FIELDVALUE?
The IDE input string has a single \n while the Java test code is getting a Windows-style \r\n.
I changed NEWLINE by adding a "1 or more," that is from
NEWLINE : ('\n'|'\r') ;
to
NEWLINE : ('\n'|'\r')+ ;
This allowed the parse go forward without the lexical error, and now it makes sense why the IDE behaved differently from generated Java: They were getting slightly different input strings.
I want to parse a language in which statements are separated by EOLs. I tried this in the lexer grammar (copied from an example in the docs):
EOL : ('\r'? '\n')+ ; // any number of consecutive linefeeds counts as a single EOL
and then used this in the parser grammar:
stmt_sequence : (stmt EOL)* ;
The parser rejected code with statements separated by one or more blank lines.
However, this was successful:
EOL : '\r'? '\n' ;
stmt_sequence : (stmt EOL+)* ;
I'm an ANTLR newbie. It seems like both should work. Is there something about greedy/nongreedy lexer scanning that I don't understand?
I tried this with both 3.2 and 3.4; I'm running the ANTLR IDE in Eclipse Indigo on OS X 10.6.
Thanks.
The error was not in the original grammar; but in the input data. I was using an editor (in Eclipse) that automatically inserted tabs after an EOL, so my "blank lines" were not really blank.
I modified the grammar as follows:
fragment SPACE: ' ' | '\t';
EOL : ( '\r'? '\n' SPACE* )+;
This grammar works as expected.
The lesson here is that one must be careful with white spaces. The lexer may see white spaces in the input that the parser does not see (because it has already been sent to the hidden channel).
My question is in regards to running the following grammar in ANTLRWorks:
INT :('0'..'9')+;
SEMICOLON: ';';
NEWLINE: ('\r\n'|'\n'|'\r');
STMTEND: (SEMICOLON (NEWLINE)*|NEWLINE+);
statement
: STMTEND
| INT STMTEND
;
program: statement+;
I get the following results with the following input (with program as the start rule), regardless of which newline NL (CR/LF/CRLF) or integer I choose:
"; NL" or "32; NL" parses without error.
";" or "45;" (without newlines) result in EarlyExitException.
"NL" by itself parses without error.
"456 NL", without the semicolon, results in MismatchedTokenException.
What I want is for a statement to be terminated by a newline, semicolon, or semicolon followed by newline, and I want the parser to eat as many contiguous newlines as it can on a termination, so "; NL NL NL NL" is just one termination, not four or five. Also, I would like the end-of-file case to be a valid termination as well, but I don't know how to do that yet.
So what's wrong with this, and how can I make this terminate nicely at EOF? I'm completely new to all of parsing, ANTLR, and EBNF, and I haven't found much material to read on it at a level somewhere in between the simple calculator example and the reference (I have The Definitive ANTLR Reference, but it really is a reference, with a quick start in the front which I haven't yet got to run outside of ANTLRWorks), so any reading suggestions (besides Wirth's 1977 ACM paper) would be helpful too. Thanks!
In case of input like ";" or "45;", the token STMTEND will never be created.
";" will create a single token: SEMICOLON, and "45;" will produce: INT SEMICOLON.
What you (probably) want is that SEMICOLON and NEWLINE never make it to real tokens themselves, but they will always be a STMTEND. You can do that by making them so called "fragment" rules:
program: statement+;
statement
: STMTEND
| INT STMTEND
;
INT : '0'..'9'+;
STMTEND : SEMICOLON NEWLINE* | NEWLINE+;
fragment SEMICOLON : ';';
fragment NEWLINE : '\r' '\n' | '\n' | '\r';
Fragment rules are only available for other lexer rules, so they will never end up in parser (production) rules. To emphasize: the grammar above will only ever create either INT or STMTEND tokens.