I'm using ANTLR4 to parse text adventure game dialogue files written in Yarn, so mostly free form text and loads of island grammars, and for the most part things are going smoothly but I am having an issue excluding certain text inside the Shortcut mode (when presenting options for the player to choose from).
Basically I need to write a rule to match anything except #, newline or <<. When it hits a << it needs to move into a new mode for handling expressions of various kinds or to just leave the current mode so that the << will get picked up by the already existing rules.
A cut down version of my lexer (ignoring rules for expressions):
lexer grammar YarnLexer;
NEWLINE : ('\n') -> skip;
CMD : '<<' -> pushMode(Command);
SHORTCUT : '->' -> pushMode(Shortcut);
HASHTAG : '#' ;
LINE_GOBBLE : . -> more, pushMode(Line);
mode Line;
LINE : ~('\n'|'#')* -> popMode;
mode Shortcut ;
TEXT : CHAR+ -> popMode;
fragment CHAR : ~('#'|'\n'|'<');
mode Command ;
CMD_EXIT : '>>' -> popMode;
// RULES FOR OPERATORS/IDs/NUMBERS/KEYWORDS/etc
CMD_TEXT : ~('>')+ ;
And the parser grammar (again ignoring all the rules for expressions):
parser grammar YarnParser;
options { tokenVocab=YarnLexer; }
dialogue: statement+ EOF;
statement : line_statement | shortcut_statement | command_statement ;
hashtag : HASHTAG LINE ;
line_statement : LINE hashtag? ;
shortcut_statement : SHORTCUT TEXT command_statement? hashtag?;
command_statement : CMD expression CMD_EXIT;
expression : CMD_TEXT ;
I have tested the Command mode when it is by itself and everything inside there is working fine, but when I try to parse my example input:
Where should we go?
-> the park
-> the zoo
-> Peter's house <<if $metPeter == true >>
ok shall we take the bus?
-> :<
-> ok
<<set $daySpent = true>>
my issue is the line:
-> Peter's house <<if $metPeter == true >>
gets matched completely as TEXT and the CMD rules just gets ignored in favour by far longer TEXT.
My first thought was to add < to the set but then I can't have text like:
-> :<
which should be perfectly valid. Any idea how to do this?
Adding a single left angle bracket to the exclusion list creates a single corner case that is easily handled:
TEXT : CHAR+ ;
CMD : '<<' -> pushMode(Command);
LAB : '<' -> type(TEXT) ;
fragment CHAR : ~('\n' | '#' | '<') ;
Related
I'm trying to make the grammar for a rather simple language using ANTLR4. It's supposed to process some theater-related text. There are just 3 rules.
1 - Any text that starts with a tab (\t), should be just printed out.
It was a rather warm
Summer day.
2 - In case the text doesn't start with a tab, it'll most likely contain a character name. For example:
Captain Go forth, my minions!
It would be perfect to grab character name and text they're saying separately.
3 - And there are commands, that also start with a tab, followed by a keyword and some arguments, kind of like this:
lights ON
curtain OPEN
This is my grammar:
grammar Theater;
module: statement+ EOF;
statement: function | print | print_with_name;
function: '\t' command NL;
command: lights | curtain;
lights: 'lights' WS ('ON' | 'OFF');
curtain: 'curtain' WS ('OPEN' | 'CLOSE');
print: PRINT;
PRINT: '\t' .*? NL NL;
print_with_name: PRINT_WITH_NAME;
PRINT_WITH_NAME: ~[ \t\r\n] .*? NL NL;
NL: '\r\n' | '\r' | '\n';
WS: [ \t]+?;
I run this on the following test file:
It was a rather warm
Summer day.
Captain Go forth, my minions!
lights ON
curtain OPEN
And these are tokens I get:
[#0,0:22='\tIt was a rather warm\r\n',<PRINT>,1:0]
[#1,23:36='\tSummer day.\r\n',<PRINT>,2:0]
[#2,37:67='Captain Go forth, my minions!\r\n',<PRINT_WITH_NAME>,3:0]
[#3,68:79='\tlights ON\r\n',<PRINT>,4:0]
[#4,80:94='\tcurtain OPEN\r\n',<PRINT>,5:0]
[#5,95:94='<EOF>',<EOF>,6:0]
print and print with name both work as expected. Commands, on the other hand, are being treated as print. I guess, this is because those are lexer rules, but commands are parser rules.
Is there any way I can make it work without converting all commands to lexer rules? I tried hard to write something like "treat all text as Print, except when it starts with one of the keywords". But couldn't really find anything that would work. I'm only starting with antlr, so I must be missing something.
I don't expect you to write the grammar for me. Just mentionion a feature I should use would be perfect.
Lexer modes can be helpful here, which is a way to nudge the lexer in the right direction (make it a bit context sensitive).
To use lexer modes, you must divide the lexer- and parser-grammar into separate files. Here is TheaterLexer.g4:
lexer grammar TheaterLexer;
Name : ~[ \t]+ -> mode(DialogMode);
K_Lights : '\tlights' -> mode(CommandMode);
K_Curtain : '\tcurtain' -> mode(CommandMode);
Tab : '\t' -> skip, mode(TabMode);
mode DialogMode;
DialogText : ~[\r\n]+;
DialogNewLine : [\r\n]+ -> skip, mode(DEFAULT_MODE);
mode CommandMode;
CommandText : ~[\r\n]+;
CommandNewLine : [\r\n]+ -> skip, mode(DEFAULT_MODE);
mode TabMode;
LiteralText : ~[\r\n]+;
LiteralNewLine : [\r\n]+ -> skip, mode(DEFAULT_MODE);
And the parser part (put it in TheaterParser.g4):
parser grammar TheaterParser;
options { tokenVocab=TheaterLexer; }
parse
: file EOF
;
file
: atom*
;
atom
: literal
| dialog
| command
;
literal
: LiteralText+
;
dialog
: Name DialogText+
;
command
: K_Lights CommandText+
| K_Curtain CommandText+
;
If you now generate the lexer and parser classes and run the following Java code:
String source =
"\tIt was a rather warm\n" +
"\tSummer day.\n" +
"Captain Go forth, my minions!\n" +
"\tlights ON\n" +
"\tcurtain OPEN";
TheaterLexer lexer = new TheaterLexer(CharStreams.fromString(source));
TheaterParser parser = new TheaterParser(new CommonTokenStream(lexer));
ParseTree root = parser.parse();
System.out.println(root.toStringTree(parser));
the following will be printed to your console:
(parse
(file
(atom
(literal It was a rather warm Summer day.))
(atom
(dialog Captain Go forth, my minions!))
(atom
(command \tlights ON))
(atom
(command \tcurtain OPEN))) <EOF>)
(the indentation is added for readability)
Note that you can use just a single mode, but I assumed you'd want to treat the tokens differently in the different modes. If this is not the case, you could just do:
lexer grammar TheaterLexer;
Name : ~[ \t]+ -> mode(Step2Mode);
K_Lights : '\tlights' -> mode(Step2Mode);
K_Curtain : '\tcurtain' -> mode(Step2Mode);
Tab : '\t' -> skip, mode(Step2Mode);
mode Step2Mode;
Text : ~[\r\n]+;
NewLine : [\r\n]+ -> skip, mode(DEFAULT_MODE);
and change the parser rules accordingly.
I have a grammar that uses modes to do string interpolation:
Something along the lines of:
lexer grammar Example;
//default mode tokens
LBRACE: '{' -> pushMode(DEFAULT_MODE);
RBRACE: '}' -> popMode;
OPEN_STRING: '"' -> pushMode(STRING);
mode STRING;
ID_INTERPOLATION: '$' IDEN;
OPEN_EXPR_INTERPOLATION: '${' -> pushMode(DEFAULT_MODE);
TEXT: '$' | (~[$\r\n])+;
CLOSE_STRING: '"' -> popMode;
parser grammar ExampleParser;
options {tokenVocab = Example;}
test: string* EOF;
string: OPEN_STRING string_part* CLOSE_STRING;
string_part: TEXT | ID_INTERPOLATION | OPEN_EXPR_INTERPOLATION expr RBRACE;
//more rules that use LBRACE and RBRACE
Now this works and tokenizes everything mostly how I want it, but it does have 2 flaws.
if the number of RBRACES goes too far, it can pop the first default mode which can glitch out the IDE, and does not just show an error.
The token for closing a block and closing interpolation is the same, so I cannot highlight them however I want. (this is the main one)
My IDE highlights based on tokens only, so this is a problem, I'd like to be able to highlight them differently. So basically I'd like a solution for this that makes the RBRACE a different token when it's in a string.
I'd prefer to do it without semantic predicates because I don't want to tie it down to a language, but if needed, I'm ok with it, I just might need a little more explanation because I haven't used them that much.
Thank you #sepp2k for helping me solve my issue.
It's a bit of a hack but it does exactly what I need it to
I solved it by changing my popMode on RBRACE to be the following:
RBRACE: '}' {
if(_modeStack.size() > 0) {
popMode();
if(_mode != DEFAULT_MODE) {
setType(EXPR_INTERPOLATION);
}
}
};
I also changed my parser to be
string_part: TEXT | ID_INTERPOLATION | EXPR_INTERPOLATION expr EXPR_INTERPOLATION;
I know it's pretty hacky to change the token type under a specific circumstance, but it got the job done for me, so I'm gonna keep it unless I find a less hacky way to do this.
So I set out to implement an interpolated string parser with using only ANTLR code (no host language code blocks). I found that this works well, including nesting interpolated strings...
lexer grammar Lexer;
LeftBrace: '{';
RightBrace: '}' -> popMode;
Backtick: '`' -> pushMode(InterpolatedString);
Integer: [0-9]+;
Plus: '+';
mode InterpolatedString;
EscapedLeftBrace: '\\{' -> type(Grapheme);
EscapedBacktick: '\\`' -> type(Grapheme);
ExprStart: '{' -> type(LeftBrace), pushMode(DEFAULT_MODE);
End: '`' -> type(Backtick), popMode;
Grapheme: ~('{' | '`');
parser grammar Parser;
options {
tokenVocab = Lexer;
}
startRule: expression EOF;
interpolatedString: Backtick (Grapheme | interpolatedStringExpression)* Backtick;
interpolatedStringExpression: LeftBrace expression RightBrace;
expression
: expression Plus expression
| atom
;
atom: Integer | interpolatedString;
You can test it with input
`{`{`{`{`{`{`{`hello world`}`}`}`}`}`}`}`
I have strings as a parser rule rather than lexer because strings may contain escapes with expressions in them, such as "The variable is \(variable)".
string
: '"' character* '"'
;
character
: escapeSequence
| .
;
escapeSequence
: '\(' expression ')'
;
IDENTIFIER
: [a-zA-Z][a-zA-Z0-9]*
;
WHITESPACE
: [ \r\t,] -> skip
;
This doesn't work because . matches any token rather than any character, so many identifiers will be matched and whitespace will be completely ignored.
How can I parse strings that can have expressions inside of them?
Looking into the parser for Swift and Javascript, both languages that support things like this, I can't figure out how they work. From what I can tell, they just output a string such as "my string with (variables) in it" without actually being able to parse the variable as its own thing.
This problem can be approached using lexical modes by having one mode for the inside of strings and one (or more) for the outside. Seeing a " on the outside would switch to the inside mode and seeing a \( or " would switch back outside. The only complicated part would be seeing a ) on the outside: Sometimes it should switch back to the inside (because it corresponds to a \() and some times it shouldn't (when it corresponds to a plain ().
The most basic way to achieve this would be like this:
Lexer:
lexer grammar StringLexer;
IDENTIFIER: [a-zA-Z_][a-zA-Z0-9_]* ;
DQUOTE: '"' -> pushMode(IN_STRING);
LPAR: '(' -> pushMode(DEFAULT_MODE);
RPAR: ')' -> popMode;
mode IN_STRING;
TEXT: ~[\\"]+ ;
BACKSLASH_PAREN: '\\(' -> pushMode(DEFAULT_MODE);
ESCAPE_SEQUENCE: '\\' . ;
DQUOTE_IN_STRING: '"' -> type(DQUOTE), popMode;
Parser:
parser grammar StringParser;
options {
tokenVocab = 'StringLexer';
}
start: exp EOF ;
exp : '(' exp ')'
| IDENTIFIER
| DQUOTE stringContents* DQUOTE
;
stringContents : TEXT
| ESCAPE_SEQUENCE
| '\\(' exp ')'
;
Here we push the default mode every time we see a ( or \( and pop the mode every time we see a ). This way it will go back inside the string only if the mode on top of the stack is the string mode, which would only be the case if there aren't any unclosed ( left since the last \(.
This approach works, but has the downside that an unmatched ) will cause an empty stack exception rather than a normal syntax error because we're calling popMode on an empty stack.
To avoid this, we can add a member that tracks how deeply nested we are inside parentheses and doesn't pop the stack when the nesting level is 0 (i.e. if the stack is empty):
#members {
int nesting = 0;
}
LPAR: '(' {
nesting++;
pushMode(DEFAULT_MODE);
};
RPAR: ')' {
if (nesting > 0) {
nesting--;
popMode();
}
};
mode IN_STRING;
BACKSLASH_PAREN: '\\(' {
nesting++;
pushMode(DEFAULT_MODE);
};
(The parts I left out are the same as in the previous version).
This works and produces normal syntax errors for unmatched )s. However, it contains actions and is thus no longer language-agnostic, which is only a problem if you plan to use the grammar from multiple languages (and depending on the language, you might even be lucky and the code might be valid in all of your targeted languages).
If you want to avoid actions, the last alternative would be to have three modes: One for code that's outside of any strings, one for the inside of the string and one for the inside of \(). The third one will be almost identical to the outer one, except that it will push and pop the mode when seeing parentheses, whereas the outer one will not. To make both modes produce the same types of tokens, the rules in the third mode will all call type(). This will look like this:
lexer grammar StringLexer;
IDENTIFIER: [a-zA-Z_][a-zA-Z0-9_]* ;
DQUOTE: '"' -> pushMode(IN_STRING);
LPAR: '(';
RPAR: ')';
mode IN_STRING;
TEXT: ~[\\"]+ ;
BACKSLASH_PAREN: '\\(' -> pushMode(EMBEDDED);
ESCAPE_SEQUENCE: '\\' . ;
DQUOTE_IN_STRING: '"' -> type(DQUOTE), popMode;
mode EMBEDDED;
E_IDENTIFIER: [a-zA-Z_][a-zA-Z0-9_]* -> type(IDENTIFIER);
E_DQUOTE: '"' -> pushMode(IN_STRING), type(DQUOTE);
E_LPAR: '(' -> type(LPAR), pushMode(EMBEDDED);
E_RPAR: ')' -> type(RPAR), popMode;
Note that we now can no longer use string literals in the parser grammar because string literals can't be used when multiple lexer rules are defined using the same string literal. So now we have to use LPAR instead of '(' in the parser and so on (we already had to do this for DQUOTE for the same reason).
Since this version involves a lot of duplication (especially as the amount of tokens rises) and prevents the use of string literals in the parser grammar, I generally prefer the version with the actions.
The full code for all three alternatives can also be found on GitHub.
I am trying to write a grammar that will recognize <<word>> as a special token but treat <word> as just a regular literal.
Here is my grammar:
grammar test;
doc: item+ ;
item: func | atom ;
func: '<<' WORD '>>' ;
atom: PUNCT+ #punctAtom
| NEWLINE+ #newlineAtom
| WORD #wordAtom
;
WS : [ \t] -> skip ;
NEWLINE : [\n\r]+ ;
PUNCT : [.,?!]+ ;
WORD : CHAR+ ;
fragment CHAR : (LETTER | DIGIT | SYMB | PUNCT) ;
fragment LETTER : [a-zA-Z] ;
fragment DIGIT : [0-9] ;
fragment SYMB : ~[a-zA-Z0-9.,?! |{}\n\r\t] ;
So something like <<word>> will be matched by two rules, both func and atom. I want it to be recognized as a func, so I put the func rule first.
When I test my grammar with <word> it treats it as an atom, as expected. However when I test my grammar and give it <<word>> it treats it as an atom as well.
Is there something I'm missing?
PS - I have separated atom into PUNCT, NEWLINE, and WORD and given them labels #punctAtom, #newlineAtom, and #wordAtom because I want to treat each of those differently when I traverse the parse tree. Also, a WORD can contain PUNCT because, for instance, someone can write "Hello," and I want to treat that as a single word (for simplicity later on).
PPS - One thing I've tried is I've included < and > in the last rule, which is a list of symbols that I'm "disallowing" to exist inside a WORD. This solves one problem, in that <<word>> is now recognized as a func, but it creates a new problem because <word> is no longer accepted as an atom.
ANTLR's lexer tries to match as much characters as possible, so both <<WORD>> and <WORD> are matched by the lexer rul WORD. Therefor, there in these cases the tokens << and >> (or < and > for that matter) will not be created.
You can see what tokens are being created by running these lines of code:
Lexer lexer = new testLexer(CharStreams.fromString("<word> <<word>>"));
CommonTokenStream tokens = new CommonTokenStream(lexer);
tokens.fill();
for (Token t : tokens.getTokens()) {
System.out.printf("%-20s %s\n", testLexer.VOCABULARY.getSymbolicName(t.getType()), t.getText());
}
which will print:
WORD <word>
WORD <<word>>
EOF <EOF>
What you could do is something like this:
func
: '<<' WORD '>>'
;
atom
: PUNCT+ #punctAtom
| NEWLINE+ #newlineAtom
| word #wordAtom
;
word
: WORD
| '<' WORD '>'
;
...
fragment SYMB : ~[<>a-zA-Z0-9.,?! |{}\n\r\t] ;
Of course, something like foo<bar will not become a single WORD, which it previously would.
I'm using antlr4 and I'm trying to make a parser for Matlab. One of the main issue there is the fact that comments and transpose both use single quotes. What I was thinking of a solution was to define the STRING lexer rule in somewhat the following manner:
(if previous token is not ')','}',']' or [a-zA-Z0-9]) than match '\'' ( ESC_SEQ | ~('\\'|'\''|'\r'|'\n') )* '\'' (but note I do not want to consume the previous token if it is true).
Does anyone knows a workaround this problem, as it does not support negative lookaheads?
You can do negative lookahead in ANTLR4 using _input.LA(-1) (in Java, see how to resolve simple ambiguity or ANTLR4 negative lookahead in lexer).
You can also use lexer mode to deal with this kind of stuff, but your lexer had to be defined in its own file. The idea is to go from a state that can match some tokens to another that can match new ones.
Here is an example from ANTLR4 lexer documentation:
// Default "mode": Everything OUTSIDE of a tag
COMMENT : '<!--' .*? '-->' ;
CDATA : '<![CDATA[' .*? ']]>' ;
OPEN : '<' -> pushMode(INSIDE) ;
...
XMLDeclOpen : '<?xml' S -> pushMode(INSIDE) ;
...
// ----------------- Everything INSIDE of a tag ------------------ ---
mode INSIDE;
CLOSE : '>' -> popMode ;
SPECIAL_CLOSE: '?>' -> popMode ; // close <?xml...?>
SLASH_CLOSE : '/>' -> popMode ;