Antlr 4 Lexer with multiple modes failing to tokenise correctly - antlr

Question

I'm trying to create a lexer with multiple modes using Antlr 4.7. My lexer currently is:
ACTIONONLY : 'AO';
BELIEFS : ':Initial Beliefs:' -> mode(INITIAL_BELIEFS);
NAME : ':name:';
WORD: ('a'..'z'|'A'..'Z'|'0'..'9'|'_')+;
COMMENT : '/*' .*? '*/' -> skip ;
LINE_COMMENT : '//' ~[\n]* -> skip ;
NEWLINE:'\r'? '\n' -> skip ;
WS : (' '|'\t') -> skip ;
mode INITIAL_BELIEFS;
GOAL_IB : ':Initial Goal:' -> mode(GOALS);
IB_COMMENT : '/*' .*? '*/' -> skip ;
IB_LINE_COMMENT : '//' ~[\n]* -> skip ;
IB_NEWLINE:'\r'? '\n' -> skip ;
IB_WS : (' '|'\t') -> skip ;
BELIEF_BLOCK: ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'('|')'|','|'.')+;
mode REASONING_RULES;
R1: 'a';
R2: 'b';
mode GOALS;
GL_COMMENT : '/*' .*? '*/' -> skip ;
GL_LINE_COMMENT : '//' ~[\n]* -> skip ;
GL_NEWLINE:'\r'? '\n' -> skip ;
GL_WS : (' '|'\t') -> skip ;
GOAL_BLOCK: ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'('|')'|','|'.')+;
Note that there is no way, at present, to get into the REASONING_RULES mode (so this should not, as I understand it have any effect on the operation of the lexer). Obviously I do want to use this mode, but this is the minimal version of the lexer that seems to display the problem I'm having.
My parser is:
grammar ActionOnly;
options { tokenVocab = ActionOnlyLexer; }
// Mas involving ActionOnly Agents
mas : aoagents;
aoagents: ACTIONONLY (aoagent)+;
// Agent stuff
aoagent :
(ACTIONONLY?)
NAME w=WORD
BELIEFS (bs=BELIEF_BLOCK )?
GOAL_IB gs=GOAL_BLOCK;
and I'm trying to parse:
AO
:name: robot
:Initial Beliefs:
abelief
:Initial Goal:
at(4, 2)
This fails with the error
line 35:0 mismatched input 'at(4,' expecting GOAL_BLOCK
which I'm assuming is because it isn't tokenising correctly.
If I omit rule R2 in the REASONING_RULES mode then it parses correctly (in general I seem to be able to have one rule in REASONING_RULES and it will work, but more than one rule and it fails to match GOAL_BLOCK)
I'm really struggling to see what I'm doing wrong here, but this is the first time I've tried to use lexer modes with Antlr.

Answer 1

I don't get that error when I try your grammars. I also tested with ANTLR 4.7.
Here's my test rig:
import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.Token;
public class Main {
public static void main(String[] args) {
String source = "AO\n" +
"\n" +
":name: robot\n" +
"\n" +
":Initial Beliefs:\n" +
"\n" +
"abelief\n" +
"\n" +
":Initial Goal:\n" +
"\n" +
"at(4, 2)";
ActionOnlyLexer lexer = new ActionOnlyLexer(CharStreams.fromString(source));
CommonTokenStream tokens = new CommonTokenStream(lexer);
tokens.fill();
System.out.println("[TOKENS]");
for (Token t : tokens.getTokens()) {
System.out.printf(" %-20s %s\n", ActionOnlyLexer.VOCABULARY.getSymbolicName(t.getType()), t.getText());
}
System.out.println("\n[PARSE-TREE]");
ActionOnlyParser parser = new ActionOnlyParser(tokens);
ParserRuleContext context = parser.mas();
System.out.println(" "+context.toStringTree(parser));
}
}
And this is printed to my console:
[TOKENS]
ACTIONONLY AO
NAME :name:
WORD robot
BELIEFS :Initial Beliefs:
BELIEF_BLOCK abelief
GOAL_IB :Initial Goal:
GOAL_BLOCK at(4,
GOAL_BLOCK 2)
EOF <EOF>
[PARSE-TREE]
(mas (aoagents AO (aoagent :name: robot :Initial Beliefs: abelief :Initial Goal: at(4,)))
Perhaps you need to generate new lexer/parser classes?
PS. note that ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'('|')'|','|'.')+ can be written as [a-zA-Z0-9_(),.]+