why is this grammar an error 208? - antlr

Question

I don't understand why the following grammar leads to error 208 complaining IF will be never matched:
error(208): test.g:11:1: The following token definitions can never be matched because prior tokens match the same input: IF
ANTLRWorks 1.4.3
ANTLT 3.4
grammar test;
#lexer::members {
private boolean rawAhead() {
}
}
parse : IF*;
RAW : ({rawAhead()}?=> . )+;
IF : 'if';
ID : ('A'..'Z'|'a'..'z')+;
Either remove RAW rule or ID rule solves the error...
From my point of view, IF does have the possibility to be matched when rawAhead() returns false.

Answer 1

Bood wrote:
I think it actually matters, say if we have an and just an 'if' outside of the mmode, e.g. <#/>if<#/>, then the if here will be matched with IF, not RAW it should be (same length, match the first), right?
Yeah, you're right, good point. Giving it some more thought that is the expected behavior AFAIK. But, it seems things work a bit differently: the RAW rule gets precedence over the ID and IF rules, even when placed at the end of the lexer grammar as you can see:
freemarker_simple.g
grammar freemarker_simple;
#lexer::members {
private boolean mmode = false;
private boolean rawAhead() {
if(mmode) return false;
int ch1 = input.LA(1), ch2 = input.LA(2), ch3 = input.LA(3);
return !(
(ch1 == '<' && ch2 == '#') ||
(ch1 == '<' && ch2 == '/' && ch3 == '#') ||
(ch1 == '$' && ch2 == '{')
);
}
}
parse
: (t=. {System.out.printf("\%-15s '\%s'\n", tokenNames[$t.type], $t.text);})* EOF
;
OUTPUT_START : '${' {mmode=true;};
TAG_START : '<#' {mmode=true;};
TAG_END_START : '</' ('#' {mmode=true;} | ~'#' {$type=RAW;});
OUTPUT_END : '}' {mmode=false;};
TAG_END : '>' {mmode=false;};
EQUALS : '==';
IF : 'if';
STRING : '"' ~'"'* '"';
ID : ('a'..'z' | 'A'..'Z')+;
SPACE : (' ' | '\t' | '\r' | '\n')+ {skip();};
RAW : ({rawAhead()}?=> . )+;
Main.java
import org.antlr.runtime.*;
public class Main {
public static void main(String[] args) throws Exception {
freemarker_simpleLexer lexer = new freemarker_simpleLexer(new ANTLRStringStream("<#/if>if<#if>foo<#if>"));
freemarker_simpleParser parser = new freemarker_simpleParser(new CommonTokenStream(lexer));
parser.parse();
}
}
will print the following to the console:
TAG_START '<#'
IF 'if'
TAG_END '>'
RAW 'if'
TAG_START '<#'
IF 'if'
TAG_END '>'
RAW 'foo'
TAG_START '<#'
IF 'if'
TAG_END '>'
As you can see, the 'if' and 'foo' are tokenized as RAW in the input:
<#/if>if<#if>foo<#if>
^^ ^^^