Selectively Skip Newline Depending on Context - antlr

Question

I must parse files made of two parts. In the first one, new lines must be skipped. In the second one, they are important and used as a delimiter.
I want to avoid solutions like http://www.antlr.org/wiki/pages/viewpage.action?pageId=1734 and use predicate instead.
For the moment, I have something like:
WS: ( ' ' | '\t' | NEWLINE) {SKIP();};
fragment NEWLINE : '\r'|'\n'|'\r\n';
I tried to add a dynamically scoped variable keepNewline that is set to true when "entering" second part of the file.
However, I am not able to create the correct predicate to switch off the "skipping" of newlines.
Any help would be greatly appreciated.
Best regards.

Answer 1

It's easier than you might think: you don't even need a predicate.
Let's say you want to preserve line breaks only inside <pre>...</pre> tags. The following dummy grammar does just that:
grammar Pre;
#lexer::members {
private boolean keepNewLine = false;
}
parse
: (t=.
{
System.out.printf("\%-10s '\%s'\n", tokenNames[$t.type], $t.text.replace("\n", "\\n"));
}
)*
EOF
;
Word
: ('a'..'z' | 'A'..'Z')+
;
OPr
: '<pre>' {keepNewLine = true;}
;
CPr
: '</pre>' {keepNewLine = false;}
;
NewLine
: ('\r'? '\n' | '\r') {if(!keepNewLine) skip();}
;
Space
: (' ' | '\t') {skip();}
;
which you can test with the class:
import org.antlr.runtime.*;
public class Main {
public static void main(String[] args) throws Exception {
PreLexer lexer = new PreLexer(new ANTLRFileStream("in.txt"));
PreParser parser = new PreParser(new CommonTokenStream(lexer));
parser.parse();
}
}
And if in.txt would contain:
foo bar
<pre>
a
b
</pre>
baz
the output of running the Main class would be:
Word 'foo'
Word 'bar'
OPr '<pre>'
NewLine '\n'
Word 'a'
NewLine '\n'
NewLine '\n'
Word 'b'
NewLine '\n'
CPr '</pre>'
Word 'baz'