Issue defining an Antlr XYZ File grammar that can consume a '\n' terminated string non greedly - antlr

Question

I just started using Antlr4. As initial project I tasked myself with writing a Grammar for XYZ files since they are relatively simple.
At the moment it works great if there are no comments in the file.
So far this has been my progress:
grammar XYZFile;
options {
accessLevel = '';
}
molecule : nAtomsLine commentLine atom ;
nAtomsLine : nAtom NEWLINE ;
nAtom : N_ATOMS ;
atom : ( atom3d | atom2d ) NEWLINE? (atom | EOF )? ;
atom3d : symbol xCoord yCoord zCoord ;
atom2d : symbol xCoord yCoord ;
xCoord : FLOAT ;
yCoord : FLOAT ;
zCoord : FLOAT ;
symbol : SYMBOLSTR ;
commentLine : comment NEWLINE ;
comment : COMMENT? ;
NEWLINE : '\r'? '\n' ;
SYMBOLSTR : 'A' ( 'c' | 'g' | 'l' | 'm' | 'r' | 's' | 't' | 'u' )
| 'B' ( 'a' | 'e' | 'h' | 'i' | 'k' | 'r' )?
| 'C' ( 'a' | 'd' | 'e' | 'f' | 'l' | 'm' | 'n' | 'o' | 'r' | 's' | 'u' )?
| 'D' ( 'b' | 's' | 'y' )
| 'E' ( 'r' | 's' | 'u' )
| 'F' ( 'e' | 'l' | 'm' | 'r' )?
| 'G' ( 'a' | 'd' | 'e' )
| 'H' ( 'e' | 'f' | 'g' | 'o' | 's' )?
| 'I' ( 'n' | 'r' )?
| 'K' 'r'?
| 'L' ( 'a' | 'i' | 'r' | 'u' | 'v' )
| 'M' ( 'c' | 'g' | 'n' | 'o' | 't' )
| 'N' ( 'a' | 'b' | 'd' | 'e' | 'h' | 'i' | 'o' | 'p' )?
| 'O' ( 'g' | 's' )?
| 'P' ( 'a' | 'b' | 'd' | 'm' | 'o' | 'r' | 't' | 'u' )?
| 'R' ( 'a' | 'b' | 'e' | 'f' | 'g' | 'h' | 'n' | 'u' )
| 'S' ( 'b' | 'c' | 'e' | 'g' | 'i' | 'm' | 'n' | 'r' )?
| 'T' ( 'a' | 'b' | 'c' | 'e' | 'h' | 'i' | 'l' | 'm' | 's' )
| 'U' | 'V' | 'W' | 'Xe' | 'Y' 'b'?
| 'Z' ( 'n' | 'r' )
;
N_ATOMS : INT ;
INT : DIGIT+ ;
FLOAT : '-'? DIGIT+ '.' DIGIT*
| '-'? '.' DIGIT+
;
WS : [ \t] -> skip ;
COMMENT : ~[\n\r].*? ;
fragment
DIGIT : [0-9] ;
I think my issue is in the lexer's COMMENT part. It is supposed to consume everything until the endo of the line. Currently its consumming only one character on purpose because all modifications I made to it only made it worse:
.*? NEWLINE
~[\n\r]*?
~[\n\r]*? NEWLINE
~[\n\r] .*? NEWLINE
Pretty sure that I tried many other things in frustration but these should be enough to illustrate where I feel stuck.
I understand that this predicate can match other cases but I'm not seeing how to avoid it.
Thank you for your time.

Answer 1

The second line can contain pretty much any kind of characters (also digits), making it hard in the lexer to make a distinction between a digit/number being part of a comment or part of a coordinate (as already explained by Mike).
It'd be a bit overkill to create a grammar for this file format: processing it line by line would be a better choice. But given this is more of an exercise to get familiar with ANTLR, I'll suggest a way how you could do it.
A solution would be to make the lexer a bit context sensitive so that it "knows" when it is in one of 3 modes:
first line mode: an integer number can be created
second line mode: any characters making a comment
last mode: remaining lines containing an atom + coordinates
ANTLR's lexer has something called lexical modes where you can guide the lexer in one of these modes I described above. To be able to use lexical modes, you must separate the lexer and parser grammar in their own file however.
Here's how that might look like:
file: XYZLexer.g4
lexer grammar XYZLexer;
INTEGER
: [0-9]+
;
END_LINE_1
: [\r\n]+ -> skip, mode(COMMENT_MODE)
;
mode COMMENT_MODE;
COMMENT
: ~[\r\n]+
;
END_LINE_2
: [\r\n]+ -> skip, mode(ATOM_MODE)
;
mode ATOM_MODE;
ATOM
: [a-zA-Z]
;
NUMBER
: '-'? [0-9]+ '.' [0-9]+
;
SPACES
: [ \t]+ -> skip
;
LINE_BREAK
: [\r\n]+
;
file: XYZParser.g4
parser grammar XYZParser;
options {
tokenVocab=XYZLexer;
}
xyz_file
: INTEGER COMMENT atom_lines EOF
;
atom_lines
: atom ( LINE_BREAK+ atom )* LINE_BREAK*
;
atom
: ATOM coordinate
;
coordinate
: NUMBER+
;
With a parser generated from the above grammar(s), input like:
2
comment example
C 0.00000 1.40272 0.00000
H 0.00000 2.49029 0.00000
would be parsed as follows:

Answer 2

The first step in ANTLR parsing your input, is to convert your input stream of characters into a stream of tokens. This process uses you Lexer rules (the rules that begin with a capital letter). At this time, the parser rules are irrelevant, the parser rules act on the stream of tokens that the Lexer produces.
When the Lexer (aka tokenizer), tokenizes your input characters, it will evaluate you input against all of your Lexer rules. When more than 1 rule can match your input, then there are two "tie-breaker" strategies:
The Lexer rule that matches the longest stream of input characters with take top priority.
If there is more than one rule that matches the same (longest) sequence of characters, then the rule that appears first "wins"
In your grammar, the COMMENT rule (~[\n\r].*?) is going to match the complete contents of any line. As a result, none of your other Lexer rules really stand a chance (excepting the NEWLINE rule of course). Having your other Lexer rules before the COMMENT rule won't matter, because they match a shorter stream of input characters than the COMMENT rule.
Looking at what little "specs" there are at the link you provided, this is going to be rather difficult. (Note: This is what most languages have some sort of "start a comment" token; often //)
If you've followed the ANTLR set up in the intro, and have defined the grun alias, it's always a good starting point to run your input through grun with the -tokens flag to see how the Lexer interprets you input stream as a stream of tokens.
You might have some success with a semantic predicate on your COMMENT rule that checks for a line beginning with an Atomic symbol or a number, and returns false to prevent the COMMENT rule from matching, but the file format seems to be pretty "relaxed", so this might not be very manageable.
The short answer is the your COMMENT rule will have to reject input that's not a comment in the XYZ format, and that seems rather ambiguous.