Returning To Default Mode Using Island Grammar Without Consuming Token - antlr

Question

I'm still trying to parse a simple Javadoc style format using ANTLR. Basically the format looks like this:
/**
* Description
*
* #name someId
*/
My parser grammar is here:
query_doc : BEGIN_QDOC description name NOMANSLAND* END_QDOC;
description : (DESCRIPTION_TEXT | NOMANSLAND)*;
name : OPEN_NAME INNER_WS NAMEID INNER_WS* CLOSE_NAME;
My lexer grammar is here:
BEGIN_QDOC : '/**';
END_QDOC : ('*/');
NOMANSLAND : '\r'? '\n' (' ' | '\t')* '*' (' ' | '\t')*;
DESCRIPTION_TEXT : ~('\n');
OPEN_NAME : '#name' -> mode(NAME);
mode NAME;
INNER_WS : (' ' | '\t')+;
NAMEID : ('a'..'z' | 'A'..'Z' | '0'..'9' | '-' | '_' | '?')+;
CLOSE_NAME : (('\r'? '\n') | '*/') -> mode(DEFAULT_MODE);
This appears to be working okay for the most part except for closing the #name definition in the following case:
/**
* #name someId*/
The above should be perfectly valid. We should not need a new line before ending the comment with '*/'. The issue I am having is that '*/' terminates the name definition successfully, but it consumes the token and only returns to the default mode so I need to have:
/**
* #name someId*/*/
if I actually want it to end the comment. I want it to return to the default mode and then realize that this token should end the comment (i.e. it should match END_QDOC). How can I accomplish this in ANTLR? I tried fixing it so that CLOSE_NAME is the inverse of ID:
CLOSE_NAME : ~('a'..'z' | 'A'..'Z' | '0'..'9' | '-' | '_' | '?');
But ANTLR still consumes the * leaving a unrecognized token error on the remaining '/'. What I would really like to do is have ANTLR exit the mode without consuming the token so that '*/' is the next token when we return to DEFAULT_MODE. Any thoughts?

Answer 1

First of all, rather than use the mode command, you probably want to use -> pushMode(NAME) and -> popMode to go back to the default mode.
For your CLOSE_NAME rule, you could use a predicate instead of a matching literal for handling the end of a comment:
CLOSE_NAME
: ( '\r'? '\n'
| {_input.LA(1) == '*' && _input.LA(2) == '/'}?
)
-> popMode
;
This can produce a zero-length token and wasn't allowed in ANTLR 4.0, but the restriction was removed (changed to a warning) in ANTLR 4.1 since we realized that a zero-length token could be used to trigger a mode change and thus avoid infinite loops.