JavaCC grammar - parse everything to the end of the file - grammar

Question

I am using FeatureBNF (and so in essence I am using JavaCC) to try and write a grammar that will produce a (very) simple parser to parse Gherkin files.
An example Gherkin file:
Feature: Calculator
In order to avoid silly mistakes
As a math idiot
I want to be told the sum of two numbers
Scenario: Add two numbers
Given I have entered 50 into the calculator
And I have also entered 70 into the calculator
When I press add
Then the result should be 120 on the screen
All I want to do, to begin with, is parse this into a Feature that has the name Calculator and a Body, which is the entirety of the rest of the file.
I have struggled with the part where I'm trying to read the rest of the file into the Body however. I think maybe partially because there is no 'natural' delimiters for when one section ends - it's denoted by a newline.
Trying the following grammar:
<DEFAULT> TOKEN :
{
<FEATURE: "Feature: " >
| <#LETTER: ["\u0027","\u0041"-"\u005a","\u005f","\u0061"-"\u007a"] >
| <FEATURE_NAME: (<LETTER>)+ >
| <NEWLINE: ("\r\n" | "\n\r" | "\r" | "\n") >
| <TEXT : ~[] >
}
GRAMMARSTART
Feature :
<FEATURE> FeatureName <NEWLINE>
Body
<EOF>
;
FeatureName: <FEATURE_NAME>;
Body: (<TEXT>)*;
I get the error:
[java] java.lang.reflect.InvocationTargetException
... lots of stack trace removed...
[java] Caused by: cide.gparser.ParseException: Encountered "\r\n" (5) at line 2, column 1.
[java] Was expecting one of:
[java] <EOF>
[java] <TEXT> ...
I have been able to achieve what I want by adding some delimiters in to the Gherkin file and using lexical states, like so:
Feature: Calculator #TITLEEND
#BODYSTART
In order to avoid silly mistakes
As a math idiot
I want to be told the sum of two numbers
Scenario: Add two numbers
Given I have entered 50 into the calculator
And I have also entered 70 into the calculator
When I press add
Then the result should be 120 on the screen
#BODYEND
With the following relevant parts of the grammar:
<DEFAULT, IN_BODY> SPECIAL_TOKEN : {
" " | "\t" | "\n" | "\r" | "\f"
}
<DEFAULT> TOKEN : {
<FEATURE: "Feature: " >
| <#LETTER: ["\u0027", "\u0041"-"\u005a", "\u005f", "\u0061"-"\u007a"] >
| <FEATURE_NAME: (<LETTER>)+ >
| <ENDFEATURETITLE: "#TITLEEND" >
}
<DEFAULT> TOKEN : { <BODYSTART : "#BODYSTART"> : IN_BODY }
<IN_BODY> TOKEN : { <TEXT : ~[] > }
<IN_BODY> TOKEN : { <BODYEND : "#BODYEND"> : DEFAULT }
GRAMMARSTART
Feature:
<FEATURE> FeatureName <ENDFEATURETITLE>
Body
<EOF>;
FeatureName: <FEATURE_NAME>;
Body: <BODYSTART> Text <BODYEND>;
Text: (<TEXT>)*;
But I am sure I must be missing something and would like to be able to achieve this without having to annotate the feature files. What is a better way to do this?
SIDE NOTE
FeatureBNF builds on top of JavaCC and outputs a grammar file for JavaCC to process. I am completely new to both FeatureBNF and JavaCC, but they seem similar enough that I hope this question might be applicable to JavaCC gurus. (FeatureBNF uses JavaCC syntax for the lexical specifications and then its own format for the grammar's production rules.)

Answer 1

Based on your grammar, you can switch states after the first newline, so the following lexical grammar will suffice:
<DEFAULT> TOKEN : {
<FEATURE: "Feature: " >
| <#LETTER: ["\u0027", "\u0041"-"\u005a", "\u005f", "\u0061"-"\u007a"] >
| <FEATURE_NAME: (<LETTER>)+ >
| <ENDFEATURETITLE: "#TITLEEND" >
| <NEWLINE: ("\r\n" | "\n\r" | "\r" | "\n") > : IN_BODY
}
<IN_BODY> TOKEN : { <TEXT : ~[] > }
Now the syntactic grammar is
Feature:
<FEATURE> FeatureName <NEWLINE>
Body
<EOF>;
FeatureName: <FEATURE_NAME>;
Body: (<TEXT>)*;