Xtext rule that negates other valid matches - grammar

Question

I am fairly new to Xtext, so it is possible I am asking the wrong thing, or using incorrect terminology. Please keep this in mind in your responses.
I am attempting to implement JBehave EBNF Spec from scratch in Xtext as a learning exercise. JBehave is a very "wordy" grammar, similar to the one I will need to be able to maintain, so I will need to understand how to handle various types of "words" in different context.
I have been able to get this test case to pass as a starting point.
#Test
def void loadModel() {
// Multi-line
var story = parseHelper.parse('''
The quick brown fox
Jumps over the lazy dog
''')
assertThat(story, notNullValue())
assertThat(
story.description,
equalTo('''
The quick brown fox
Jumps over the lazy dog
''')
)
// Single-line description
story = parseHelper.parse('''
The quick brown fox
''')
assertThat(
story.description,
equalTo("The quick brown fox\n")
)
}
Using this grammar definition...
grammar org.example.jbehave.JBehave hidden (WS)
import "http://www.eclipse.org/emf/2002/Ecore" as ecore
generate jbehave "http://www.example.org/jbehave"
// The story describes a feature via description, narrative and a set of scenarios
// Story := Description? Meta? Narrative? GivenStories? Lifecycle? Scenario+ ;
Story:
description=Description?
;
// The Description is expressed by any sequence of words that must not contain any keywords at start of lines.
// Description := (Word Space?)* ;
Description:
((WORD) (WORD)* EOL+)+
// ((NON_KEYWORD) (WORD)* EOL+)+
;
// Key Words
////
// TODO: parser fails when uncommented
//terminal NON_KEYWORD:
// !(IN_ORDER_TO
// | AS_A
// | I_WANT_TO
// | SO_THAT
// | SCENARIO
// | GIVEN_STORIES
// | GIVEN
// | THEN
// | WHEN
// | AND
// )
//;
terminal fragment IN_ORDER_TO: "In order to";
terminal fragment AS_A: "As a";
terminal fragment I_WANT_TO: "I want to";
terminal fragment SO_THAT: "So that";
terminal fragment SCENARIO: "Scenario:";
terminal fragment GIVEN_STORIES: "GivenStories:";
terminal fragment GIVEN: "Given";
terminal fragment WHEN: "When";
terminal fragment THEN: "Then";
terminal fragment AND: "And";
// Common Terminals
////
terminal WORD: ('a'..'z'|'A'..'Z'|'_')('a'..'z'|'A'..'Z'|'_'|'0'..'9')*;
terminal WS: (' '|'\t')+;
terminal EOL: NL;
terminal fragment NL: ('\r'? '\n');
The problems I am running into are outlined in the comments.
When I uncomment terminal NON_KEYWORD, the test fails with
Expected: "The quick brown fox\nJumps over the lazy dog\n"
but: was "The"
If I then replace the line commented out in Description, the test fails to parse at all with
Expected: not null
but: was null
I sort of understand what is happening here in a vague sense. Tokens I define before WORD are also valid words, and so it is throwing off the parser. Therefore my questions are as follows.
Where can I find in the Xtext documentation (or other sources) that describe the underlying principals that are in affect here. I've read Xtext docs many times by now, but all I could find was a brief note on the order-dependence of terminal statements.
What is a good way to debug how the parser is interpreting my grammar? Is there something similar to dumping IFormattableDocument to the console, but for the lexer/parser/whatever?
And finally, what is the best way to tackle this problem from an Xtext perspective. Should I be looking into custom Data Types, or is this expressible in pure Xtext?
I am seeking to understand the underlying principals.
Update
Well this is certainly odd. I attempted to move past this for now and implement the next part of the spec.
; The narrative is identified by keyword "Narrative:" (or equivalent in I18n-ed locale),
; It is followed by the narrative elements
Narrative:= "Narrative:" ( InOrderTo AsA IWantTo | AsA IWantTo SoThat ) ;
I actually couldn't get this working on it's own. However, when I uncommented the original code and tried them together, it works!
#Test
def void narrativeOnly() {
var story = _th.parse('''
Narrative:
In order check reports
As a Developer
I want to workin with todos using examples
''')
assertThat(story, notNullValue())
}
#Test
def void descriptionOnly() {
// Multi-line
var story = _th.parse('''
The quick brown fox
Jumps over the lazy dog
''')
assertThat(story, notNullValue())
assertThat(
story.description,
equalTo('''
The quick brown fox
Jumps over the lazy dog
''')
)
// Single-line description
story = _th.parse('''
The quick brown fox
''')
assertThat(
story.description,
equalTo("The quick brown fox\n")
)
}
grammar org.agileware.natural.jbehave.JBehave hidden (WS)
import "http://www.eclipse.org/emf/2002/Ecore" as ecore
generate jbehave "http://www.agileware.org/natural/jbehave"
// Story
////
// The story describes a feature via description, narrative and a set of scenarios
// Story := Description? Meta? Narrative? GivenStories? Lifecycle? Scenario+ ;
Story:
description=Description?
narrative=Narrative?
;
// Narrative
////
// The narrative is identified by keyword "Narrative:" (or equivalent in I18n-ed locale),
// It is followed by the narrative elements
// Narrative:= "Narrative:" ( InOrderTo AsA IWantTo | AsA IWantTo SoThat ) ;
// The narrative element content is any sequence of characters that do not match a narrative starting word
// NarrativeElementContent := ? Any sequence of NarrativeCharacter that does not match NarrativeStartingWord ? ;
Narrative:
'Narrative:'
inOrderTo=InOrderTo
asA=AsA
wantTo=IWantTo
;
// InOrderTo:= "In order to" NarrativeElementContent ;
InOrderTo:
IN_ORDER_TO (WORD) (WORD)* EOL+;
// AsA:= "As a" NarrativeElementContent ;
AsA:
AS_A (WORD) (WORD)* EOL+;
// IWantTo:= "I want to" NarrativeElementContent ;
IWantTo:
I_WANT_TO (WORD) (WORD)* EOL+;
// SoThat:= "So that" NarrativeElementContent ;
SoThat:
SO_THAT (WORD) (WORD)* EOL+;
// The Description is expressed by any sequence of words that must not contain any keywords at start of lines.
// Description := (Word Space?)* ;
Description:
((WORD) (WORD)* EOL+)+
;
// Key Words
////
//terminal NON_KEYWORD:
// !(IN_ORDER_TO
// | AS_A
// | I_WANT_TO
// | SO_THAT
// | SCENARIO
// | GIVEN_STORIES
// | GIVEN
// | THEN
// | WHEN
// | AND
// )
//;
terminal IN_ORDER_TO: "In order to";
terminal AS_A: "As a";
terminal I_WANT_TO: "I want to";
terminal SO_THAT: "So that";
//terminal SCENARIO: "Scenario:";
//terminal GIVEN_STORIES: "GivenStories:";
//terminal GIVEN: "Given";
//terminal WHEN: "When";
//terminal THEN: "Then";
//terminal AND: "And";
// Common Terminals
////
terminal WORD: (LETTER)(LETTER|DIGIT)*;
terminal fragment LETTER: ('a'..'z'|'A'..'Z');
terminal fragment DIGIT: ('0'..'9');
terminal WS: (' '|'\t')+;
terminal EOL: NL;
terminal fragment NL: ('\r'? '\n');
This takes care of #3 I guess, but arriving there by accident sort of defeats the purpose. I will now accept any answer that can point me to, or describe to me, the underlying principals that cause the behavior I've described.
Why can't I just match a random group of words? How does defining the narrative assignment along with the description assignment in Story change how the parser interprets the grammar?

Answer 1

I have been able to answer all 3 of my questions using ANTLRWorks, a gui tool in the form of an executable jar with the express purpose of debugging, visualizing, and helping one understand the behavior of the parser.
To get this working with Xtext, one needs to add the following thier mwe2 generator:
language = StandardLanguage {
// ...
parserGenerator = {
debugGrammar = true
}
}
Then open the generated debug file in the ANTLRWorks tool and hit the "Bug" (debug) icon. This file should be located at src-gen/*/parser/antlr/internal/DebugInternal*.g
Source: https://blogs.itemis.com/en/debugging-xtext-grammars-what-to-do-when-your-language-is-ambiguous