Within the context of scanning, what do i need to override, extend, listen to, visit to be able to print out this form of informative output when my text is being scanned?
-- Example output only ---------
DEBUG ... current mode: DEFAULT_MODE
DEBUG ... matching text '#' on rule SHARP ; pushing and switching to DIRECTIVE_MODE
DEBUG ... matching text 'IF" on rule IF ; pushing and switching to IF_MODE
DEBUG ... matching text ' ' on rule WS; skipping
DEBUG ... no match for text %
DEBUG ... no match for text &
DEBUG ... mathcing text '\r\n' on rule EOL; popping mode; current mode: DIRECTIVE_MODE
...
thanks
The solution was a lot simpler than I thought.
You just need to subclass the generated Lexer and override methods such as popMode(), pushMode() to get the printout you want. If you do this you should also override emit() methods as well to get properly sequential and contextual information.
Here's an example in C#:
class ExtendedLexer : MyGeneratedLexer
{
public ExtendedLexer(ICharStream input)
: base(input) { }
public override int PopMode()
{
Console.WriteLine($"Mode is being popped: Line: {Line} Column:{Column} ModeName: {ModeNames[ModeStack.Peek()]}");
return base.PopMode();
}
public override void PushMode(int m)
{
Console.WriteLine($"Mode is being pushed: Line: {Line} Column:{Column} ModeName: {ModeNames[m]}");
base.PushMode(m);
}
public override void Emit(IToken t)
{
Console.WriteLine($"[#{t.TokenIndex},{t.StartIndex}:{t.StopIndex}, <{Vocabulary.GetSymbolicName(t.Type)}> = '{t.Text}']");
base.Emit(t);
}
}
And the output would be something like:
Mode is being pushed: Line: 4 Column:3 ModeName: IF_MODE
[#-1,163:165, <IF> = '#IF']
Mode is being pushed: Line: 4 Column:4 ModeName: CONDITION_MODE
[#-1,166:166, <LPAREN> = '(']
[#-1,167:189, <EXP> = '#setStartDateAndEndDate']
Mode is being popped: Line: 4 Column:28 ModeName: IF_MODE
[#-1,190:190, <RPAREN> = ')']
Related
Is there a more elegant way of processing input coming from either the command line arguments or STDIN if no files were given on the command line? I'm currently doing it like this:
sub MAIN(*#opt-files, Bool :$debug, ... other named options ...) {
# note that parentheses are mandatory here for some reason
my $input = #opt-files ?? ([~] .IO.slurp for #opt-files) !! $*IN.slurp;
... process $input ...
}
and it's not too bad, but I wonder if I'm missing some simpler way of doing it?
I would probably go for a multi sub MAIN, something like:
multi sub MAIN(Bool :$debug)
{
process-input($*IN.slurp);
}
multi sub MAIN(*#opt-files, Bool :$debug)
{
process-input($_.IO.slurp) for #opt-files;
}
I'd probably do two things to change this. I'd break up the ?? !! onto different lines, and I'd go for a full method chain:
sub MAIN(*#opt-files, Bool :$debug, ... other named options ...) {
my $input = #opt-files
?? #opt-files».IO».slurp.join
!! $*IN.slurp;
... process $input ...
}
You can also map it by using #opt-files.map(*.IO.slurp).join
Edit: building on ugexe's answer, you could do
sub MAIN(*#opt-files, Bool :$debug, ... other named options ...) {
# Default to $*IN if not files
#opt-files ||= '-';
my $input = #opt-files».IO».slurp.join
... process $input ...
}
Something that I might expect to work is to set #*ARGS to the list of file names in the signature.
And then just use $*ARGFILES.
sub MAIN( *#*ARGS, Bool :$debug, ... other named options ...) {
my $input = slurp; # implicitly calls $*ARGFILES.slurp()
... process $input ...
}
It doesn't work though.
You could get Rakudo to update $*ARGFILES by nulling it with a low-level null before you use it.
sub MAIN( *#*ARGS, Bool :$debug, ... other named options ...) {
{ use nqp; $*ARGFILES := nqp::null }
my $input = slurp;
... process $input ...
}
But that is using an implementation detail that may change in the future.
A better way is to just directly create a new instance of IO::ArgFiles yourself.
You can even store it in $*ARGFILES. Then slurp on its own would slurp in all of the file contents.
sub MAIN( *#opt-files, Bool :$debug, ... other named options ...) {
my $*ARGFILES = IO::ArgFiles.new( #opt-files || $*IN );
my $input = slurp;
... process $input ...
}
Note that IO::ArgFiles is just an empty subclass of IO::CatHandle.
So you could write IO::CatHandle.new( #opt‑files || $*IN ) instead.
Parsing input files with size bigger than computer memory requires three steps:
The use of unbuffered character and token streams
Copy text out of sliding buffer and store in tokens
Ask the parser not to create parse trees
The objective is a simplified grammar to parse a file composed of many lines, each of one composed of many words.
With this requirement in mind, the following code uses #members action and sub classing the parser generated by ANTLR
The method printPagesAndWords receives a List of LineContext objects.
It prints the total number of lines (with the start method provided by LineContext), but it cannot access to the number of WORDs per line, whith the method WORD() provided by the LineContext object.
This is the output obtained:
Number of lines in page: 3
Line start token: word1
Number of words in line: 0
Line start token: 2323
Number of words in line: 0
Line start token: 554545
Number of words in line: 0
Furthermore, If I try to get the WORDs in a line, for example changing the line
System.out.println("Number of words in line:\t"+row.WORD().size()+"\n");
by the line
System.out.println("Number of words in line:\t"+row.WORD(1).getText()+"\n");
The following exception is thrown:
Exception in thread "main" java.lang.NullPointerException
at TestContext.printPagesAndWords(TestContext.java:11)
at ContextParser.read(ContextParser.java:130)
at Main.main(Main.java:11)
The following is the complete set of files:
ContextLexer.g4
lexer grammar ContextLexer;
NL
: '\r'? '\n'
;
WORD
: ~[ \t\n\r]+
;
SPACE
: [ \t] -> skip
;
ContextParser.g4
parser grammar ContextParser;
options {
tokenVocab=ContextLexer;
}
#members{
void printPagesAndWords(List<LineContext> rows){};
}
read
: dataLine+=line* {printPagesAndWords($dataLine);}
;
line
: WORD* NL
;
TestContext which extends ContextParser
import org.antlr.v4.runtime.TokenStream;
import java.util.List;
public class TestContext extends ContextParser {
public TestContext(TokenStream input) {
super(input);
}
void printPagesAndWords(List<LineContext> rows){
System.out.println("Number of lines in page:\t" + rows.size()+"\n");
for(LineContext row: rows){
System.out.println("Line start token:\t\t\t"+row.start.getText());
System.out.println("Number of words in line:\t"+row.WORD().size()+"\n");
}
};
}
The Main Class:
import org.antlr.v4.runtime.*;
import java.io.IOException;
public class Main {
public static void main(String[] args) throws IOException {
String source = "word1 word2 number anothernumber \n 2323 55r ere\n554545 lll 545\n";
ContextLexer lexer = new ContextLexer(CharStreams.fromString(source));
lexer.setTokenFactory(new CommonTokenFactory(true));
TokenStream tokens = new UnbufferedTokenStream<CommonToken>(lexer);
TestContext parser = new TestContext(tokens);
parser.setBuildParseTree(false);
parser.read();
}
}
I just put this test method together:
#Unroll
def 'super start edit should be called if cell is not empty'( boolean empty ){
given:
DueDateEditor editor = GroovySpy( DueDateEditor ){
isEmpty() >> empty
}
when:
editor.startEdit()
then:
if( empty){
0 * editor.callSuperStartEdit()
}
else {
1 * editor.callSuperStartEdit()
}
where:
empty | _
true | _
false | _
}
... it works OK in terms of the two tests passing... but when you make it fail it's very odd: the output if the parameter empty is false is
super start edit should be called if cell is not empty[1]
... and it is 0 if the parameter empty is true. Is this a bug?
I am writing an additional answer because
there is a small bug in Tim's solutions with regard to the title (but still his answer is technically absolutely correct!),
you don't need a GroovySpy here, a simple Spy absolutely suffices,
I want to show you an alternative way of testing without stubbing isEmpty(),
I want to show you how you can use just one interaction with the number of calls in a ternary expression instead of an if-else (even though error reporting is ugly then),
I want to comment on your way of testing in general (see the end of this post).
package de.scrum_master.stackoverflow.q61032514;
import java.time.LocalDate;
public class DueDateEditor {
String text;
public boolean isEmpty() {
return text == null || text.trim() == "";
}
public void startEdit() {
if (!isEmpty())
callSuperStartEdit();
}
public void callSuperStartEdit() {}
}
package de.scrum_master.stackoverflow.q61032514
import spock.lang.Specification
import spock.lang.Unroll
class DueDateEditorTest extends Specification {
#Unroll
def 'super start edit #shouldMsg be called if the cell is #cellStateMsg'() {
given:
DueDateEditor editor = Spy() {
isEmpty() >> empty
}
when:
editor.startEdit()
then:
(empty ? 0 : 1) * editor.callSuperStartEdit()
where:
empty << [true, false]
shouldMsg = empty ? 'should not' : 'should'
cellStateMsg = empty ? 'empty' : 'not empty'
}
#Unroll
def "super start edit #shouldMsg be called if cell text is '#text'"() {
given:
DueDateEditor editor = Spy()
editor.text = text
when:
editor.startEdit()
then:
(editor.isEmpty() ? 0 : 1) * editor.callSuperStartEdit()
// Or, if 'isEmpty()' has a side effect:
// (text ? 1 : 0) * editor.callSuperStartEdit()
where:
text << ["foo", "", null, "line 1\nline 2"]
shouldMsg = text ? 'should' : 'should not'
cellStateMsg = text ? 'not empty' : 'empty'
}
}
General remarks:
I would not test the internal wiring of a single class with interactions. The test will be brittle and if you refactor the class internally without changing the API at all, the test might break if the interactions are no longer as expected. I think this is over-specification and I would only use interaction testing for crucial interactions between different classes or maybe different instances of one class - "crucial" meaning things like the functionality of design patterns like Observer.
Having an if-else for differentiating two cases by two different interaction patterns if the whole test only knows exactly those two cases just makes the test less readable and more complex, see your own code as well as mine and Tim's. In such a case I would rather write two feature methods with simple titles and simple functionality, but without if-else or ternary expressions, without helper variables for titles etc.
P.S.: Sorry, I had to make up a sample class under test DueDateEditor in order to make my test compile and run as expected. As usual, Mike unfortunately didn't provide an MCVE but just a part of it.
Update: We talked about GroovySpy in our comments and, as I said, it will not work if your classes are Java classes and there is a final method in you want to stub, see the Spock manual. Here is proof for you:
package de.scrum_master.stackoverflow.q61032514;
public class TreeTableCell<A, B> {
String text;
public final boolean isEmpty() {
return text == null || text.trim() == "";
}
}
package de.scrum_master.stackoverflow.q61032514;
import java.time.LocalDate;
public class DueDateEditor extends TreeTableCell<String, LocalDate> {
public void startEdit() {
if (!isEmpty())
callSuperStartEdit();
}
public void callSuperStartEdit() {}
}
package de.scrum_master.stackoverflow.q61032514
import spock.lang.Specification
import spock.lang.Unroll
class DueDateEditorTest extends Specification {
#Unroll
def 'super start edit #shouldMsg be called if the cell is #cellStateMsg'() {
given:
DueDateEditor editor = GroovySpy() {
isEmpty() >> empty
}
when:
editor.startEdit()
then:
(empty ? 0 : 1) * editor.callSuperStartEdit()
where:
empty << [true, false]
shouldMsg = empty ? 'should not' : 'should'
cellStateMsg = empty ? 'empty' : 'not empty'
}
}
The test would work if your application classes were Groovy classes only. But if they are Java classes like in my example, the test will fail like this:
Too few invocations for:
(empty ? 0 : 1) * editor.callSuperStartEdit() (0 invocations)
Unmatched invocations (ordered by similarity):
1 * editor.startEdit()
methodName == "callSuperStartEdit"
| |
startEdit false
10 differences (44% similarity)
(s---------)tartEdit
(callSuperS)tartEdit
So in this case you cannot just use Groovy magic to check interactions. But as I said, you shouldn't do that anyway. Rather make sure that both startEdit() and callSuperStartEdit() do the right things. Check their results or, if they are void, check their side effects on the state of the subject under test or its collaborators.
Update 2: Regarding your original question about indexed method naming, actually #tim_yates gave the correct answer. I just want to add the corresponding Spock manual link explaining method unrolling and how you can influence naming using variables from the where: block.
No it's not a bug
You didn't tell spock how to name your tests differently so it adds the iteration (0 then 1) to the name
Change it to
#Unroll
def 'super start edit should be called if isEmpty for the cell returns #empty'(){
given:
DueDateEditor editor = GroovySpy( DueDateEditor ){
isEmpty() >> empty
}
when:
editor.startEdit()
then:
if( empty){
0 * editor.callSuperStartEdit()
}
else {
1 * editor.callSuperStartEdit()
}
where:
empty << [true, false]
}
I changed the where section, as a table with one column felt odd
edit
You can also do this:
#Unroll
def 'super start edit should be called if the cell is #msg'(){
given:
DueDateEditor editor = GroovySpy( DueDateEditor ){
isEmpty() >> empty
}
when:
editor.startEdit()
then:
if( empty){
0 * editor.callSuperStartEdit()
}
else {
1 * editor.callSuperStartEdit()
}
where:
empty << [true, false]
msg = empty ? 'empty' : 'not empty'
}
I am testing the idea of making my dsl Jvm compatible and I wanted to test the possibility of extending Xbase and using the interpreter. I have tried to make a minimal test project to use with the interpreter but I am getting a runtime error. I think I understand the general concepts of adapting Xbase, but am unsure about how the setup/entrypoints for the interpreter and could not find any information regarding the error I am getting or how to resolve. Here are the relevant files for my situation:
Text.xtext:
import "http://www.eclipse.org/xtext/xbase/Xbase" as xbase
import "http://www.eclipse.org/xtext/common/JavaVMTypes" as types
Program returns Program:
{Program}
'program' name=ID '{'
variables=Var_Section?
run=XExpression?
'}'
;
Var_Section returns VarSection:
{VarSection}
'variables' '{'
decls+=XVariableDeclaration+
'}'
;
#Override // Change syntax
XVariableDeclaration returns xbase::XVariableDeclaration:
type=JvmTypeReference name=ID '=' right=XLiteral ';'
;
#Override // Do not allow declarations outside of variable region
XExpressionOrVarDeclaration returns xbase::XExpression:
XExpression;
TestJvmModelInferrer:
def dispatch void infer(Program element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
acceptor.accept(element.toClass(element.fullyQualifiedName)) [
documentation = element.documentation
if (element.variables !== null) {
for (decl : element.variables.decls) {
members += decl.toField(decl.name, decl.type) [
static = true
initializer = decl.right
visibility = JvmVisibility.PUBLIC
]
}
}
if (element.run !== null) {
members += element.run.toMethod('main', typeRef(Void::TYPE)) [
parameters += element.run.toParameter("args", typeRef(String).addArrayTypeDimension)
visibility = JvmVisibility.PUBLIC
static = true
body = element.run
]
}
]
}
Test case:
#Inject ParseHelper<Program> parseHelper
#Inject extension ValidationTestHelper
#Inject XbaseInterpreter interpreter
#Test
def void basicInterpret() {
val result = parseHelper.parse('''
program program1 {
variables {
int var1 = 0;
double var2 = 3.4;
}
var1 = 13
}
''')
result.assertNoErrors
var interpretResult = interpreter.evaluate(result.run)
println(interpretResult.result)
Partial stack trace:
java.lang.IllegalStateException: Could not access field: program1.var1 on instance: null
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter._assignValueTo(XbaseInterpreter.java:1262)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.assignValueTo(XbaseInterpreter.java:1221)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter._doEvaluate(XbaseInterpreter.java:1213)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.doEvaluate(XbaseInterpreter.java:216)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.internalEvaluate(XbaseInterpreter.java:204)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.evaluate(XbaseInterpreter.java:190)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.evaluate(XbaseInterpreter.java:180)
The interpreter does only support expressions, but does not work with types that are created by a JvmModelInferrer. Your code tries to work with fields of such an inferred type.
Rather than using the interpreter, I'd recommend to use an InMemoryCompiler in your test. The domainmodel example may serve as an inspiration: https://github.com/eclipse/xtext-eclipse/blob/c2b15c3ec118c4c200e2b28ea72d8c9116fb6800/org.eclipse.xtext.xtext.ui.examples/projects/domainmodel/org.eclipse.xtext.example.domainmodel.tests/xtend-gen/org/eclipse/xtext/example/domainmodel/tests/XbaseIntegrationTest.java
You may find this project interesting, which (among other stuff) implements an interpreter for Xtend based on the Xbase interpreter. It might be a bit outdated, though, and also will not fully support all Xtend concepts. But it could be a starting point, and your contrbutions are welcome :-)
https://github.com/kbirken/xtendency
I have an Xtext grammar which reads (in part):
grammar mm.ecxt.MMLanguage hidden(WS, COMMENT)
import "http://www.eclipse.org/emf/2002/Ecore" as ecore
...
Statement:
ConstantStatement |
VariableStatement |
LabeledStatement |
...
LabeledStatement:
EssentialHypothesisStatement |
...
ConstantStatement:
DOLLAR_C (constants+=ConstDecl)+ DOLLAR_DOT;
VariableStatement:
DOLLAR_V (variables+=VarDecl)+ DOLLAR_DOT;
EssentialHypothesisStatement:
name=LABEL DOLLAR_E (symbols+=[Decl|MATHSYMBOL])+ DOLLAR_DOT;
Decl: ConstDecl | VarDecl;
ConstDecl returns ConstDecl: name=MATHSYMBOL;
VarDecl returns VarDecl: name=MATHSYMBOL;
MATHSYMBOL: PARENOPEN | PARENCLOSE | QUESTIONMARK | COMPRESSED | TLABEL | WORD;
...
(The full grammar is MMLanguage.xtext from current commit 328a5e7 of https://github.com/marnix/metamath-eclipse-xtext/.)
My question: How do I highlight the symbols in an EssentialHypothesisStatement, by using a different color for constants and variables? So if the MATHSYMBOL refers to a ConstDecl, then it should be highlighted one way, and some other way for a VarDecl.
I've tried to create an ISemanticHighlightingCalculator in all kinds of ways, but I can't seem to detect what the actual reference type is, neither through the node model nor through the Ecore model. On the one hand, the grammar-related methods only tell me that the reference goes to a Decl. On the other hand, the Ecore model's EReferences tell me whether the target is a ConstDecl or a VarDecl, but there I can't find the location of the source MATHSYMBOL.
Note that I prefer to use the node model (as opposed to the Ecore model) since I also want to highlight comments, and for performance reasons I cannot afford multiple passes over the document.
What is a good/canonical/efficient/simple way to achieve this?
from EObject perspective have a look at org.eclipse.xtext.nodemodel.util.NodeModelUtils.findNodesForFeature(EObject, EStructuralFeature)
from the node model perspective you can use EObjectAtOffsetHelper
sample grammar
Model:
defs+=Def*
uses+=Use*
;
Def:
ADef | BDef;
ADef:
"adef" name=ID
;
BDef:
"bdef" name=ID
;
Use:
"use" def=[Def]
;
And here the Impl
public class MyDslSemanticHighlightingCalculator implements ISemanticHighlightingCalculator {
#Inject
private MyDslGrammarAccess ga;
#Inject
private EObjectAtOffsetHelper helper;
#Override
public void provideHighlightingFor(XtextResource resource, IHighlightedPositionAcceptor accoptor, CancelIndicator cancelIndicator) {
if (resource == null)
return;
IParseResult parseResult = resource.getParseResult();
if (parseResult == null || parseResult.getRootNode() == null)
return;
BidiTreeIterable<INode> tree = parseResult.getRootNode().getAsTreeIterable();
for (INode node : tree) {
if (cancelIndicator.isCanceled()) {
return;
}
if (node.getGrammarElement() instanceof CrossReference) {
if (ga.getUseAccess().getDefDefCrossReference_1_0() == node.getGrammarElement()) {
EObject target = helper.resolveElementAt(resource, node.getOffset());
if (target instanceof ADef) {
accoptor.addPosition(node.getOffset(), node.getLength(), HighlightingStyles.COMMENT_ID);
} else if (target instanceof BDef) {
accoptor.addPosition(node.getOffset(), node.getLength(), HighlightingStyles.STRING_ID);
}
}
}
}
}
}