I am testing the idea of making my dsl Jvm compatible and I wanted to test the possibility of extending Xbase and using the interpreter. I have tried to make a minimal test project to use with the interpreter but I am getting a runtime error. I think I understand the general concepts of adapting Xbase, but am unsure about how the setup/entrypoints for the interpreter and could not find any information regarding the error I am getting or how to resolve. Here are the relevant files for my situation:
Text.xtext:
import "http://www.eclipse.org/xtext/xbase/Xbase" as xbase
import "http://www.eclipse.org/xtext/common/JavaVMTypes" as types
Program returns Program:
{Program}
'program' name=ID '{'
variables=Var_Section?
run=XExpression?
'}'
;
Var_Section returns VarSection:
{VarSection}
'variables' '{'
decls+=XVariableDeclaration+
'}'
;
#Override // Change syntax
XVariableDeclaration returns xbase::XVariableDeclaration:
type=JvmTypeReference name=ID '=' right=XLiteral ';'
;
#Override // Do not allow declarations outside of variable region
XExpressionOrVarDeclaration returns xbase::XExpression:
XExpression;
TestJvmModelInferrer:
def dispatch void infer(Program element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
acceptor.accept(element.toClass(element.fullyQualifiedName)) [
documentation = element.documentation
if (element.variables !== null) {
for (decl : element.variables.decls) {
members += decl.toField(decl.name, decl.type) [
static = true
initializer = decl.right
visibility = JvmVisibility.PUBLIC
]
}
}
if (element.run !== null) {
members += element.run.toMethod('main', typeRef(Void::TYPE)) [
parameters += element.run.toParameter("args", typeRef(String).addArrayTypeDimension)
visibility = JvmVisibility.PUBLIC
static = true
body = element.run
]
}
]
}
Test case:
#Inject ParseHelper<Program> parseHelper
#Inject extension ValidationTestHelper
#Inject XbaseInterpreter interpreter
#Test
def void basicInterpret() {
val result = parseHelper.parse('''
program program1 {
variables {
int var1 = 0;
double var2 = 3.4;
}
var1 = 13
}
''')
result.assertNoErrors
var interpretResult = interpreter.evaluate(result.run)
println(interpretResult.result)
Partial stack trace:
java.lang.IllegalStateException: Could not access field: program1.var1 on instance: null
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter._assignValueTo(XbaseInterpreter.java:1262)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.assignValueTo(XbaseInterpreter.java:1221)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter._doEvaluate(XbaseInterpreter.java:1213)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.doEvaluate(XbaseInterpreter.java:216)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.internalEvaluate(XbaseInterpreter.java:204)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.evaluate(XbaseInterpreter.java:190)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.evaluate(XbaseInterpreter.java:180)
The interpreter does only support expressions, but does not work with types that are created by a JvmModelInferrer. Your code tries to work with fields of such an inferred type.
Rather than using the interpreter, I'd recommend to use an InMemoryCompiler in your test. The domainmodel example may serve as an inspiration: https://github.com/eclipse/xtext-eclipse/blob/c2b15c3ec118c4c200e2b28ea72d8c9116fb6800/org.eclipse.xtext.xtext.ui.examples/projects/domainmodel/org.eclipse.xtext.example.domainmodel.tests/xtend-gen/org/eclipse/xtext/example/domainmodel/tests/XbaseIntegrationTest.java
You may find this project interesting, which (among other stuff) implements an interpreter for Xtend based on the Xbase interpreter. It might be a bit outdated, though, and also will not fully support all Xtend concepts. But it could be a starting point, and your contrbutions are welcome :-)
https://github.com/kbirken/xtendency
Related
I'm playing around with Antlr, designing a toy language, which I think is where most people start! - I had a question on how best to think about switching on token type.
consider a 'function call' in the language, where a function can consume a string, number or variable - for example like the below (project() is the function call)
project("ABC") vs project(123) vs project($SOME_VARIABLE)
I have the alteration operator in my grammar, so the grammar parses the right thing, but in the visitor code, it would be nice to tell the difference between the three versions of the above.
#Override
public ASTRoot visitCreateproj(projectmgmtParser.CreateprojContext ctx) {
try {
s1 = ctx.STRING_LITERAL().getText();
}catch(Exception e){}
try{
s2 = ctx.NUM().getText();
}catch(Exception e){}
System.out.println("Created Project via => " + ctx.getChild(1).toString());
}
The code above worked, depending on whether s1 or s2 are null, I can infer how I was called (with a literal or a number, I haven't shown the variable case above), but I'm interested if there is a better or more elegant way - for example switching on token type inside the visitor code to actually process the language.
The grammar I had for the above was
createproj: 'project('WS?(STRING_LITERAL|NUM)')';
and when I use the intellij antlr plugin, it seems to know the token type of the argument to the project() function - but I don't seem to be able to get to it from my code.
You could do something like this:
createproj
: 'project' '(' WS? param ')'
;
param
: STRING_LITERAL
| NUM
;
and in your visitor code:
#Override
public ASTRoot visitCreateproj(projectmgmtParser.CreateprojContext ctx) {
switch(ctx.param().start.getType()) {
case YourLexerName.STRING_LITERAL:
...
case YourLexerName.NUM:
...
...
}
}
so by inlining the token in the grammar I had originally, I've lost the opportunity to inspect it in the visitor code?
No really, you could also do it like this:
createproj
: 'project' '(' WS? param_token=(STRING_LITERAL | NUM) ')'
;
and could then do this:
#Override
public ASTRoot visitCreateproj(projectmgmtParser.CreateprojContext ctx) {
switch(ctx.param_token.getType()) {
case YourLexerName.STRING_LITERAL:
...
case YourLexerName.NUM:
...
...
}
}
Just make sure you don't mix lexer rules (tokens) and parser rules in your set param_token=( ... ). When it's a parser rule, ctx.param_token.getType() will fail (it must then be ctx.param_token.start.getType()). That is why I recommended adding an extra parser rule, because this would then still work:
param
: STRING_LITERAL
| NUM
| some_parser_rule
;
I have some classes (and will need quite a few more) that look like this:
use Unit;
class Unit::Units::Ampere is Unit
{
method TWEAK { with self {
.si = True;
# m· kg· s· A ·K· mol· cd
.si-signature = [ 0, 0, 0, 1, 0, 0, 0 ];
.singular-name = "ampere";
.plural-name = "ampere";
.symbol = "A";
}}
sub postfix:<A> ($value) returns Unit::Units::Ampere is looser(&prefix:<->) is export(:short) {
return Unit::Units::Ampere.new( :$value );
};
sub postfix:<ampere> ($value) returns Unit::Units::Ampere is looser(&prefix:<->) is export(:long) {
$value\A;
};
}
I would like to be able to construct and export the custom operators dynamically at runtime. I know how to work with EXPORT, but how do I create a postfix operator on the fly?
I ended up basically doing this:
sub EXPORT
{
return %(
"postfix:<A>" => sub is looser(&prefix:<->) {
#do something
}
);
}
which is disturbingly simple.
For the first question, you can create dynamic subs by returning a sub from another. To accept only an Ampere parameter (where "Ampere" is chosen programmatically), use a type capture in the function signature:
sub make-combiner(Any:U ::Type $, &combine-logic) {
return sub (Type $a, Type $b) {
return combine-logic($a, $b);
}
}
my &int-adder = make-combiner Int, {$^a + $^b};
say int-adder(1, 2);
my &list-adder = make-combiner List, {(|$^a, |$^b)};
say list-adder(<a b>, <c d>);
say list-adder(1, <c d>); # Constraint type check fails
Note that when I defined the inner sub, I had to put a space after the sub keyword, lest the compiler think I'm calling a function named "sub". (See the end of my answer for another way to do this.)
Now, on to the hard part: how to export one of these generated functions? The documentation for what is export really does is here: https://docs.perl6.org/language/modules.html#is_export
Half way down the page, they have an example of adding a function to the symbol table without being able to write is export at compile time. To get the above working, it needs to be in a separate file. To see an example of a programmatically determined name and programmatically determined logic, create the following MyModule.pm6:
unit module MyModule;
sub make-combiner(Any:U ::Type $, &combine-logic) {
anon sub combiner(Type $a, Type $b) {
return combine-logic($a, $b);
}
}
my Str $name = 'int';
my $type = Int;
my package EXPORT::DEFAULT {
OUR::{"&{$name}-eater"} := make-combiner $type, {$^a + $^b};
}
Invoke Perl 6:
perl6 -I. -MMyModule -e "say int-eater(4, 3);"
As hoped, the output is 7. Note that in this version, I used anon sub, which lets you name the "anonymous" generated function. I understand this is mainly useful for generating better stack traces.
All that said, I'm having trouble dynamically setting a postfix operator's precedence. I think you need to modify the Precedence role of the operator, or create it yourself instead of letting the compiler create it for you. This isn't documented.
I have an Xtext grammar which reads (in part):
grammar mm.ecxt.MMLanguage hidden(WS, COMMENT)
import "http://www.eclipse.org/emf/2002/Ecore" as ecore
...
Statement:
ConstantStatement |
VariableStatement |
LabeledStatement |
...
LabeledStatement:
EssentialHypothesisStatement |
...
ConstantStatement:
DOLLAR_C (constants+=ConstDecl)+ DOLLAR_DOT;
VariableStatement:
DOLLAR_V (variables+=VarDecl)+ DOLLAR_DOT;
EssentialHypothesisStatement:
name=LABEL DOLLAR_E (symbols+=[Decl|MATHSYMBOL])+ DOLLAR_DOT;
Decl: ConstDecl | VarDecl;
ConstDecl returns ConstDecl: name=MATHSYMBOL;
VarDecl returns VarDecl: name=MATHSYMBOL;
MATHSYMBOL: PARENOPEN | PARENCLOSE | QUESTIONMARK | COMPRESSED | TLABEL | WORD;
...
(The full grammar is MMLanguage.xtext from current commit 328a5e7 of https://github.com/marnix/metamath-eclipse-xtext/.)
My question: How do I highlight the symbols in an EssentialHypothesisStatement, by using a different color for constants and variables? So if the MATHSYMBOL refers to a ConstDecl, then it should be highlighted one way, and some other way for a VarDecl.
I've tried to create an ISemanticHighlightingCalculator in all kinds of ways, but I can't seem to detect what the actual reference type is, neither through the node model nor through the Ecore model. On the one hand, the grammar-related methods only tell me that the reference goes to a Decl. On the other hand, the Ecore model's EReferences tell me whether the target is a ConstDecl or a VarDecl, but there I can't find the location of the source MATHSYMBOL.
Note that I prefer to use the node model (as opposed to the Ecore model) since I also want to highlight comments, and for performance reasons I cannot afford multiple passes over the document.
What is a good/canonical/efficient/simple way to achieve this?
from EObject perspective have a look at org.eclipse.xtext.nodemodel.util.NodeModelUtils.findNodesForFeature(EObject, EStructuralFeature)
from the node model perspective you can use EObjectAtOffsetHelper
sample grammar
Model:
defs+=Def*
uses+=Use*
;
Def:
ADef | BDef;
ADef:
"adef" name=ID
;
BDef:
"bdef" name=ID
;
Use:
"use" def=[Def]
;
And here the Impl
public class MyDslSemanticHighlightingCalculator implements ISemanticHighlightingCalculator {
#Inject
private MyDslGrammarAccess ga;
#Inject
private EObjectAtOffsetHelper helper;
#Override
public void provideHighlightingFor(XtextResource resource, IHighlightedPositionAcceptor accoptor, CancelIndicator cancelIndicator) {
if (resource == null)
return;
IParseResult parseResult = resource.getParseResult();
if (parseResult == null || parseResult.getRootNode() == null)
return;
BidiTreeIterable<INode> tree = parseResult.getRootNode().getAsTreeIterable();
for (INode node : tree) {
if (cancelIndicator.isCanceled()) {
return;
}
if (node.getGrammarElement() instanceof CrossReference) {
if (ga.getUseAccess().getDefDefCrossReference_1_0() == node.getGrammarElement()) {
EObject target = helper.resolveElementAt(resource, node.getOffset());
if (target instanceof ADef) {
accoptor.addPosition(node.getOffset(), node.getLength(), HighlightingStyles.COMMENT_ID);
} else if (target instanceof BDef) {
accoptor.addPosition(node.getOffset(), node.getLength(), HighlightingStyles.STRING_ID);
}
}
}
}
}
}
I have a grammar that can parse expressions like 1+2-4 or 1+x-y, creating an appropriate structure on the fly which later, given a Map<String, Integer> with appropriate content, can be evaluated numerically (after parsing is complete, i.e. for x or y only known later).
Inside the grammar, there are also places where an expression that can be evaluated on the spot, i.e. does not contain variables, should occur. I figured I could parse these with the same logic, adding a boolean parameter variablesAllowed to the rule, like so:
grammar MiniExprParser;
INT : ('0'..'9')+;
ID : ('a'..'z'| 'A'..'Z')('a'..'z'| 'A'..'Z'| '0'..'9')*;
PLUS : '+';
MINUS : '-';
numexpr returns [Double val]:
expr[false] {$val = /* some evaluation code */ 0.;};
varexpr /*...*/:
expr[true] {/*...*/};
expr[boolean varsAllowed] /*...*/:
e=atomNode[varsAllowed] {/*...*/}
(PLUS e2=atomNode[varsAllowed] {/*...*/}
|MINUS e2=atomNode[varsAllowed] {/*...*/}
)* ;
atomNode[boolean varsAllowed] /*...*/:
(n=INT {/*...*/})
|{varsAllowed}?=> ID {/*...*/}
;
result:
(numexpr) => numexpr {System.out.println("Numeric result: " + $numexpr.val);}
|varexpr {System.out.println("Variable expression: " + $varexpr.text);};
However, the generated Java code does not compile. In the part apparently responsible for the final rule's syntactic predicate, varsAllowed occurs even although the variable is never defined at this level.
/* ... */
else if ( (LA3_0==ID) && ((varsAllowed))) {
int LA3_2 = input.LA(2);
if ( ((synpred1_MiniExprParser()&&(varsAllowed))) ) {
alt3=1;
}
else if ( ((varsAllowed)) ) {
alt3=2;
}
/* ... */
Am I using it wrong? (I am using Eclipse' AntlrIDE 2.1.2 with Antlr 3.5.2.)
This problem is part of the hoisting process the parser uses for prediction. I encountered the same problem and ended up with a member var (or static var for the C target) instead of a parameter.
I examine heterogeneous trees in ANTLR (using ANTLRWorks 1.4.2).
Here is the example of what I have already done in ANTLR.
grammar test;
options {
language = java;
output = AST;
}
tokens {
PROGRAM;
VAR;
}
#members {
class Program extends CommonTree {
public Program(int ttype) {
token = new CommonToken(ttype, "<start>");
}
}
}
start
: program var function
// Works fine:
//-> ^(PROGRAM program var function)
// Does not work (described below):
-> ^(PROGRAM<Program> program var function)
;
program
: 'program'! ID ';'!
;
var
: TYPE^ ID ';'!
;
function
: ID '('! ')'! ';'!
;
TYPE
: 'int'
| 'string'
;
ID
: ('a'..'z' | 'A'..'Z')+
;
WHITESPACE
: (' ' | '\t' '\n'| '\r' | '\f')+ {$channel = HIDDEN;}
;
Sample input:
program foobar;
int foo;
bar();
When I use rewrite rule ^(PROGRAM<Program> program var function), ANTLR stumbles over and I get AST like this:
Whereas when I use this rewrite rule ^(PROGRAM program var function) it works:
Could anyone explain where am I wrong, please? Frankly, I do not really get the idea of heterogeneous trees and how do I use <…> syntax in ANTLR.
What do r0 and r1 mean (first picture)?
I have no idea what these r0 and r1 mean: I don't use ANTLRWorks for debugging, so can't comment on that.
Also, language = java; causes ANTLR 3.2 to produce the error:
error(10): internal error: no such group file java.stg
error(20): cannot find code generation templates java.stg
error(10): internal error: no such group file java.stg
error(20): cannot find code generation templates java.stg
ANTLR 3.2 expects it to be language = Java; (capital "J"). But, by default the target is Java, so, mind as well remove the language = ... entirely.
Now, as to you problem: I cannot reproduce it. As I mentioned, I tested it with ANTLR 3.2, and removed the language = java; part from your grammar, after which everything went as (I) expected.
Enabling the rewrite rule -> ^(PROGRAM<Program> program var function) produces the following ATS:
and when enabling the rewrite rule -> ^(PROGRAM program var function) instead, the following AST is created:
I tested both rewrite rules this with the following class:
import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;
public class Main {
public static void main(String[] args) throws Exception {
ANTLRStringStream in = new ANTLRStringStream("program foobar; int foo; bar();");
testLexer lexer = new testLexer(in);
CommonTokenStream tokens = new CommonTokenStream(lexer);
testParser parser = new testParser(tokens);
testParser.start_return returnValue = parser.start();
CommonTree tree = (CommonTree)returnValue.getTree();
DOTTreeGenerator gen = new DOTTreeGenerator();
StringTemplate st = gen.toDOT(tree);
System.out.println(st);
}
}
And the images are produced using graph.gafol.net (and the output of the Main class, of course).