How has a Language like Apache Velocity to be parsed with Antlr4? - apache

Question

i am working on a grammar to parse apache velocity on my own and i ran into the issue that i am not able to detect normal text neither the markup.
I am getting this message during the first line of the source.
line 1:0 extraneous input '// ${Name}.java' expecting {BREAK, FOREACH, IF, INCLUDE, PARSE, SET, STOP, '#[[', RAW_TEXT, '$'}
The input '// ${Name}.Java' should be tokenized to RAW_TEXT '$' '{' IDENTIFIER '}' RAW_TEXT. The parser rules should be rawText reference rawText. These parser rules are statements.
This is my source file. It is a java template in this case but the source file could or might be also a html template like mentioned in the user guide of apache velocity.
// ${Name}.java
#foreach ( $vertice in $Vertices )
#if ( $vertice.Type == "Class" )
public class $vertice.Name {
#foreach ( $edge in $Edges )
#if ( $edge.from == $vertice.Name)
// From $edge.from to $edge.to
private $edge.to $edge.to.toLowerCase();
public $edge.to get{$edge.to}() {
return this.${edge.to.toLowerCase()};
}
public void set${edge.to}(${edge.to} new${edge.to}) {
$edge.to old${edge.to} = this.${edge.to.toLowerCase()};
if (old${edge.to} != new${edge.to}) {
if (old${edge.to} != null) {
this.${edge.to.toLowerCase()} = null;
old${edge.to}.set${edge.from}(null);
}
this.${edge.to.toLowerCase()} = new${edge.to};
if (new${edge.to} != null) {
new${edge.to}.set${edge.from}(this);
}
}
}
public $edge.from with${edge.to}(${edge.to} new${edge.to}) {
this.set${edge.to}(new${edge.to});
return this;
}
#end
#end
}
#end
#end
This is my grammar.
grammar Velocity;
/* -- Parser Rules --- */
/*
* Start Rule
*/
template
: statementSet EOF?
;
/*
* Statements
*/
statementSet
: statement+
;
statement
: rawText # RawTextStatement
| unparsed # UnparsedStatement
| reference # ReferenceStatement
| setDirective # SetStatement
| ifDirective # IfStatement
| foreachDirective # ForeachStatement
| includeDirective # IncludeStatement
| parseDirective # ParseStatement
| breakDirective # BreakStatement
| stopDirective # StopStatement
;
rawText
: RAW_TEXT
;
unparsed
: UNPARSED UnparsedText=(TEXT | NL)* UNPARSED_END
;
setDirective
: SET '(' assignment ')'
;
ifDirective
: ifPart (elseifPart)* (elsePart)? END
;
foreachDirective
: FOREACH '(' variableReference 'in' enumerable ')' statementSet END
;
includeDirective
: INCLUDE '(' stringValue (',' stringValue)* ')'
;
parseDirective
: PARSE '(' stringValue ')'
;
breakDirective
: BREAK
;
stopDirective
: STOP
;
/*
* Expressions
*/
assignment
: assignableReference '=' expression
;
expression
: reference # ReferenceExpression
| string # StringLiteralExpression
| NUMBER # NumberLiteralExpression
| array # ArrayExpression
| map # MapExpression
| range # RangeExpression
| arithmeticOperation # ArithmeticOperationExpression
| booleanOperation # BooleanOperationExpression
;
enumerable
: array
| map
| range
| reference
;
stringValue
: string # StringValue_String
| reference # StringValue_Reference
;
/*
* References
*/
reference
: DOLLAR Quiet='!'? (referenceType | '{' referenceType '}')
;
assignableReference
: DOLLAR Quiet='!'? (assignableReferenceType | '{' assignableReferenceType '}')
;
referenceType
: assignableReferenceType # ReferenceType_AssignableReferenceType
| methodReference # ReferenceType_MethodReference
;
assignableReferenceType
: variableReference # AssignableReferenceType_VariableReference
| propertyReference # AssignableReferenceType_PropertyReference
;
variableReference
: IDENTIFIER indexNotation?
;
propertyReference
: IDENTIFIER ('.' IDENTIFIER)+ indexNotation?
;
methodReference
: IDENTIFIER ('.' IDENTIFIER)* '.' IDENTIFIER '(' (expression (',' expression)*)? ')' indexNotation?
;
indexNotation
: '[' NUMBER ']' # IndexNotation_Number
| '[' reference ']' # IndexNotation_Reference
| '[' string ']' # IndexNotation_String
;
/*
* Parsed Types
*/
string
: '"' stringText* '"' # DoubleQuotedString
| '\'' TEXT? '\'' # SingleQuotedString
;
stringText
: TEXT # StringText_Text
| reference # StringText_Reference
;
/*
* Container Types
*/
array
: '[' (expression (',' expression)*)? ']'
;
map
: '{' (expression ':' expression (',' expression ':' expression))? '}'
;
range
: '[' n=NUMBER '..' m=NUMBER ']'
;
/*
* Arithmetic Operators
*/
arithmeticOperation
: sum
;
sum
: term (followingTerm)*
;
followingTerm
: Operator=('+' | '-') term
;
term
: factor (followingFactor)*
;
followingFactor
: Operator=('*' | '/' | '%') factor
;
factor
: NUMBER # Factor_Number
| reference # Factor_Reference
| '(' arithmeticOperation ')' # Factor_InnerArithmeticOperation
;
/*
* Boolean Operators
*/
booleanOperation
: disjunction
;
disjunction
: conjunction (followingConjunction)*
;
followingConjunction
: Operator=OR conjunction
;
conjunction
: booleanComparison (followingBooleanComparison)*
;
followingBooleanComparison
: Operator=AND booleanComparison
;
booleanComparison
: booleanFactor (followingBooleanFactor)*
;
followingBooleanFactor
: Operator=(EQUALS | NOT_EQUALS) booleanFactor
;
booleanFactor
: BOOLEAN # BooleanFactor_Boolean
| reference # BooleanFactor_Reference
| negation # BooleanFactor_Negation
| arithmeticComparison # BooleanFactor_ArithmeticComparison
| '(' booleanOperation ')' # BooleanFactor_InnerBooleanOperation
;
arithmeticComparison
: LeftHandSide=arithmeticOperation Operator=(EQUALS | NOT_EQUALS | GREATER_THAN | GREATER_THAN_OR_EQUAL_TO | LESS_THAN | LESS_THAN_OR_EQUAL_TO) RightHandSide=arithmeticOperation
;
negation
: NOT booleanFactor
;
/*
* Conditionals
*/
ifPart
: IF '(' booleanOperation ')' statementSet
;
elseifPart
: ELSEIF '(' booleanOperation ')' statementSet
;
elsePart
: ELSE statementSet
;
/* --- Lexer Rules --- */
/*
* Comments
*/
SINGLE_LINE_COMMENT
: '##' TEXT? NL -> skip
;
MULTI_LINE_COMMENT
: '#*' (TEXT | NL)* '*#' -> skip
;
COMMENT_BLOCK
: '#**' (TEXT | NL)* '*#' -> skip
;
/*
* Directives
*/
BREAK
: '#break'
| '#{break}'
;
DEFINE
: '#define'
| '#{define}'
;
ELSE
: '#else'
| '#{else}'
;
ELSEIF
: '#elseif'
| '#{elseif}'
;
END
: '#end'
| '#{end}'
;
EVALUATE
: '#evaluate'
| '#{evaluate}'
;
FOREACH
: '#foreach'
| '#{foreach}'
;
IF
: '#if'
| '#{if}'
;
INCLUDE
: '#include'
| '#{include}'
;
MACRO
: '#macro'
| '#{macro}'
;
PARSE
: '#parse'
| '#{parse}'
;
SET
: '#set'
| '#{set}'
;
STOP
: '#stop'
| '#{stop}'
;
UNPARSED
: '#[['
;
UNPARSED_END
: ']]#'
;
/*
* Identifier
*/
DOLLAR
: '$' -> more
;
IDENTIFIER
: CHARACTER+ (CHARACTER | INTEGER | HYPHEN | UNDERSCORE)*
;
/*
* Boolean Values
*/
TRUE
: 'true'
;
FALSE
: 'false'
;
/*
* Boolean Operators
*/
EQUALS
: '=='
| 'eq'
;
NOT_EQUALS
: '!='
| 'ne'
;
GREATER_THAN
: '>'
| 'gt'
;
GREATER_THAN_OR_EQUAL_TO
: '>='
| 'ge'
;
LESS_THAN
: '<'
| 'lt'
;
LESS_THAN_OR_EQUAL_TO
: '<='
| 'le'
;
OR
: '||'
;
AND
: '&&'
;
NOT
: '!'
| 'not'
;
/*
* Literals
*/
BOOLEAN
: TRUE
| FALSE
;
NUMBER
: '-'? INTEGER
| '-'? INTEGER '.' INTEGER
;
/*
* Content
*/
RAW_TEXT
: ~[*#$]+
;
TEXT
: (ESC | SAFE_CODE_POINT)+
;
fragment ESC
: '\\' (["\\/#$!bftrn] | UNICODE)
;
fragment UNICODE
: 'u' HEX HEX HEX HEX
;
fragment HEX
: [0-9a-fA-F]
;
fragment SAFE_CODE_POINT
: ~["\\\u0000-\u001F]
;
/*
* Atomic elements
*/
CHARACTER
: [a-zA-Z]+
;
INTEGER
: [0-9]+
;
HYPHEN
: '-'
;
UNDERSCORE
: '_'
;
NL
: '\r'
| '\n'
| '\r\n'
;
WS
: ('\t' | ' ' | '\r' | '\n' | '\r\n')+ -> skip
;
What details am i missing here? What has to be done to actually parse velocity code?
Best Regards
Update:
I have changed these lexer rules.
DOLLAR
: '$'
;
RAW_TEXT
: ~[*#$]*
;
TEXT
: (ESC | SAFE_CODE_POINT)*?
;
fragment SAFE_CODE_POINT
: ~[$"\\\u0000-\u001F]
;
And now i'm getting this messages.
[0] line 1:4 mismatched input '{Name}.java\r\n' expecting {'!', '{', IDENTIFIER}
[0] line 2:8 mismatched input ' ( ' expecting '('
[0] line 2:12 mismatched input 'vertice in ' expecting {'!', '{', IDENTIFIER}
[0] line 2:24 mismatched input 'Vertices )\r\n' expecting {'!', '{', IDENTIFIER}
[0] line 3:3 mismatched input ' ( ' expecting '('
[0] line 3:7 mismatched input 'vertice.Type == "Class" )\r\npublic class ' expecting {'!', '{', IDENTIFIER}
[0] line 4:14 mismatched input 'vertice.Name {\r\n\t' expecting {'!', '{', IDENTIFIER}
[0] line 5:9 mismatched input ' ( ' expecting '('
[0] line 5:13 mismatched input 'edge in ' expecting {'!', '{', IDENTIFIER}
[0] line 5:22 mismatched input 'Edges )\r\n\t' expecting {'!', '{', IDENTIFIER}
[0] line 6:4 mismatched input ' ( ' expecting '('
[0] line 6:8 mismatched input 'edge.from == ' expecting {'!', '{', IDENTIFIER}
[0] line 6:22 mismatched input 'vertice.Name)\r\n\t' expecting {'!', '{', IDENTIFIER}
It helped, but the lexer is still stealing the $ symbol and why is it expecting a '{' character while the input starts with a '{' character? I will have a look at this problem.