Antlr setText not working in the way I expected - antlr

Question

I have a requirement to convert an identifier into a beanutil string for retrieving an item from an object. The the identifiers to string conversions look like:
name ==> name
attribute.name ==> attributes(name)[0].value
attribute.name[2] ==> attributes(name)[2].value
address.attribute.postalcode ==> contactDetails.addresses[0].attributes(postalcode)[0].value
address[2].attribute.postalcode ==> contactDetails.addresses[2].attributes(postalcode)[0].value
address[2].attribute.postalcode[3] ==> contactDetails.addresses[2].attributes(postalcode)[3].value
Now I have decided to do this using antlr as I feel its probably going to be just as quick as using a set of 'if' statements. Feel free to tell me I'm wrong.
Right now, I've got this partial working using antlr, however once I start doing the 'address' ones, the setText part seems to stop working for Attribute.
Am I doing this the correct way or is there a better way of using antlr to get the result I want?
grammar AttributeParser;
parse returns [ String result ]
: Address EOF { $result = $Address.text; }
| Attribute EOF { $result = $Attribute.text; }
| Varname EOF { $result = $Varname.text; }
;
Address
: 'address' (Arraypos)* '.' Attribute { setText("contactDetails.addresses" + ($Arraypos == null ? "[0]" : $Arraypos.text ) + "." + $Attribute.text); }
;
Attribute
: 'attribute.' Varname (Arraypos)* { setText("attributes(" + $Varname.text + ")" + ($Arraypos == null ? "[0]" : $Arraypos.text ) + ".value"); }
;
Arraypos
: '[' Number+ ']'
;
Varname
: ('a'..'z'|'A'..'Z')+
;
Number
: '0'..'9'+
;
Spaces
: (' ' | '\t' | '\r' | '\n')+ { setText(" "); }
;
Below are two unit tests, the first returns what I expect, the second doesn't.
#Test
public void testSimpleAttributeWithArrayRef() throws Exception {
String source = "attribute.name[2]";
ANTLRStringStream in = new ANTLRStringStream(source);
AttributeParserLexer lexer = new AttributeParserLexer(in);
CommonTokenStream tokens = new CommonTokenStream(lexer);
AttributeParserParser parser = new AttributeParserParser(tokens);
String result = parser.parse();
assertEquals("attributes(name)[2].value", result);
}
#Test
public void testAddress() throws Exception {
String source = "address.attribute.postalcode";
ANTLRStringStream in = new ANTLRStringStream(source);
AttributeParserLexer lexer = new AttributeParserLexer(in);
CommonTokenStream tokens = new CommonTokenStream(lexer);
AttributeParserParser parser = new AttributeParserParser(tokens);
String result = parser.parse();
System.out.println("Result: " + result);
assertEquals("contactDetails.addresses[0].attributes(postalcode)[0].value", result);
}

Answer 1

No, you can't do (Arraypos)* and then refer to the contents as this: $Arraypos.text.
I wouldn't go changing the inner text of the tokens, but create a couple of parser rules and let them return the appropriate text.
A little demo:
grammar AttributeParser;
parse returns [String s]
: input EOF {$s = $input.s;}
;
input returns [String s]
: address {$s = $address.s;}
| attribute {$s = $attribute.s;}
| Varname {$s = $Varname.text;}
;
address returns [String s]
: Address arrayPos '.' attribute
{$s = "contactDetails.addresses" + $arrayPos.s + "." + $attribute.s;}
;
attribute returns [String s]
: Attribute '.' Varname arrayPos
{$s = "attributes(" + $Varname.text + ")" + $arrayPos.s + ".value" ;}
;
arrayPos returns [String s]
: Arraypos {$s = $Arraypos.text;}
| /* nothing */ {$s = "[0]";}
;
Attribute : 'attribute';
Address : 'address';
Arraypos : '[' '0'..'9'+ ']';
Varname : ('a'..'z' | 'A'..'Z')+;
which can be tested with:
import org.antlr.runtime.*;
public class Main {
public static void main(String[] args) throws Exception {
String[][] tests = {
{"name", "name"},
{"attribute.name", "attributes(name)[0].value"},
{"attribute.name[2]", "attributes(name)[2].value"},
{"address.attribute.postalcode", "contactDetails.addresses[0].attributes(postalcode)[0].value"},
{"address[2].attribute.postalcode", "contactDetails.addresses[2].attributes(postalcode)[0].value"},
{"address[2].attribute.postalcode[3]", "contactDetails.addresses[2].attributes(postalcode)[3].value"}
};
for(String[] test : tests) {
String input = test[0];
String expected = test[1];
AttributeParserLexer lexer = new AttributeParserLexer(new ANTLRStringStream(input));
AttributeParserParser parser = new AttributeParserParser(new CommonTokenStream(lexer));
String output = parser.parse();
if(!output.equals(expected)) {
throw new RuntimeException(output + " != " + expected);
}
System.out.printf("in = %s\nout = %s\n\n", input, output, expected);
}
}
}
And to run the demo do:
java -cp antlr-3.3.jar org.antlr.Tool AttributeParser.g
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar Main
which will print the following to the console:
in = name
out = name
in = attribute.name
out = attributes(name)[0].value
in = attribute.name[2]
out = attributes(name)[2].value
in = address.attribute.postalcode
out = contactDetails.addresses[0].attributes(postalcode)[0].value
in = address[2].attribute.postalcode
out = contactDetails.addresses[2].attributes(postalcode)[0].value
in = address[2].attribute.postalcode[3]
out = contactDetails.addresses[2].attributes(postalcode)[3].value
EDIT
Note that you can also let parser rules return more than just one object like this:
bar
: foo {System.out.println($foo.text + ", " + $foo.number);}
;
foo returns [String text, int number]
: 'FOO' {$text = "a"; $number = 1;}
| 'foo' {$text = "b"; $number = 2;}
;