Antlr3 .Net how to build a simple grammar - antlr

I'm trying to build a simple grammar to parse a .Net type name string, supporting generics. I admit to being completely new to building grammars in any language. A type string might look like the following.
Foo.Bar.Blah(Mom.Dad, Son.Daughter(Frank.Bob), Dog)
Basically, it's recursive. Ya'll should understand this.
I'm completely out in the woods with this one. Not sure how to begin. What I've built currently, which doesn't actually work, is this:
tree grammar XmlTypeName;
options {
language=CSharp2;
}
RPAREN
: '('
;
LPAREN
: ')'
;
SEP
: ','
;
TYPE
: ('a'..'z'|'A'..'Z'|'0'..'9'|'_')+
;
prog
: type;
type
: TYPE (RPAREN type (SEP type)? LPAREN)? (EOF)?
;
This doesn't even get close to working. Antlr3.exe throws errors saying that RPARAM and LPARAM aren't allowed in a tree parser. Is a tree parser even what I need?
I'd like to produce a simple AST that lets me navigate down the types.

No, you shouldn't use a tree grammar. A tree grammar is used after a parser has created an AST. Simply remove the keyword tree from it.
A couple of other remarks:
you want to match one or more comma separated types inside parenthesis, but you used type (SEP type)?, which matches one or two types. You'll need type (SEP type)* instead;
you didn't account for the . inside the types;
you should discard literal spaces in the lexer.
Something like this will do the trick, most probably:
grammar XmlTypeName;
options {
language=CSharp2;
}
prog
: type EOF
;
type
: name (RPAREN type (SEP type)* LPAREN)?
;
name
: ID (DOT ID)*
;
RPAREN
: '('
;
LPAREN
: ')'
;
SEP
: ','
;
DOT
: '.'
;
ID
: ('a'..'z'|'A'..'Z'|'0'..'9'|'_')+
;
SPACE
: (' '|'\t')+ {Skip();} // if 'Skip()' doesn't work, try 'skip()'
;
However, the above just creates a flat list of tokens. If you want to create a proper AST, you need to "tell" ANTLR which nodes/tokens are root tokens, and which ones to discard (like the comma's, parenthesis, ...).
grammar XmlTypeName;
options {
output=AST;
language=CSharp2;
}
tokens {
TYPE;
NAME;
}
prog
: type EOF -> type
;
type
: name (RPAREN type (SEP type)* LPAREN)? -> ^(TYPE name type*)
;
name
: ID (DOT ID)* -> ^(NAME ID+)
;
RPAREN
: '('
;
LPAREN
: ')'
;
SEP
: ','
;
DOT
: '.'
;
ID
: ('a'..'z'|'A'..'Z'|'0'..'9'|'_')+
;
SPACE
: (' '|'\t')+ {skip();}
;
which creates the following AST:
More info about creating AST's with ANTLR: How to output the AST built using ANTLR?

Related

What's wrong with this ANTLR grammar?

I want to parse query expressions that look like this:
Person Name=%John%
(Person Name=John% and Address=%Ontario%)
Person Fullname_3="John C. Smith"
But I'm totally new to Antlr4 and can't even figure out how to parse one single TABLE FIELD=QUERY clause. When I run the grammar below in Go as target, I get
line 1:7 mismatched input 'Name' expecting {'not', '(', FIELDNAME}
for a simple query like
Person Name=John
Why can't the Grammar parse FIELDNAME via parsing fieldsearch->field EQ searchterm->FIELDNAME?
I guess I'm misunderstanding something very fundamental here about how Antlr Grammars work, but what?
/* ANTLR Grammar for Minidb Query Language */
grammar Mdb;
start : searchclause EOF ;
searchclause
: table expr
;
expr
: fieldsearch
| unop fieldsearch
| LPAREN expr relop expr RPAREN
;
unop
: NOT
;
relop
: AND
| OR
;
fieldsearch
: field EQ searchterm
;
field
: FIELDNAME
;
table
: TABLENAME
;
searchterm
: STRING
;
AND
: 'and'
;
OR
: 'or'
;
NOT
: 'not'
;
EQ
: '='
;
LPAREN
: '('
;
RPAREN
: ')'
;
fragment VALID_ID_START
: ('a' .. 'z') | ('A' .. 'Z') | '_'
;
fragment VALID_ID_CHAR
: VALID_ID_START | ('0' .. '9')
;
TABLENAME
: VALID_ID_START VALID_ID_CHAR*
;
FIELDNAME
: VALID_ID_START VALID_ID_CHAR*
;
STRING: '"' ~('\n'|'"')* ('"' | { panic("syntax-error - unterminated string literal") } ) ;
WS
: [ \r\n\t] + -> skip
;
Try looking at the tokens produced for that input using grun Mdb tokens -tokens. It will tell you that the input consists of two table names, an equals sign and then another table name. To match your grammar it would have needed to be a table name, a field name, an equals sign and a string.
The first problem is that TABLENAME and FIELDNAME have the exact same definition. In cases where two lexer rules would produce a match of the same length on the current input, ANTLR prefers the one that comes first in the grammar. So it will never produce a FIELDNAME token. To fix that just replace both of those rules with a single ID rule. If you want to, you can then introduce parser rules tableName : ID ; and fieldName : ID ; if you want to keep the names.
The other problem is more straight forward: John simply does not match your rules for a string since it's not in quotes. If you do want to allow John as a valid search term, you might want to define it as searchterm : STRING | ID ; instead of only allowing STRINGs.

Making changes to lexer while parsing

I'm trying to parse N3 using ANTLR4;
In pseudo-ANTLR:
{plain_is = plain_of = false}
document : (keywords | triple)* ;
keywords : '#keywords' NCNAME (',' NCNAME)* '.'
{plain_is = 'is' in ncnames
plain_of = 'of' in ncnames} ;
triple : value predicate value '.' ;
predicate : value | KW_IS value KW_OF ;
value : QNAME | NCNAME ;
QNAME : [a-z]* ':' [a-z]+
NCNAME : [a-z]+ ;
KW_IS : {plain_is}? 'is' | {!plain_is}? '#is' ;
KW_OF : {plain_of}? 'of' | {!plain_of}? '#of' ;
WS : [ \t\r\n]+ -> skip ;
Ignoring the #keywords semantics for now, this is a valid document:
:x :y :z.
:z is :y of :z.
But if #keywords is used, all keywords not specified in the list must be prefixed with #, which allows using them as NCNAMEs:
#keywords is.
:z is :y #of of.
Is there a way to describe this in ANTLR? i.e. after parsing the keywords production, pass some flags (like plain_is) back to the lexer?
Currently I'm using a permissive grammar and check after parsing, converting the non-listed keywords back to NCNAMEs.
The parser does not have the ability to change the operation of the lexer. Although initiated by the parser, The lexer will process the entire input stream before the parser begins its evaluation of the parser rules.
You may be looking for lexer 'modes', aka lexical modes.

Antlr lexer tokens that match similar strings, what if the greedy lexer makes a mistake?

It seems that sometimes the Antlr lexer makes a bad choice on which rule to use when tokenizing a stream of characters... I'm trying to figure out how to help Antlr make the obvious-to-a-human right choice. I want to parse text like this:
d/dt(x)=a
a=d/dt
d=3
dt=4
This is an unfortunate syntax that an existing language uses and I'm trying to write a parser for. The "d/dt(x)" is representing the left hand side of a differential equation. Ignore the lingo if you must, just know that it is not "d" divided by "dt". However, the second occurrence of "d/dt" really is "d" divided by "dt".
Here's my grammar:
grammar diffeq_grammar;
program : (statement? NEWLINE)*;
statement
: diffeq
| assignment;
diffeq : DDT ID ')' '=' ID;
assignment
: ID '=' NUMBER
| ID '=' ID '/' ID
;
DDT : 'd/dt(';
ID : 'a'..'z'+;
NUMBER : '0'..'9'+;
NEWLINE : '\r\n'|'\r'|'\n';
When using this grammar the lexer grabs the first "d/dt(" and turns it to the token DDT. Perfect! Now later the lexer sees the second "d" followed by a "/" and says "hmmm, I can match this as an ID and a '/' or I can be greedy and match DDT". The lexer chooses to be greedy... but little does it know, there is no "(" a few characters later in the input stream. When the lexer looks for the missing "(" it throws a MismatchedTokenException!
The only solution I've found so far, is to move all the rules into the parser with a grammar like:
grammar diffeq_grammar;
program : (statement? NEWLINE)*;
statement
: diffeq
| assignment;
diffeq : ddt id ')' '=' id;
assignment
: id '=' number
| id '=' id '/' id
;
ddt : 'd' '/' 'd' 't' '(';
id : CHAR+;
number : DIGIT+;
CHAR : 'a'..'z';
DIGIT : '0'..'9';
NEWLINE : '\r\n'|'\r'|'\n';
This is a fine solution if I didn't already have thousands of lines of working code that depend on the first grammar working. After spending 2 days researching this problem I have come to the conclusion that a lexer... really ought to be able to distinguish the two cases. At some point the Antlr lexer is deciding between two rules: DDT and ID. It chooses DDT because the lexer is greedy. But when matching DDT fails, I'd like the lexer to go back to using ID.
I'm okay with using predicates or other tricks as long as the grammar remains basically the same (i.e., the rules in the lexer, stay in the lexer. And most rules are left untouched.).
Ideally I can modify the lexer rule for DDT with any valid Antlr code... and be done.
My target language is Java.
Thanks!
UPDATE
Thank you guys for some great answers!! I accepted the answer that best fit my question. The actual solution I used is in my own answer (not the accepted answer), and there are more answers that could have worked. Readers, check out all the answers; some of them may suit your case better than mine.
I'm okay with using predicates or other tricks as long as the grammar remains basically the same (i.e., the rules in the lexer, stay in the lexer. And most rules are left untouched.).
In that case, force the lexer to look ahead in the char-stream to make sure there really is "d/dt(" using a gated syntactic predicate.
A demo:
grammar diffeq_grammar;
#parser::members {
public static void main(String[] args) throws Exception {
String src =
"d/dt(x)=a\n" +
"a=d/dt\n" +
"d=3\n" +
"dt=4\n";
diffeq_grammarLexer lexer = new diffeq_grammarLexer(new ANTLRStringStream(src));
diffeq_grammarParser parser = new diffeq_grammarParser(new CommonTokenStream(lexer));
parser.program();
}
}
#lexer::members {
private boolean ahead(String text) {
for(int i = 0; i < text.length(); i++) {
if(input.LA(i + 1) != text.charAt(i)) {
return false;
}
}
return true;
}
}
program
: (statement? NEWLINE)* EOF
;
statement
: diffeq {System.out.println("diffeq : " + $text);}
| assignment {System.out.println("assignment : " + $text);}
;
diffeq
: DDT ID ')' '=' ID
;
assignment
: ID '=' NUMBER
| ID '=' ID '/' ID
;
DDT : {ahead("d/dt(")}?=> 'd/dt(';
ID : 'a'..'z'+;
NUMBER : '0'..'9'+;
NEWLINE : '\r\n' | '\r' | '\n';
If you now run the demo:
java -cp antlr-3.3.jar org.antlr.Tool diffeq_grammar.g
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar diffeq_grammarParser
(when using Windows, replace the : with ; in the last command)
you will see the following output:
diffeq : d/dt(x)=a
assignment : a=d/dt
assignment : d=3
assignment : dt=4
Although this is not what you are trying to do considering the large amount of working code that you have in the project, you should still consider separating your parser and lexer more thoroughly. I is best to let the parser and the lexer do what they do best, rather than "fusing" them together. The most obvious indication of something being wrong is the lack of symmetry between your ( and ) tokens: one is part of a composite token, while the other one is a stand-alone token.
If refactoring is at all an option, you could change the parser and lexer like this:
grammar diffeq_grammar;
program : (statement? NEWLINE)* EOF; // <-- You forgot EOF
statement
: diffeq
| assignment;
diffeq : D OVER DT OPEN id CLOSE EQ id; // <-- here, id is a parser rule
assignment
: id EQ NUMBER
| id EQ id OVER id
;
id : ID | D | DT; // <-- Nice trick, isn't it?
D : 'D';
DT : 'DT';
OVER : '/';
EQ : '=';
OPEN : '(';
CLOSE : ')';
ID : 'a'..'z'+;
NUMBER : '0'..'9'+;
NEWLINE : '\r\n'|'\r'|'\n';
You may need to enable backtracking and memoization for this to work (but try compiling it without backtracking first).
Here's the solution I finally used. I know it violates one of my requirements: to keep lexer rules in the lexer and parser rules in the parser, but as it turns out moving DDT to ddt required no change in my code. Also, dasblinkenlight makes some good points about mismatched parenthesis in his answer and comments.
grammar ddt_problem;
program : (statement? NEWLINE)*;
statement
: diffeq
| assignment;
diffeq : ddt ID ')' '=' ID;
assignment
: ID '=' NUMBER
| ID '=' ID '/' ID
;
ddt : ( d=ID ) { $d.getText().equals("d") }? '/' ( dt=ID ) { $dt.getText().equals("dt") }? '(';
ID : 'a'..'z'+;
NUMBER : '0'..'9'+;
NEWLINE : '\r\n'|'\r'|'\n';

Left-factoring grammar of coffeescript expressions

I'm writing an Antlr/Xtext parser for coffeescript grammar. It's at the beginning yet, I just moved a subset of the original grammar, and I am stuck with expressions. It's the dreaded "rule expression has non-LL(*) decision" error. I found some related questions here, Help with left factoring a grammar to remove left recursion and ANTLR Grammar for expressions. I also tried How to remove global backtracking from your grammar, but it just demonstrates a very simple case which I cannot use in real life. The post about ANTLR Grammar Tip: LL() and Left Factoring gave me more insights, but I still can't get a handle.
My question is how to fix the following grammar (sorry, I couldn't simplify it and still keep the error). I guess the trouble maker is the term rule, so I'd appreciate a local fix to it, rather than changing the whole thing (I'm trying to stay close to the rules of the original grammar). Pointers are also welcome to tips how to "debug" this kind of erroneous grammar in your head.
grammar CoffeeScript;
options {
output=AST;
}
tokens {
AT_SIGIL; BOOL; BOUND_FUNC_ARROW; BY; CALL_END; CALL_START; CATCH; CLASS; COLON; COLON_SLASH; COMMA; COMPARE; COMPOUND_ASSIGN; DOT; DOT_DOT; DOUBLE_COLON; ELLIPSIS; ELSE; EQUAL; EXTENDS; FINALLY; FOR; FORIN; FOROF; FUNC_ARROW; FUNC_EXIST; HERECOMMENT; IDENTIFIER; IF; INDENT; INDEX_END; INDEX_PROTO; INDEX_SOAK; INDEX_START; JS; LBRACKET; LCURLY; LEADING_WHEN; LOGIC; LOOP; LPAREN; MATH; MINUS; MINUS; MINUS_MINUS; NEW; NUMBER; OUTDENT; OWN; PARAM_END; PARAM_START; PLUS; PLUS_PLUS; POST_IF; QUESTION; QUESTION_DOT; RBRACKET; RCURLY; REGEX; RELATION; RETURN; RPAREN; SHIFT; STATEMENT; STRING; SUPER; SWITCH; TERMINATOR; THEN; THIS; THROW; TRY; UNARY; UNTIL; WHEN; WHILE;
}
COMPARE : '<' | '==' | '>';
COMPOUND_ASSIGN : '+=' | '-=';
EQUAL : '=';
LOGIC : '&&' | '||';
LPAREN : '(';
MATH : '*' | '/';
MINUS : '-';
MINUS_MINUS : '--';
NEW : 'new';
NUMBER : ('0'..'9')+;
PLUS : '+';
PLUS_PLUS : '++';
QUESTION : '?';
RELATION : 'in' | 'of' | 'instanceof';
RPAREN : ')';
SHIFT : '<<' | '>>';
STRING : '"' (('a'..'z') | ' ')* '"';
TERMINATOR : '\n';
UNARY : '!' | '~' | NEW;
// Put it at the end, so keywords will be matched earlier
IDENTIFIER : ('a'..'z' | 'A'..'Z')+;
WS : (' ')+ {skip();} ;
root
: body
;
body
: line
;
line
: expression
;
assign
: assignable EQUAL expression
;
expression
: value
| assign
| operation
;
identifier
: IDENTIFIER
;
simpleAssignable
: identifier
;
assignable
: simpleAssignable
;
value
: assignable
| literal
| parenthetical
;
literal
: alphaNumeric
;
alphaNumeric
: NUMBER
| STRING;
parenthetical
: LPAREN body RPAREN
;
// term should be the same as expression except operation to avoid left-recursion
term
: value
| assign
;
questionOp
: term QUESTION?
;
mathOp
: questionOp (MATH questionOp)*
;
additiveOp
: mathOp ((PLUS | MINUS) mathOp)*
;
shiftOp
: additiveOp (SHIFT additiveOp)*
;
relationOp
: shiftOp (RELATION shiftOp)*
;
compareOp
: relationOp (COMPARE relationOp)*
;
logicOp
: compareOp (LOGIC compareOp)*
;
operation
: UNARY expression
| MINUS expression
| PLUS expression
| MINUS_MINUS simpleAssignable
| PLUS_PLUS simpleAssignable
| simpleAssignable PLUS_PLUS
| simpleAssignable MINUS_MINUS
| simpleAssignable COMPOUND_ASSIGN expression
| logicOp
;
UPDATE:
The final solution will use Xtext with an external lexer to avoid to intricacies of handling significant whitespace. Here is a snippet from my Xtext version:
CompareOp returns Operation:
AdditiveOp ({CompareOp.left=current} operator=COMPARE right=AdditiveOp)*;
My strategy is to make a working Antlr parser first without a usable AST. (Well, it would deserve a separates question if this is a feasible approach.) So I don't care about tokens at the moment, they are included to make development easier.
I am aware that the original grammar is LR. I don't know how close I can stay to it when transforming to LL.
UPDATE2 and SOLUTION:
I could simplify my problem with the insights gained from Bart's answer. Here is a working toy grammar to handle simple expressions with function calls to illustrate it. The comment before expression shows my insight.
grammar FunExp;
ID: ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*;
NUMBER: '0'..'9'+;
WS: (' ')+ {skip();};
root
: expression
;
// atom and functionCall would go here,
// but they are reachable via operation -> term
// so they are omitted here
expression
: operation
;
atom
: NUMBER
| ID
;
functionCall
: ID '(' expression (',' expression)* ')'
;
operation
: multiOp
;
multiOp
: additiveOp (('*' | '/') additiveOp)*
;
additiveOp
: term (('+' | '-') term)*
;
term
: atom
| functionCall
| '(' expression ')'
;
When you generate a lexer and parser from your grammar, you see the following error printed to your console:
error(211): CoffeeScript.g:52:3: [fatal] rule expression has non-LL(*) decision due to recursive rule invocations reachable from alts 1,3. Resolve by left-factoring or using syntactic predicates or using backtrack=true option.
warning(200): CoffeeScript.g:52:3: Decision can match input such as "{NUMBER, STRING}" using multiple alternatives: 1, 3
As a result, alternative(s) 3 were disabled for that input
(I've emphasized the important bits)
This is only the first error, but you start with the first and with a bit of luck, the errors below that first one will also disappear when you fix the first one.
The error posted above means that when you're trying to parse either a NUMBER or a STRING with the parser generated from your grammar, the parser can go two ways when it ends up in the expression rule:
expression
: value // choice 1
| assign // choice 2
| operation // choice 3
;
Namely, choice 1 and choice 3 both can parse a NUMBER or a STRING, as you can see by the "paths" the parser can follow to match these 2 choices:
choice 1:
expression
value
literal
alphaNumeric : {NUMBER, STRING}
choice 3:
expression
operation
logicOp
relationOp
shiftOp
additiveOp
mathOp
questionOp
term
value
literal
alphaNumeric : {NUMBER, STRING}
In the last part of the warning, ANTLR informs you that it ignores choice 3 whenever either a NUMBER or a STRING will be parsed, causing choice 1 to match such input (since it is defined before choice 3).
So, either the CoffeeScript grammar is ambiguous in this respect (and handles this ambiguity somehow), or your implementation of it is wrong (I'm guessing the latter :)). You need to fix this ambiguity in your grammar: i.e. don't let the expression's choices 1 and 3 both match the same input.
I noticed 3 other things in your grammar:
1
Take the following lexer rules:
NEW : 'new';
...
UNARY : '!' | '~' | NEW;
Be aware that the token UNARY can never match the text 'new' since the token NEW is defined before it. If you want to let UNARY macth this, remove the NEW rule and do:
UNARY : '!' | '~' | 'new';
2
In may occasions, you're collecting multiple types of tokens in a single one, like LOGIC:
LOGIC : '&&' | '||';
and then you use that token in a parser rules like this:
logicOp
: compareOp (LOGIC compareOp)*
;
But if you're going to evaluate such an expression at a later stage, you don't know what this LOGIC token matched ('&&' or '||') and you'll have to inspect the token's inner text to find that out. You'd better do something like this (at least, if you're doing some sort of evaluating at a later stage):
AND : '&&';
OR : '||';
...
logicOp
: compareOp ( AND compareOp // easier to evaluate, you know it's an AND expression
| OR compareOp // easier to evaluate, you know it's an OR expression
)*
;
3
You're skipping white spaces (and no tabs?) with:
WS : (' ')+ {skip();} ;
but doesn't CoffeeScript indent it's code block with spaces (and tabs) just like Python? But perhaps you're going to do that in a later stage?
I just saw that the grammar you're looking at is a jison grammar (which is more or less a bison implementation in JavaScript). But bison, and therefor jison, generates LR parsers while ANTLR generates LL parsers. So trying to stay close to the rules of the original grammar will only result in more problems.

How do I make a TreeParser in ANTLR3?

I'm attemping to learn language parsing for fun...
I've created a ANTLR grammar which I believe will match a simple language I am hoping to implement. It will have the following syntax:
<FunctionName> ( <OptionalArguments>+) {
<OptionalChildFunctions>+
}
Actual Example:
ForEach(in:[1,2,3,4,5] as:"nextNumber") {
Print(message:{nextNumber})
}
I believe I have the grammar working correctly to match this construct, and now I am attemping to build an Abstract Syntax Tree for the language.
Firstly, I must admit I'm not exactly sure HOW this tree should look. Secondly, I'm at a complete loss how to do this in my Antlr grammar...I've been trying without much success for hours.
This is the current idea I'm going with for the tree:
FunctionName
/ \
Attributes \
/ \ / \
ID /\ ChildFunctions
/ \ ID etc
/ \
Attribute AttributeValue
Type
This is my current Antlr grammar file:
grammar Test;
options {output=AST;ASTLabelType=CommonTree;}
program : function ;
function : ID (OPEN_BRACKET (attribute (COMMA? attribute)*)? CLOSE_BRACKET)? (OPEN_BRACE function* CLOSE_BRACE)?;
attribute : ID COLON datatype;
datatype : NUMBER | STRING | BOOLEAN | array | lookup ;
array : OPEN_BOX (datatype (COMMA datatype)* )? CLOSE_BOX ;
lookup : OPEN_BRACE (ID (PERIOD ID)*) CLOSE_BRACE;
NUMBER
: ('+' | '-')? (INTEGER | FLOAT)
;
STRING
: '"' ( ESC_SEQ | ~('\\'|'"') )* '"'
;
BOOLEAN
: 'true' | 'TRUE' | 'false' | 'FALSE'
;
ID : (LETTER|'_') (LETTER | INTEGER |'_')*
;
COMMENT
: '//' ~('\n'|'\r')* '\r'? '\n' {$channel=HIDDEN;}
| '/*' ( options {greedy=false;} : . )* '*/' {$channel=HIDDEN;}
;
WHITESPACE : (' ' | '\t' | '\r' | '\n') {$channel=HIDDEN;} ;
COLON : ':' ;
COMMA : ',' ;
PERIOD : '.' ;
OPEN_BRACKET : '(' ;
CLOSE_BRACKET : ')' ;
OPEN_BRACE : '{' ;
CLOSE_BRACE : '}' ;
OPEN_BOX : '[' ;
CLOSE_BOX : ']' ;
fragment
LETTER
: 'a'..'z' | 'A'..'Z'
;
fragment
INTEGER
: '0'..'9'+
;
fragment
FLOAT
: INTEGER+ '.' INTEGER*
;
fragment
ESC_SEQ
: '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\')
;
ANY help / advice would be great. I've tried reading dozens of tutorials and nothing about the AST generation seems to stick :(
Step 1 is to make the tree look like the little graph that you posted. Right now, you don't have any tree construction operators, so you're going to end up with a flat list.
See tree construction on the antlr.org website.
You can use ANTLRWorks to see what your getting for a parse tree and AST. Start adding tree construction operators and watch how things change.
EDIT / Additional Info:
Here's a process you can follow to give you a rough idea of how to do it:
Download ANTLRWorks and use it's graphing facilities. You will definitely want to see the parse tree and the AST before and after you make changes. Once you understand how everything works, then you can use any IDE or editor you want.
There are two basic operators for tree construction - The exclamation mark ! which tells the compiler to not place the node within the AST, and the carot ^, which tells ANTLR to make something the root node. Start by going through each non-terminal rule and deciding which elements don't need to be in the AST. For example, you don't need commas or parenthesis. Once you have all the information you can populate the a structure (or create your own AST structure) that provides all the information. Commas don't help any more, so add a ! to them. For example:
function: ID (OPEN_BRACKET! (attribute (COMMA!? attribute)*)? CLOSE_BRACKET!)? (OPEN_BRACE! function* CLOSE_BRACE!)?;
Take a look at the AST in ANTLRWorks before and after. Compare.
Now decide which element should be the root node. It looks like you want ID to be the root node, so add a ^ after ID and compare in ANTLRWorks.
Here's a few changes that bring it closer to what I think you want:
program : function ;
function : ID^ (OPEN_BRACKET! attributeList? CLOSE_BRACKET!)? (OPEN_BRACE! function* CLOSE_BRACE!)?;
attributeList: (attribute (COMMA!? attribute)*);
attribute : ID COLON! datatype;
datatype : NUMBER | STRING | BOOLEAN | array | lookup ;
array : OPEN_BOX! (datatype^ (COMMA! datatype)* )? CLOSE_BOX!;
lookup : OPEN_BRACE! (ID (PERIOD! ID)*) CLOSE_BRACE!;
With that under your belt, now go look at some of the tutorials.