Antlr parser SQL how to distinguish between a field and a table? - sql

Question

I write some Antlr4 rules to parse SQL,just want to distinguish between fields and tables.But they did something unexpected.My rule:
grammar Col;
stat : SELECT select_list FROM table_ref_list;
select_list : select_ele (',' select_ele)* ;
select_ele : Subquery_in_field_nor //subquery select column
| ID '(' .*? ')' //function calls
| NoDotId '(' .*? ')' NoDotId '(' .*? ')' //window function call
| ID //like column of tab
| DIGIT //number
| STRING //like this 'dad'
;
table_ref_list
: table_ref (',' table_ref)*
;
table_ref:table_block (NoDotId)?;
table_block : ID //So much like select_ele
| Subquery_in_field_nor
;
Subquery_in_field_nor : '(' (Subquery_in_field | ~[()])* ')'; //Resolving function nesting
Subquery_in_field : '(' .*? ')' ;
SELECT : [Ss][Ee][Ll][Ee][Cc][Tt];
FROM :[Ff][Rr][Oo][Mm];
NL : [ \r\n]+ ->skip;
ID : [A-Za-z] [A-Za-z0-9.]*;
NoDotId : [A-Za-z] [A-Za-z0-9]*;
DIGIT : ('-')? [0-9]+('.' [0-9]+)?;
STRING : '\'' .*? '\'';
And my sql file like this
SELECT substr(A.EMPNO,1,2),
A.ENAME,
'1',
'wwet',
18,
A.DEPTNO FROM EMP A
Display message
line 1:13 missing FROM at '(A.EMPNO,1,2)'
line 3:7 mismatched input ''1'' expecting {Subquery_in_field_nor, ID}
line 4:7 mismatched input ''wwet'' expecting {Subquery_in_field_nor, ID}
line 5:7 mismatched input '18' expecting {Subquery_in_field_nor, ID}
(stat SELECT (select_list (select_ele substr)) <missing FROM> (table_ref_list (table_ref (table_block (A.EMPNO,1,2))) , (table_ref (table_block A.ENAME)) , (table_ref (table_block '1')) , (table_ref (table_block 'wwet')) , (table_ref (table_block 18)) , (table_ref (table_block A.DEPTNO))))
I don't know why ?Why i user select_list : select_ele (',' select_ele)* ,but still Match table_ref?

Answer 1

substr(A.EMPNO,1,2) is not matched as a select_ele because it produces the tokens:
ID: substr
Subquery_in_field_nor: (A.EMPNO,1,2)
and not by any of the two alternatives:
| ID '(' .*? ')' //function calls
| NoDotId '(' .*? ')' NoDotId '(' .*? ')' //window function call
The part '(' .*? ')' is interpreted as: match a ( token, reluctantly followed by zero or more other tokens, ending with a ) token. But because ANTLR's lexer tries to grab as much characters as possible when tokenising the input, (A.EMPNO,1,2) will always be matched by a single Subquery_in_field_nor token.
Besides the greedy tokenisation in the lexer, you must also be aware that when 2 or more lexer rules can match the same characters, the one defined first "wins". So the input select can be matched by SELECT, ID and NoDotId. But since SELECT is defined first, that input will always become a SELECT token. However, the input foo can be matched by ID and NoDotId and since ID is defined first, it will win. You'll notice that there will never be a NoDotId token at all since whatever it matches is also matched by ID.
Here is a slightly modified grammar that works for your input:
grammar Col;
stat
: SELECT select_list FROM table_ref_list
;
select_list
: select_ele (',' select_ele)*
;
select_ele
: id Subquery_in_field_nor (id Subquery_in_field_nor)?
| id
| DIGIT
| STRING
;
table_ref_list
: table_ref (',' table_ref)*
;
table_ref
: table_block id?
;
table_block
: id
| Subquery_in_field_nor
;
id
: ID ('.' ID)*
;
Subquery_in_field_nor : '(' (Subquery_in_field | ~[()])* ')'; //Resolving function nesting
Subquery_in_field : '(' .*? ')' ;
SELECT : [Ss][Ee][Ll][Ee][Cc][Tt];
FROM : [Ff][Rr][Oo][Mm];
NL : [ \r\n]+ ->skip;
ID : [A-Za-z] [A-Za-z0-9]*;
DIGIT : ('-')? [0-9]+('.' [0-9]+)?;
STRING : '\'' .*? '\'';
Or better, grab an existing SQL grammar from the ANTLR4 Github repo: https://github.com/antlr/grammars-v4 Be aware that all of these grammars are opensource contributions from the community: test them properly, because many of them will have limitations (both in accuracy and in performance).