JavaCC: Defining a *password* token or grammar rule - passwords

Question

I'm using JavaCC do simulate a small part of SQL grammars, and I'm having a problem with defining a password.
I'm writting grammar rules for a
CREATE USER user_name IDENTIFIED BY a_password
statement, and I'm stuck. Since a password can match with ANYTHING like asdkj*!##, or !#%^%ASDjnkj, _ASDJLJK##& etc. Note that in Oracle, it's totally legal to input your password without single quote mark ('). I could solve this problem easily if the quote marks are compulsory, but unfortunately they're not.
I've tried many ways to define a token/grammar rule for this password, but it didn't work as I expected, the latest rule I've tried is:
TOKEN : {
< S_PASSWORD: ( < DIGIT > | < LETTER > |< S_PASSCHAR >)+ >
| <#S_PASSCHAR : "!"|"#"|"#"|"$"|"%"|"^"|"&"|"*" >
| <#LETTER: ["a"-"z", "A"-"Z", "_"]>
| <#DIGIT: ["0" - "9"]>
}
But since < S_PASSWORD > can match ANYTHING, any other token that I defined earlier will be match with it, and I always get a JavaCC warning like this:
Warning: "#" cannot be matched as a string literal token at line 33515, column 13. It will be matched as < S_PASSWORD >.
There are similar suggestions from my friends, but they didn't work either.
Can someone help me with this?

Answer 1

Assuming there is some lexical way to tell where the password begins and ends, you can use lexical states. For example, if the sequence IDENTIFIED BY is only ever followed by spaces that are followed by a password, you make a state machine so that IDENTIFIED transitions from DEFAULT to S0. In S0 spaces are skipped and BY transitions to S1. In S1 spaces are skipped and a sequence of password characters is a PASSWORD token; the PASSWORD token transitions back to DEFAULT. Of course this only works if IDENTIFIED BY can only ever be followed by a password. Also in S0 you need to by able to deal with all the normal stuff, so most of your token rules should apply in both states S0 and DEFAULT but transition to DEFAULT. See the FAQ for more on lexical states.
If BY is only ever followed by a password, then it is even easier, as you don't need S0.
Edit
Here are some example rules. If the keyword BY is only ever followed by a password, you only need two states
TOKEN : { <BY : "BY"> : S1>
<S1> TOKEN : { <PASSWORD : ( <PASSWORDCHAR> )+ } : DEFAULT }
<DEFAULT, S1> : SKIP { " " } // Stays in the same state.
If you can use IDENTIFIED followed by BY then you need three states
<DEFAULT, S0> TOKEN : { <CREATE : "CREATE"> : DEFAULT } // And similar for most token rules.
<DEFAULT, S0> TOKEN : { <IDENTIFIED : "IDENTIFIED"> : S1 }
<S0> TOKEN : { <BY : "BY"> : S1> // BYs that follow IDENTIFIED
<DEFAULT> TOKEN : { <BY : "BY"> : DEFAULT } BYs that don't follow IDENTIFIED.
<S1> TOKEN : { <PASSWORD : ( <PASSWORDCHAR> )+ } : DEFAULT }
<DEFAULT, S0, S1> : SKIP { " " } // Stays in the same state.