So i am writing a small language and i am using antlrv4 as my tool. Antlr autogenerates lexer and parser files when u compile your grammar file(.g4). I am using javac btw. I want my language to have no semicolons and the way i want to do this is: if there is an identifier or ")" as the last token in a line, the lexer will automatically put the semicolon(Similar to what "go" language does). How would i approach something like this? There are other things like ATN(which i think is augmented transition network) and dfa(which i think is deterministic finite automaton) in the lexer file which i don't understand or how they relate to the lexing process?. Any help is appreciated. (btw i am still working on the grammar file so i don't have that fully completed).
Several points here: the ATN and the DFA are internal structures for parser + lexer and not something you would touch to change parsing behavior. Also, it's not clear to me why you want to have the lexer insert a semicolon at some point. What exactly do you want to accomplish by that (don't say: to make semicolons optional in the parser, I mean the underlying reason).
If you want to accept a command without a trailing semicolon you can make that optional:
assignment: simpleAssignment | complexAssignment SEMI?;
The parser will give you the content of the assignment rule regardless whether there is a trailing semicolon or not. Is that what you want?
Related
I am tasked to parse (and transform) a code of a computer language, that has a slight quirk in its rules, at least I see it this way. To be exact, the compiler treats new lines (as well as semicolons) as statement separators, but other than that (e.g. inside the statement) it treats them as spacers (whitespace).
As an example, this code:
try
local x = 5 / 0
catch (i)
print(i + "\n")
is proved to be equivalent to this:
try local x = 5 / 0 catch (i) print(i + "\n")
I don't see how I can express such a rule in EBNF, or specifically in Lark EBNF dialect. I mean in a sensible way. I probably could define all possible newline positions inside all statements, but it would be cumbersome and error-prone.
I wish to find a way to treat newlines contextually. Is there a proven method for this, preferably within Python/Lark domain? If I have to modify the parser for that purpose, then where should I start?
Or if I misunderstood something in this language in particular or in machine language parsing in general, or my statement of the problem is wrong, I'd also be happy to get educated.
(As you may guess, the language in question has a well proven implementation, but no officially defined grammar. Also, it is Squirrel, for all that it matters.)
The relevant quote from the "specification" is this:
A squirrel program is a simple sequence of statements.:
stats := stat [';'|'\n'] stats
[...] Statements can be separated with a new line or ‘;’ (or with the keywords case or default if inside a switch/case statement), both symbols are not required if the statement is followed by ‘}’.
These are relatively complex rules and in their totality not context free if newlines can also be ignored everywhere else. Note however that in my understanding the text implies that ; or \n are required when no of the other cases apply. That would make your example illegal. That probably means that the BNF as written is correct, e.g. both ; and \n are optionally everywhere. In that case you can (for lark) just put an %ignore "\n" statement and it should work fine.
Also, lark should not complain if you both ignore the \n and use it in a rule: Where useful it will match it in a rule, otherwise it will just ignore it. Note however that this breaks if you use a Terminal that includes the \n (e.g. WS or /\s/). Just have \n as an extra case.
(For the future: You will probably get faster response for lark questions if you ask over on gitter or at least put a link to SO there.)
I'm tryna do an ANTLR translator from Markdown format to HTML document and I found this problem when I try to recognize bold format. This is my ANTLR rule:
TxtNegrita : ('**' | '__') .*? ('**' | '__') {System.out.println('<span class="bold">' + getText() + '</span>');};
Unfortunately, the getText() function retrieves all the recognized String, including ** at the beginning and at the end of the String. Is it a way to delete that chars using ANTLR (obviously, in Java is perfectly possible).
Thanks!
You’ve created a Lexer rule which results in a single token. That is the expected behavior.
That rule looks more like something I’d expect in a parser rule.
(rules begin with upper case characters (conventionally all uppercase to make them stand out), and parser rules begin with lowercase letters and result in parse trees where each node has a context which gives you access to the component parts of your parser rule.
In ANTLR it is quite important to understand the difference between Lexer rules and parser rules.
Put simply... your input stream of characters is converted to an input stream of tokens using Lexer rules, and that stream of tokens is processed by parser rules.
Tokens are pretty much the “atoms” that parser rules deal with and their values are simply the string of characters that matched the Lexer Rule.
Is there any built-in tag for block, line or in-line comments for the parser generator?
e.g.
comment blocks "(*" Exp "*)" or inline comments "//" Exp.
In a parser generator like menhir, I would normally handle comments by pattern matching with the lexer, so comments wouldn't be part of the AST. Is there an equivalent in K?
If not, what is the recommended way of implementing comments?
You can declare the builtin sort #Layout to be the concatenation via pipes of a set of regular expression terminals (i.e., r"//[^\\n]*"). Any tokens which lex as one of these tokens are simply discarded by the lexer and the parser does not even see them. Note that this applies only to parsing terms using a generated parser or kast; parsing rules in .k files will still require the usual K syntax for comments.
Note that this is also how whitespace is parsed, so unless your language is whitespace sensitive, make sure to include in #Layout any whitespace characters which you want the parser to ignore.
When I was debugging my grammar for C# I noticed something very unusual: some inputs that are not accepted by a full grammar are being accepted by the same grammar with some independent rules deleted. I could not find a logical explanation. For example:
CS - this grammar does not accept the input a<a<a>><EOF>
CS' - and this grammar which is basically the same as CS but with some independent rules deleted (rules are not reordered) does accept a<a<a>><EOF>
As you can see both grammars start with the rule start: namespaceOrTypeName EOF; and therefore they should call the same set of rules (CS will never call those rules that are deleted in CS'). I spent a day debugging this, deleting or adding new rules, but couldn't find a flaw in the logic. Any help would be of use, thank you.
Unicode
EDIT:
After changing the start rule in CS to start: Identifier EOF; the grammar starts rejecting the input method which is normally accepted when only Identifier rules are defined. So I guess, since there is a rule attributeTarget: ...| 'method' | ..., that after compiling the grammar some phrases get reserved such as 'method' in this case but I'm not still sure if thats the case.
The first grammar includes the overloadableBinaryOperator rule which implicitly defines the >> token. Since >> is a 2-character token, the lexer will never treat the input >> as two separate 1-character tokens >, >. If you open the grammar in ANTLRWorks 2, you'll see a warning indicator for each implicitly-defined token. You should remove all of these warnings by:
Creating explicit lexer rules for every token you intend to appear in the input.
Only using the syntax 'new' in a parser rule if a corresponding lexer rule exists for the literal 'new'.
How to code grammar or lexer rule to describe JSP/EL identifier or string literal in ANTLR? Remember, that JSP/EL is Unicode and you cannot list all possible symbols in a rule. Also remember, that strings can contain EL expressions, which may be complex, so lexer is insufficient to describe them, parser is required, while ANTLR parser is unable to match character classes or any character.
Checkout the new "Lexical Modes":
Lexical Modes
Modes allow you to group lexical rules by context, such as inside and outside of XML tags. It’s like having multiple sublexers, one for context. The lexer can only return tokens matched by entering a rule in the current mode. Lexers start out in the so-called default mode. All rules are considered to be within the default mode unless you specify a mode command. Modes are not allowed within combined grammars, just lexer grammars.
-- http://www.antlr.org/wiki/display/ANTLR4/Lexer+Rules