how can I define the lex pattern ( ), or ( /* rem / ), and ( / foo / 100 / foo */ )
in using gnu (f)lex tool.
_space [ \t]
id [a-zA-Z_]+[a-zA-Z0-9_]
digit [0-9]
math_ops [\+\-\/\*\^\%]
rem_expr (({_space}*)*|("/*".*"*/")*|("//".*)*|([\n]*))*
arr_digid ("("*({digit}*|{id}*)*")"*){arr_expr1}*{math_ops}+
arr_expr1 {rem_expr}*{digit}*{rem_expr}*
arr_expr2 {rem_expr}*
%%
\({arr_expr2}*\) {
return _REM_;
}
\({arr_expr1}*\) {
return _PATTERN2_;
}
Generally, you do not return comments or whitespace from a lexer. Why would you? They are, by definition, not part of the semantics of the program you are trying to parse.
On the whole, the easiest way to deal with them is to just ignore them. Below, the first pattern matches any whitespace character other than newline (Use [[:space:]] to also ignore newlines), and the second one is a way of matching C-style comments. ("/*".*"*/" doesn't work because it will match from the beginning of the first comment on a line to the end of the last one.)
[[:blank:]] ;
[/][*][^*]*[*]+([^/*][^*]*[*]+)[/] ;
The fact that the patterns do not have an action (or, in general, do not have a return statement in their action) means that the (f)lex-generated scanner will simply proceed to analyze the next token.
Some other notes:
It's really not necessary to define a shortcut for every pattern. There is no problem with putting a pattern directly in the lex actions. And you certainly don't need to define shortcuts for character classes which already have shortcuts (like [[:blank:]] and [[:digit:]].
You don't need to backslash escape characters inside a character class, although with a couple of characters order is important. (That's why I used [*] in the C-comment pattern; I could equally have used "*" or \*, but I personally prefer [*].) So you could have defined:
math_ops [+/*^%-]
The - must go either at the end or the beginning of the list; ^ cannot go at the beginning, and (though you don't use it) ] would have to go at the beginning. The only character which requires backslash-escaping is a backslash itself.
However, my preference is always to let single-character tokens be handled with a single default rule at the end:
. { return yytext[0]; }
This is much more maintainable, and avoids the need to invent arbitrary token names for single-character tokens. You can just use a single-quoted character in your bison/yacc file.
Related
I am trying to avoid matching whitespace at the end of a string while still matching whitespace in the middle of words.
Here is an example of a regex that matches underscores within x but does not match up to three trailing underscores.
say 'x_x___x________' ~~ /
[
| 'x'
| '_' <!before [
| $
| '_' <?before $>
| '_' <?before ['_' <?before $>]>
| '_' <?before ['_' <?before ['_' <?before $>]>]>
# ...
]>
]+
/;
Is there a way to construct the rest of the pattern implied by the ...?
It is a little difficult to discern what you are asking for.
You could be looking for something as simple as this:
say 'x_x___x________' ~~ / 'x'+ % '_' ** 1..3 /
# ｢x_x___x｣
or
say 'x_x___x________' ~~ / 'x'+ % '_' ** 1..2 /
# ｢x_x｣
or
say 'x_x___x________' ~~ / 'x'+ % '_'+ /
# ｢x_x___x｣
I would suggest using a Capture..., thusly:
'x_x___x________' ~~ /(.*?) _* $/;
say $0; #｢x_x___x｣
(The ? modifier makes the * 'non-greedy'.)
Please let me know if I have missed the point!
avoid matching whitespace at the end of a string while still matching whitespace in the middle of words
Per Brad's answer, and your comment on it, something like this:
/ \w+ % \s+ /
what I'm looking for is a way to match arbitrarily long streams that end with a known pattern
Per #user0721090601's comment on your Q, and as a variant of #p6steve's answer, something like this:
/ \w+ % \s+ )> \s* $ /
The )> capture marker marks where capture is to end.
You can use arbitrary patterns on the left and right of that marker.
an infinite family of <!before> patterns
Generalizing to an infinite family of patterns of any type, whether they are zero-width or not, the most natural solution in a regex is iteration using any of the standard quantifiers that are open ended. For example, \s+ for one or more whitespace characters.[1] [2]
Is there a way to construct the rest of the pattern implied by the ...?
I'll generalize that to "Is there a way in a Raku regex to match some arbitrary pattern that could in theory be recognized by a computer program?"
The answer is always "Yes":
While Raku rules/regexes might look like traditional regexes they are in fact arbitrary functions embedded in an arbitrary program over which you ultimately have full control.
Rules have arbitrary read access to capture state.[3]
Rules can do arbitrary turing complete computation.[4]
A collection of rules/regexes can arbitrarily consume input and drive the parse/match state, i.e. can implement any parser.
In short, if it can be matched/parsed by any program written in any programming language, it can be matched/parsed using Raku rules/regexes.
Footnotes
[1] If you use an open ended quantifier you do need to make sure that each match iteration/recursion either consumes at least one character, or fails, so that you avoid an infinite loop. For example, the * quantifier will succeed even if the pattern it qualifies does not match, so be careful that that won't lead to an infinite loop.
[2] Given the way you wrote your example, perhaps you are curious about recursion rather than iteration. Suffice to say, it's easy to do that too.[1]
[3] In Raku rules, captures form a hierarchy. There are two special variables that track the capture state of two key levels of this hierarchy:
$¢ is the capture state of the innermost enclosing overall capture. Think of it as something analogous to a return value being constructed by the current function call in a stack of function calls.
$/ is the capture state of the innermost enclosing capture. Think of it as something analogous to a value being constructed by a particular block of code inside a function.
For example:
'123' ~~ / 1* ( 2* { print "$¢ $/" } ) 3* { print "$¢ $/" } / ; # 1 2123 123
The overall / ... / is analogous to an ordinary function call. The first 1 and first 123 of the output show what has been captured by that overall regex.
The ( ... ) sets up an inner capture for a part of the regex. The 2* { print "$¢ $/" } within it is analogous to a block of code. The 2 shows what it has captured.
The final 123 shows that, at the top level of the regex, $/ and $¢ have the same value.
[4] For example, the code in footnote 3 above includes arbitrary code inside the { ... } blocks. More generally:
Rules can be invoked recursively;
Rules can have full signatures and pass arguments;
Rules can contain arbitrary code;
Rules can use multiple dispatch semantics for resolution. Notably, this can include resolution based on longest match length.
I’m wondering if Raku’s trim() routines might suit your purpose, for example: .trim, .trim-trailing or even .trim-leading. In the Raku REPL:
> say 'x x x ' ~~ m:g/ 'x'+ \s* /;
(｢x ｣ ｢x ｣ ｢x ｣)
> say 'x x x '.trim-trailing ~~ m:g/ 'x'+ \s* /;
(｢x ｣ ｢x ｣ ｢x｣)
HTH.
https://docs.raku.org/routine/trim https://docs.raku.org/routine/trim-trailing https://docs.raku.org/routine/trim-leading
As part of my grammar I have:
rule EX1 { <EX2> ( '/' <EX2>)* }
In my actions class I have written:
method EX1($/) {
my #ex2s = map *.made, $/.<EX2>;
my $ex1 = #ex2s.join('|');
#say "EX1 making $ex1";
$/.make($ex1);
}
So basically I am just trying to join all the EX2's together with a '|' between them instead of a '/'. However something is not right with my code, as it only picks up the first EX2, not the subsequent ones. How do I find out what the optional ones are?
TL;DR Your action method would work if your rule created the data structure the method is expecting. So we'll fix the rule and leave the method alone.
The main problem
Let's assume the EX1 rule is slotted into a working grammar; a string has been successfully parsed; the substring ex2/ex2/ex2 matched the EX1 rule; and we've displayed the corresponding part of the parse tree (by just saying the results of .parse using the grammar):
EX1 => ｢ex2/ex2/ex2｣
EX2 => ｢ex2｣
0 => ｢/ex2｣
EX2 => ｢ex2｣
0 => ｢/ex2｣
EX2 => ｢ex2｣
Note the extraneous 0 => captures and how the second and third EX2s are indented under them and indented relative to the first EX2. That's the wrong nesting structure relative to your method's assumptions.
Brad's solution to the main problem
As Brad++ points out in their comment responding to the first version of this answer, you can simply switch from the construct that both groups and captures ((...)) to the one that only groups ([...]).
rule EX1 { <EX2> [ '/' <EX2>]* }
Now the corresponding parse tree fragment for the same input string as above is:
EX1 => ｢ex2/ex2/ex2｣
EX2 => ｢ex2｣
EX2 => ｢ex2｣
EX2 => ｢ex2｣
The 0 captures are gone and the EX2s are now all siblings. For further discussion of when and why P6 nests captures the way it does, see jnthn's answer to Why/how ... capture groups?.
Your action method should now work -- for some inputs...
Håkon's solution to another likely problem
If Brad's solution works for some of the inputs you'd expect it to work for, but not all, part of the problem is likely how your rule matches between <EX2> and the / character.
As Håkon++ points out in their answer, your rule has spacing that probably doesn't do what you want.
If you don't intend the spacing in your pattern to be significant, then don't use a rule. In a token or regex all spaces in a pattern (ignoring inside a string eg ' ') is just to make your pattern more readable and isn't meaningful relative to any input string being matched. If in doubt, use a token (or regex) not a rule:
token EX1 { <EX2> ( '/' <EX2>)* }
🡅 🡅 🡅 🡅 🡅 🡅
Spacing indicated with 🡅 is NOT significant. You could omit it or extend it and it'll make no difference to how the rule matches input. It's only for readability.
In contrast, the entire point of the rule construct is that whitespace after each atom and each quantifier in a pattern is significant. Such spacing implicitly applies a (user overridable) boundary matching rule (by default a rule that allows whitespace and/or a transition between "word" and non-"word" characters) after the corresponding substring in the input.
In your EX1 rule, which I repeat below with exaggerated spacing to ensure clarity, some of the spacing is not significant, just as it isn't in a token or regex:
rule EX1 { <EX2> ( '/' <EX2>)* }
🡅 🡅 🡅
As before 🡅 indicates spacing that is NOT significant -- you can omit or extend it and it'll make no difference. The thing to remember is that spaces at the start of a pattern (or sub-pattern) is just for readability. (Experience from use showed that it was much better if any spacing there is not treated as significant.)
But spacing or lack of spacing after an atom or quantifier is significant:
This spacing is significant: ⮟ ⮟ ⮟
rule EX1 { <EX2> ( '/' <EX2>)* }
This LACK of spacing is significant: ⮝⮝
By writing your rule as you did you're telling P6 to match input with boundary matching (which by default allows whitespace) only:
after the first <EX2> (and thus before the first /);
between / and subsequent <EX2> matches;
after the last <EX2> match.
So your rule tells P6 to allow spaces between a / and <EX2> match when they occur in that order -- /, then <EX2>.
But it also tells P6 to not allow spaces the other way around -- between an <EX2> match and a / match in that order! Except with the very first <EX2> '/' pair!! P6 will let you declare match patterns of arbitrary complexity, including spacing, but I doubt this is what you meant or want.
For a complete listing of what "after an atom" means (i.e. when whitespace in rules is significant) see When is white space really important in Perl6 grammars?.
This significant spacing feature is:
Classic Perl DWIMery designed to make life easier;
Idiomatic -- used in most grammars because it does indeed make life easier;
The only reason the rule declarator exists (this significant whitespace aspect is the only difference between a rule and a token);
Completely optional because you can just use a token instead.
If someone reading this thinks they'd rather not take advantage of this significant space feature, then they can just use tokens instead. (This in turn will likely lead them to see why rule exists as an option, and then, or perhaps later, to see why it works the way it does, and to appreciate its DWIMery anew. :) )
The built in construct for the pattern you're matching
Finally, here's the idiomatic way to write the pattern you're trying to match:
rule EX1 { <EX2> + % '/' }
This tells P6 to match one or more <EX2>s separated by / characters. See Modified quantifier: %, %% for an explanation of this nice construct.
This is still a rule so most of the spacing in it remains significant. The precise details for when it is and isn't are at their most apparently fiddly for this construct because it has up to three significant spacers and one that's not:
NOT significant: ⮟ ⮟
rule EX1 { <EX2> + % '/' }
Significant: ⮝ ⮝ ⮝
Including spacing both before and after the + is redundant:
rule EX1 { <EX2> + % '/' }
rule EX1 { <EX2> +% '/' } # same match result
rule EX1 { <EX2>+ % '/' } # same match result
White space is significant in rules. So I think you are missing a whitespace after the last <EX2>:
rule EX1 { <EX2> ( '/' <EX2>)+ }
It should be:
rule EX1 { <EX2> ( '/' <EX2> )+ }
This allows for space to separate the terms in EX1.
I have two questions. Is the behavior I show correct, and if so, is it documented somewhere?
I was playing with the grammar TOP method. Declared as a rule, it implies beginning- and end-of-string anchors along with :sigspace:
grammar Number {
rule TOP { \d+ }
}
my #strings = '137', '137 ', ' 137 ';
for #strings -> $string {
my $result = Number.parse( $string );
given $result {
when Match { put "<$string> worked!" }
when Any { put "<$string> failed!" }
}
}
With no whitespace or trailing whitespace only, the string parses. With leading whitespace, it fails:
<137> worked!
<137 > worked!
< 137 > failed!
I figure this means that rule is applying :sigspace first and the anchors afterward:
grammar Foo {
regex TOP { ^ :sigspace \d+ $ }
}
I expected a rule to allow leading whitespace, which would happen if you switched the order:
grammar Foo {
regex TOP { :sigspace ^ \d+ $ }
}
I could add an explicit token in rule for the beginning of the string:
grammar Number {
rule TOP { ^ \d+ }
}
Now everything works:
<137> worked!
<137 > worked!
< 137 > worked!
I don't have any reason to think it should be one way or the other. The Grammars docs say two things happen, but the docs do not say which order these effects apply:
Note that if you're parsing with .parse method, token TOP is automatically anchored
and
When rule instead of token is used, any whitespace after an atom is turned into a non-capturing call to ws.
I think the answer is that the rule isn't actually anchored in the pattern sense. It's the way .parse works. The cursor has to start at position 0 and end at the last position in the string. That's something outside of the pattern.
The behavior is intended, and is a culmination of these language features:
Sigspace ignores whitespace before the first atom.
From the design docs1 (S05: Regexes and Rules, line 348, emphasis added):
The new :s (:sigspace) modifier causes certain whitespace sequences to be considered "significant"; they are replaced by a whitespace matching rule, . Only whitespace sequences immediately following a matching construct (atom, quantified atom, or assertion) are eligible. Initial whitespace is ignored at the front of any regex, to make it easy to write rules that can participate in longest-token-matching alternations. Trailing space inside the regex delimiters is significant.
This means:
rule TOP { \d+ }
^-------- <.ws> automatically inserted
rule TOP { ^ \d+ $ }
^---^-^---- <.ws> automatically inserted
Regexes are first-class compiled code with lexical scoping.
A regex/rule is not a string that may have characters concatenated to it later to change its behavior. It is a self-contained routine, which is parsed and has its behavior nailed down at compile time.
Regex modifiers like :sigspace, including the one implicitly added by the rule keyword, apply only to their lexical scope - i.e. to the fragment of source code they appear in at compile time. S05, line 6291:
The :i, :m, :r, :s, :dba, :Perl5, and Unicode-level modifiers can be placed inside the regex (and are lexically scoped)
The anchoring of rule TOP is done at run time by .parse.
S05, line 44231:
The .parse and .parsefile methods anchor to the beginning and ending of the text, and fail if the end of text is not reached. (The TOP rule can check against $ itself if it wishes to produce its own error message.)
I.e. the anchoring to the beginning of the string is not intrinsic to the rule TOP, and doesn't affect how the lexical scope of TOP is parsed and compiled. It is done when method .parse is called.
It has to be this way, because because the same grammar can be used with different starting rules instead of TOP, using .parse(..., rule => ...).
So when you write
rule TOP { \d+ }
it is compiled as
regex TOP { :r \d+ <.ws> }
And when you .parse that grammar, it effectively invokes the regex code ^ <TOP> $, with the anchors not being part of TOP's lexical scope but rather of a scope that merely calls the routine TOP. The combined behavior is as if the rule TOP had been written as:
regex TOP { ^ [:r :s \d+] $ }
1) The design docs are in general not to be taken as gospel for what is or isn't part of the Perl 6 language, but S05 is pretty accurate in that regard, except that it mentions some features that haven't been implemented yet but are planned. Anyone who wants to truly grok the intricacies of Perl 6 regexes/grammars, is IMO well served by reading the full S05 from top to bottom at least once.
There aren't two regex effects going on. The rule applies :sigspace. After that, the grammar is defined. When you call .parse, it starts at the beginning of the string and goes to the end (or fails). That anchoring isn't part of the grammar. It's part of how .parse applies the grammar.
My main issue was the odd way some of the things are worded in the docs. They aren't technically wrong, but they also tend to assume knowledge about things the reader might not know. In this case, the casual comment about anchoring TOP isn't as special as it seems. Any rule passed to .parse is anchored in the same way. There's no special behavior for that rule name other than it's the default value for :rule in a call to .parse.
I am writing a parser for Wolfram Language. The language has a concept of "named characters", which are specified by a name delimited by \[, and ]. For example: \[Pi].
Suppose I want to specify a regular expression for an identifier. Identifiers can include named characters. I see two ways to do it: one is to have a preprocessor that would convert all named characters to their unicode representation, and two is to enumerate all possible named characters in their source form as part of the regular expression.
The second approach does not seem feasible because there are a lot of named characters. I would prefer to have ranges of unicode characters in my regex.
So I want to preprocess my token stream. In other words, it seems to me that the lexer needs to check if the named characters syntax is correct and then look up the name and convert it to unicode.
But if the syntax is incorrect or the name does not exist I need to tell the user about it. How do I propagate this error to the user and yet let antlr4 recover from the error and resume? Maybe I can sort of "pipe" lexers/parsers? (I am new to antlr).
EDIT:
In Wolfram Language I can have this string as an identifier: \[Pi]Squared. The part between brackets is called "named character". There is a limited set of named characters, each of which corresponds to a unicode code point. I am trying to figure out how to tokenize identifiers like this.
I could have a rule for my token like this (simplified to just a combination of named characters and ASCII characters):
NAME : ('\\[' [a-z]+ ']'|[a-zA-Z])+ ;
but I would like to check if the named character actually exists (and other attributes such as if it is a letter, but the latter part is outside of the scope of the question), so this regex won't work.
I considered making a list of allowed named characters and just making a long regex that enumerates all of them, but this seems ugly.
What would be a good approach to this?
END OF EDIT
A common approach is to write the lexer/parser to allow syntactically correct input and defer semantic issues to the analysis of the generated parse tree. In this case, the lexer can naively accept named characters:
NChar : NCBeg .? RBrack ;
fragment NCBeg : '\\[' ;
fragment LBrack: '[' ;
fragment RBrack: ']' ;
Update
In the parser, allow the NChar's to exist in the parse-tree as discrete terminal nodes:
idents : ident+ ;
ident : NChar // named character string
| ID // simple character string?
| Literal // something quoted?
| ....
;
This makes analysis of the parse tree considerably easier: each ident context will contain only one non-null value for a discretely identifiable alt; and isolates analysis of all ordering issues to the idents context.
Update2
For an input \[Pi]Squared, the parse tree form that would be easiest to analyze would be an idents node with two well-ordered children, \[Pi] and Squared.
Best practice would not be to pack both children into the same token - would just have to later manually break the token text into the two parts to check if it is contains a valid named character and whether the particular sequence of parts is allowable.
No regex is going to allow conclusive verification of the named characters. That will require a list. Tightening the lexer definition of an NChar can, however, achieve a result equivalent to a regex:
NChar : NCBeg [A-Z][A-Za-z]+ RBrack ;
If the concern is that there might be a space after the named character, consider that this circumstance is likely better treated with a semantic warning as opposed to a syntactic error. Rather than skipping whitespace in the lexer, put the whitespace on the hidden channel. Then, in the verification analysis of each idents context, check the hidden channel for intervening whitespace and issue a warning as appropriate.
----
A parse-tree visitor can then examine, validate, and warn as appropriate regarding unknown or misspelled named characters.
To do the validation in the parser, if more desirable, use a predicated rule to distinguish known from unknown named characters:
#members {
ArrayList<String> keyList = .... // list of named chars
public boolean inList(String id) {
return keyList.contains(id) ;
}
}
nChar : known
| unknown
;
known : NChar { inList($NChar.getText()) }? ;
unknown : NChar { error("Unknown " + $NChar.getText()); } ;
The inList function could implement a distance metric to detect misspellings, but correcting the text directly in the parse-tree is a bit complex. Easier to do when implemented as a parse-tree decoration during a visitor operation.
Finally, a scrape and munge of the named characters into a usable map (both unicode and ascii) is likely worthwhile to handle both representations as well as conversions and misspelling.
I am using Lex and Yacc to design a parser and encounter some issue about comment.
I use the following Lex rule.
'#'[^('\r'|'\n')]* { /* do nothing */ }
It works, but at the end of execution all the comments are printed to the standard output. Is there way to clear that? Thank you for the suggestion.
The characters ', |, (, and ) have no special meaning in [], so you're only matching (and discarding) comments that don't contain them. In addition, in most versions of lex ' has no special meaning at all -- only " can be used to quote literal strings. What you probably want is:
"#"[^\r\n]* { /* do nothing */ }
In addition, # has no special meaning either, so there's no real need to quote it.
In general, if you're using lex (or flex) as the input to a parser, you NEVER want the default echoing behavior, so its best to add a 'catch-all' rule at the very end:
.|\n { fprintf(stderr, "Unexpected character '%c' in input\n", *yytext); }