Passing data to form grammar rules in Perl 6 - raku

Not sure whether grammars are meant to do such things: I want tokens to be defined in runtime (in future — with data from a file). So I wrote a simple test code, and as expected it wouldn't even compile.
grammar Verb {
token TOP {
<root>
<ending>
}
token root {
(\w+) <?{ ~$0 (elem) #root }>
}
token ending {
(\w+) <?{ ~$0 (elem) #ending }>
}
}
my #root = <go jump play>;
my #ending = <ing es s ed>;
my $string = "going";
my $match = Verb.parse($string);
.Str.say for $match<root>;
What's the best way of doing such things in Perl 6?

To match any of the elements of an array, just write the name of the array variable (starting with a # sigil) in the regex:
my #root = <go jump play>;
say "jumping" ~~ / #root /; # Matches 「jump」
say "jumping" ~~ / #root 'ing' /; # Matches 「jumping」
So in your use-case, the only tricky part is passing the arrays from the code that creates them (e.g. by parsing data files), to the grammar tokens that need them.
The easiest way would probably be to make them dynamic variables (signified by the * twigil):
grammar Verb {
token TOP {
<root>
<ending>
}
token root {
#*root
}
token ending {
#*ending
}
}
my #*root = <go jump play>;
my #*ending = <ing es s ed>;
my $string = "going";
my $match = Verb.parse($string);
say $match<root>.Str;
Another way would be to pass a Capture with the arrays to the args adverb of method .parse, which will pass them on to token TOP, from where you can in turn pass them on to the sub-rules using the <foo(...)> or <foo: ...> syntax:
grammar Verb {
token TOP (#known-roots, #known-endings) {
<root: #known-roots>
<ending: #known-endings>
}
token root (#known) {
#known
}
token ending (#known) {
#known
}
}
my #root = <go jump play>;
my #ending = <ing es s ed>;
my $string = "going";
my $match = Verb.parse($string, args => \(#root, #ending));
say $match<root>.Str; # go

The approach you were taking could have worked but you made three mistakes.
Scoping
Lexical variable declarations need to appear textually before the compiler encounters their use:
my $foo = 42; say $foo; # works
say $bar; my $bar = 42; # compile time error
Backtracking
say .parse: 'going' for
grammar using-token {token TOP { \w+ ing}}, # Nil
grammar using-regex-with-ratchet {regex TOP {:ratchet \w+ ing}}, # Nil
grammar using-regex {regex TOP { \w+ ing}}; # 「going」
The regex declarator has exactly the same effect as the token declarator except that it defaults to doing backtracking.
Your first use of \w+ in the root token matches the entire input 'going', which then fails to match any element of #root. And then, because there's no backtracking, the overall parse immediately fails.
(Don't take this to mean that you should default to using regex. Relying on backtracking can massively slow down parsing and there's typically no need for it.)
Debugging
See https://stackoverflow.com/a/19640657/1077672
This works:
my #root = <go jump play>;
my #ending = <ing es s ed>;
grammar Verb {
token TOP {
<root>
<ending>
}
regex root {
(\w+) <?{ ~$0 (elem) #root }>
}
token ending {
(\w+) <?{ ~$0 (elem) #ending }>
}
}
my $string = "going";
my $match = Verb.parse($string);
.Str.say for $match<root>;
outputs:
go

Related

Can * be used in sym tokens for more than one character?

The example for sym shows * (WhateverCode) standing in for a single symbol
grammar Foo {
token TOP { <letter>+ }
proto token letter {*}
token letter:sym<P> { <sym> }
token letter:sym<e> { <sym> }
token letter:sym<r> { <sym> }
token letter:sym<l> { <sym> }
token letter:sym<*> { . }
}.parse("I ♥ Perl", actions => class {
method TOP($/) { make $<letter>.grep(*.<sym>).join }
}).made.say; # OUTPUT: «Perl␤»
It will, however, fail if we use it to stand in for a symbol composed of several letters:
grammar Foo {
token TOP { <action>+ % " " }
proto token action {*}
token action:sym<come> { <sym> }
token action:sym<bebe> { <sym> }
token action:sym<*> { . }
}.parse("come bebe ama").say; # Nil
Since sym, by itself, does work with symbols with more than one character, how can we define a default sym token that matches a set of characters?
Can * be used in sym tokens for more than one character? ... The example for sym shows * (WhateverCode) standing in for a single symbol
It's not a WhateverCode or Whatever.1
The <...> in foo:sym<...> is a quote words constructor, so the ... is just a literal string.
That's why this works:
grammar g { proto token foo {*}; token foo:sym<*> { <sym> } }
say g.parse: '*', rule => 'foo'; # matches
As far as P6 is concerned, the * in foo:sym<*> is just a random string. It could be abracadabra. I presume the writer chose * to represent the mental concept of "whatever" because it happens to match the P6 concept Whatever. Perhaps they were being too cute.
For the rest of this answer I will write JJ instead of * wherever the latter is just an arbitrary string as far as P6 is concerned.
The * in the proto is a Whatever. But that's completely unrelated to your question:
grammar g { proto token foo {*}; token foo:sym<JJ> { '*' } }
say g.parse: '*', rule => 'foo'; # matches
In the body of a rule (tokens and regexes are rules) whose name includes a :sym<...> part, you can write <sym> and it will match the string between the angles of the :sym<...>:
grammar g { proto token foo {*}; token foo:sym<JJ> { <sym> } }
say g.parse: 'JJ', rule => 'foo'; # matches
But you can write anything you like in the rule/token/regex body. A . matches a single character:
grammar g { proto token foo {*}; token foo:sym<JJ> { . } }
say g.parse: '*', rule => 'foo'; # matches
It will, however, fail if we use it to stand in for a symbol composed of several letters
No. That's because you changed the grammar.
If you change the grammar back to the original coding (apart from the longer letter:sym<...>s) it works fine:
grammar Foo {
token TOP { <letter>+ }
proto token letter {*}
token letter:sym<come> { <sym> }
token letter:sym<bebe> { <sym> }
token letter:sym<JJ> { . }
}.parse(
"come bebe ama",
actions => class { method TOP($/) { make $<letter>.grep(*.<sym>).join } })
.made.say; # OUTPUT: «comebebe␤»
Note that in the original, the letter:sym<JJ> token is waiting in the wings to match any single character -- and that includes a single space, so it matches those and they're dealt with.
But in your modification you added a required space between tokens in the TOP token. That had two effects:
It matched the space after "come" and after "bebe";
After the "a" was matched by letter:sym<JJ>, the lack of a space between the "a" and "m" meant the overall match failed at that point.
sym, by itself, does work with symbols with more than one character
Yes. All token foo:sym<bar> { ... } does is add:
A multiple dispatch alternative to foo;
A token sym, lexically scoped to the body of the foo token, that matches 'bar'.
how can we define a default sym token that matches a set of characters?
You can write such a sym token but, to be clear, because you don't want it to match a fixed string it can't use the <sym> in the body.(Because a <sym> has to be a fixed string.) If you still want to capture under the key sym then you could write $<sym>= in the token body as Håkon showed in a comment under their answer. But it could also be letter:whatever with $<sym>= in the body.
I'm going to write it as a letter:default token to emphasize that it being :sym<something> doesn't make any difference. (As explained above, the :sym<something> is just about being an alternative, along with other :baz<...>s and :bar<...>s, with the only addition being that if it's :sym<something>, then it also makes a <sym> subrule available in the body of the associated rule, which, if used, matches the fixed string 'something'.)
The winning dispatch among all the rule foo:bar:baz:qux<...> alternatives is chosen according to LTM logic among the rules starting with foo. So you need to write such a token that does not win as a longest token prefix but only matches if nothing else matches.
To immediately go to the back of the pack in an LTM race, insert a {} at the start of the rule body2:
token letter:default { {} \w+ }
Now, from the back of the pack, if this rule gets a chance it'll match with the \w+ pattern, which will stop the token when it hits a non-word character.
The bit about making it match if nothing else matches may mean listing it last. So:
grammar Foo {
token TOP { <letter>+ % ' ' }
proto token letter {*}
token letter:sym<come> { <sym> } # matches come
token letter:sym<bebe> { <sym> } # matches bebe
token letter:boo { {} \w**6 } # match 6 char string except eg comedy
token letter:default { {} \w+ } # matches any other word
}.parse(
"come bebe amap",
actions => class { method TOP($/) { make $<letter>.grep(*.<sym>).join } })
.made.say; # OUTPUT: «comebebe␤»
that just can't be the thing causing it ... "come bebe ama" shouldn't work in your grammar
The code had errors which I've now fixed and apologize for. If you run it you'll find it works as advertised.
But your comment prodded me to expand my answer. Hopefully it now properly answers your question.
Footnote
1 Not that any of this has anything to do with what's actually going on but... In P6 a * in "term position" (in English, where a noun belongs, in general programming lingo, where a value belongs) is a Whatever, not a WhateverCode. Even when * is written with an operator, eg. +* or * + *, rather than on its own, the *s are still just Whatevers, but the compiler automatically turns most such combinations of one or more *s with one or more operators into a sub-class of Code called a WhateverCode. (Exceptions are listed in a table here.)
2 See footnote 2 in my answer to SO "perl6 grammar , not sure about some syntax in an example".
The :sym<...> contents are for the reader of your program, not for the compiler, and are used to distinguish multi tokens of otherwise identical names.
It just so happened that programmers started to write grammars like this:
token operator:sym<+> { '+' }
token operator:sym<-> { '-' }
token operator:sym</> { '/' }
To avoid duplicating the symbols (here +, -, /), a special rule <sym> was introduced that matches whatever is inside :sym<...> as a literal, so you can write the above tokens as
token operator:sym<+> { <sym> }
token operator:sym<-> { <sym> }
token operator:sym</> { <sym> }
If you don't use <sym> inside the regex, you are free to write anything you want inside :sym<...>, so you can write something like
token operator:sym<fallback> { . }
Maybe like this:
grammar Foo {
token TOP { <action>+ % " " }
proto token action {*}
token action:sym<come> { <sym> }
token action:sym<bebe> { <sym> }
token action:sym<default> { \w+ }
}.parse("come bebe ama").say;
Output:
「come bebe ama」
action => 「come」
sym => 「come」
action => 「bebe」
sym => 「bebe」
action => 「ama」

Perl6 Regex Match Num

I would like to match any Num from part of a text string. So far, this (stolen from from https://docs.perl6.org/language/regexes.html#Best_practices_and_gotchas) does the job...
my token sign { <[+-]> }
my token decimal { \d+ }
my token exponent { 'e' <sign>? <decimal> }
my regex float {
<sign>?
<decimal>?
'.'
<decimal>
<exponent>?
}
my regex int {
<sign>?
<decimal>
}
my regex num {
<float>?
<int>?
}
$str ~~ s/( <num>? \s*) ( .* )/$1/;
This seems like a lot of (error prone) reinvention of the wheel. Is there a perl6 trick to match built in types (Num, Real, etc.) in a grammar?
If you can make reasonable assumptions about the number, like that it's delimited by word boundaries, you can do something like this:
regex number {
« # left word boundary
\S+ # actual "number"
» # right word boundary
<?{ defined +"$/" }>
}
The final line in this regex stringifies the Match ("$/"), and then tries to convert it to a number (+). If it works, it returns a defined value, otherwise a Failure. This string-to-number conversion recognizes the same syntax as the Perl 6 grammar. The <?{ ... }> construct is an assertion, so it makes the match fail if the expression on the inside returns a false value.

How can I pass arguments to a Perl 6 grammar?

In Edit distance: Ignore start/end, I offered a Perl 6 solution to a fuzzy fuzzy matching problem. I had a grammar like this (although maybe I've improved it after Edit #3):
grammar NString {
regex n-chars { [<.ignore>* \w]**4 }
regex ignore { \s }
}
The literal 4 itself was the length of the target string in the example. But the next problem might be some other length. So how can I tell the grammar how long I want that match to be?
Although the docs don't show an example or using the $args parameter, I found one in S05-grammar/example.t in roast.
Specify the arguments in :args and give the regex an appropriate signature. Inside the regex, access the arguments in a code block:
grammar NString {
regex n-chars ($length) { [<.ignore>* \w]**{ $length } }
regex ignore { \s }
}
class NString::Actions {
method n-chars ($/) {
put "Found $/";
}
}
my $string = 'The quick, brown butterfly';
loop {
state $from = 0;
my $match = NString.subparse(
$string,
:rule('n-chars'),
:actions(NString::Actions),
:c($from++),
:args( \(5) )
);
last unless ?$match;
}
I'm still not sure about the rules for passing the arguments though. This doesn't work:
:args( 5 )
I get:
Too few positionals passed; expected 2 arguments but got 1
This works:
:args( 5, )
But that's enough thinking about this for one night.

Can I insert named captures in the Match tree without actually matching anything?

I was curious if I could insert things into the Match tree without actually anything. There's no associated problem I'm trying to solve.
In this example, I have a token market that checks that its match is a key in the hash. I was trying to then insert the value of that hash into the match tree somehow. I figured I could have a token that always matches, long_market_string, and then look into the tree somehow to see what market had matched.
grammar OrderNumber::Grammar {
token TOP {
<channel> <product> <market> <long_market_string> '/' <revision>
}
token channel { <[ M F P ]> }
token product { <[ 0..9 A..Z ]> ** 4 }
token market {
(<[ A..Z ]>** 1..2) <?{ %Market_Shortcode{$0}:exists }>
}
# this should figure out what market matched
# I don't particularly care how this happens as long as
# I can insert this into the match tree
token long_market_string { <?> }
token revision { <[ A..C ]> }
}
Is there some way to mess with the Match tree as it is being created?
I could do something that inverts things:
grammar AppleOrderNumber::Grammar {
token TOP {
<channel> <product> <long_market_string> '/' <revision>
}
token channel { <[ M F P ]> }
token product { <[ 0..9 A..Z ]> ** 4 }
token market {
(<[ A..Z ]>** 1..2) <?{ %Market_Shortcode{$0}:exists }>
}
token long_market_string { <market> }
token revision { <[ A..C ]> }
}
But, that handles that case. I'm more interested in inserting an arbitrary number of things.
Tokens are a type of method, so if you wrote a method that did all of the setup work that a token does for you, you could do almost anything.
This is not specced, and is currently not easy.
( I only have a vague idea of where to start looking in the source code to figure it out )
What you can do easily is add to the .made/.ast of the result
( .made and .ast are synonyms )
$/ = grammar {
token TOP {
.*
{
make 'World'
}
}
}.parse('Hello');
say "$/ $/.made()"; # Hello World
It doesn't even have to be inside of a grammar
'asdf' ~~ /{make 42}/;
say $/; # 「」
say $/.made # 42
Most of the time you would use an Actions class for this type of thing
grammar example-grammar {
token TOP {
[ <number> | <word> ]+ % \s*
}
token word {
<.alpha>+
}
token number {
\d+
{ make +$/ }
}
}
class example-actions {
method TOP ($/) { make $/.pairs.map:{ .key => .value».made} }
method number ($/) { #`( already done in grammar, so this could be removed ) }
method word ($/) { make ~$/ }
}
.say for example-grammar.parse(
'Hello 123 World',
:actions(example-actions)
).made».perl
# :number([123])
# :word(["Hello", "World"])
It sounds like you want to subvert the match tree into doing something the match tree isn't really supposed to do. The match tree tracks what substrings were matched where in the input string, not arbitrary data generated by the parser. If you want to track arbitrary data, what's wrong with the AST tree?
Sure, in one sense the AST tree has to mirror the parse tree, since it's constructed in a bottom-up fashion as the match methods complete successfully. But the AST itself, in the sense of "the object attached to any given node" is not so restricted. Consider for example:
grammar G {
token TOP { <foo> <bar> {make "TOP is: " ~ $<foo> ~ $<bar>} }
token foo { foo {make "foo"} }
token bar { bar {make "bar"} }
}
G.parse("foobar");
Here $/.made will simply be the string "TOP is: foobar" while the match tree has child nodes with the components that were used to construct the top-level AST. If then return to your example, we can make it:
grammar G {
my %Market_Shortcode = :AA('Double A');
token TOP {
<channel> <product> <market>
{} # Force the computation of the $/ object. Note that this will also terminate LTM here.
<long_market_string(~$<market>)> '/' <revision>
}
token channel { <[ M F P ]> }
token product { <[ 0..9 A..Z ]> ** 4 }
token market {
(<[ A..Z ]>** 1..2) <?{ %Market_Shortcode{$0}:exists }>
}
token long_market_string($shortcode) { <?> { say 'c='~$shortcode; make %Market_Shortcode{$shortcode} } }
token revision { <[ A..C ]> }
}
G.parse('M0000AA/A');
$<long_market_string>.ast will now be 'Double A'. Of course, I'd probably dispense with token long_market_name and just make the AST of token market whatever is in %Market_Shortcode (or a Market object with both short and long name, if you want to track both at once).
A less trivial example of this kind of thing would be something like a grammar of Python. Since Python's block level structure is line-based, your grammar (and thus match tree) need to reflect this in some way. But you can also chain several simple statements together on a single line by separating them with semi-colons. Now, you'll probably want the AST of a block to be a list of statements, while the AST of a single line may itself be a list of several statements. Thus you'd construct the AST of the block by (for example) flatmaping together the list of the lines (or something along those lines, depending on how you represent block statements like if and while).
Now, if you really, really, really want to do nasty things to the match tree I'm pretty sure it can be done, of course. You'll have to implement the parsing code yourself with method long_market_name, the API for which is undocumented and internal, and will likely involve at least some dropping down into nqp::ops. The stuff pointed to here will probably be useful. Other relevant files are src/core/{Match,Cursor}.pm in the Rakudo repo. Note also that the stringification of Matches is computed by extracting the matched substring from the input string, so if you want it to stringify usefully, you'll have to subclass Match.

parse string with pairs of values into hash the Perl6 way

I have a string which looks like that:
width=13
height=15
name=Mirek
I want to turn it into hash (using Perl 6). Now I do it like that:
my $text = "width=13\nheight=15\nname=Mirek";
my #lines = split("\n", $text);
my %params;
for #lines {
(my $k, my $v) = split('=', $_);
%params{$k} = $v;
}
say %params.perl;
But I feel there should exist more concise, more idiomatic way to do that. Is there any?
In Perl, there's generally more than one way to do it, and as your problem involves parsing, one solution is, of course, regexes:
my $text = "width=13\nheight=15\nname=Mirek";
$text ~~ / [(\w+) \= (\N+)]+ %% \n+ /;
my %params = $0>>.Str Z=> $1>>.Str;
Another useful tool for data extraction is comb(), which yields the following one-liner:
my %params = $text.comb(/\w+\=\N+/)>>.split("=").flat;
You can also write your original approach that way:
my %params = $text.split("\n")>>.split("=").flat;
or even simpler:
my %params = $text.lines>>.split("=").flat;
In fact, I'd probably go with that one as long as your data format does not become any more complex.
If you have more complex data format, you can use grammar.
grammar Conf {
rule TOP { ^ <pair> + $ }
rule pair {<key> '=' <value>}
token key { \w+ }
token value { \N+ }
token ws { \s* }
}
class ConfAct {
method TOP ($/) { make (%).push: $/.<pair>».made}
method pair ($/) { make $/.<key>.made => $/.<value>.made }
method key ($/) { make $/.lc }
method value ($/) { make $/.trim }
}
my $text = " width=13\n\theight = 15 \n\n nAme=Mirek";
dd Conf.parse($text, actions => ConfAct.new).made;