How should I bundle a library of text files with my module? - raku

I have the following structure in the resources directory in a module I'm building:
resources
|-- examples
|-- Arrays
| |-- file
|-- Lists
|-- file1
|-- file2
I have the following code to collect and process these files:
use v6.d;
unit module Doc::Examples::Resources;
class Resource {
has Str $.name;
has Resource #.resources;
has Resource %.resource-index;
method resource-names() {
#.resources>>.name.sort
}
method list-resources() {
self.resource-names>>.say;
}
method is-resource(Str:D $lesson) {
$lesson ~~ any self.resource-names;
}
method get-resource(Str:D $lesson) {
if !self.is-resource($lesson) {
say "Sorry, that lesson does not exist.";
return;
}
return %.resource-index{$lesson};
}
}
class Lesson is Resource {
use Doc::Parser;
use Doc::Subroutines;
has IO $.file;
method new(IO:D :$file) {
my $name = $file.basename;
self.bless(:$name, :$file)
}
method parse() {
my #parsed = parse-file $.file.path;
die "Failed parse examples from $.file" if #parsed.^name eq 'Any';
for #parsed -> $section {
my $heading = $section<meta>[0] || '';
my $intro = $section<meta>[1] || '';
say $heading.uc ~ "\n" if $heading && !$intro;
say $heading.uc if $heading && $intro;
say $intro ~ "\n" if $intro;
for $section<code>.Array {
die "Failed parse examples from $.file, check it's syntax." if $_.^name eq 'Any';
das |$_>>.trim;
}
}
}
}
class Topic is Resource {
method new(IO:D :$dir) {
my $files = dir $?DISTRIBUTION.content("$dir");
my #lessons;
my $name = $dir.basename;
my %lesson-index;
for $files.Array -> $file {
my $lesson = Lesson.new(:$file);
push #lessons, $lesson;
%lesson-index{$lesson.name} = $lesson;
}
self.bless(:$name, resources => #lessons, resource-index => %lesson-index);
}
}
class LocalResources is Resource is export {
method new() {
my $dirs = dir $?DISTRIBUTION.content('resources/examples');
my #resources;
my %resource-index;
for $dirs.Array -> $dir {
my $t = Topic.new(:$dir);
push #resources, $t;
%resource-index{$t.name} = $t;
}
self.bless(:#resources, :%resource-index)
}
method list-lessons(Str:D $topic) {
self.get-resource($topic).list-lessons;
}
method parse-lesson(Str:D $topic, Str:D $lesson) {
self.get-resource($topic).get-resource($lesson).parse;
}
}
It works. However, I'm told that this is not reliable and there there is no guarantee that lines like my $files = dir $?DISTRIBUTION.content("$dir"); will work after the module is installed or will continue to work into the future.
So what are better options for bundling a library of text files with my module that can be accessed and found by the module?

Files under the resources directory will always be available as keys to the %?RESOURCES compile-time variable if you declare them in the META6.json file this way:
"resources": [
"examples/Array/file",
]
and so on.

I've settled on a solution. As pointed out by jjmerelo, the META6.json file contains a list of resources and, if you use the comma IDE, the list of resources is automatically generated for you.
From within the module's code, the list of resources can be accessed via the $?DISTRIBUTION variable like so:
my #resources = $?DISTRIBUTION.meta<resources>
From here, I can build up my list of resources.
One note on something I discovered: the $?DISTRIBUTION variable is not accessible from a test script. It has to be placed inside a module in the lib directory of the distribution and exported.

Related

How can I recursively read a directory in Deno?

I'm trying to recursively read a file in Deno using Deno.readDir, but the example they provide only does the folder given:
for await (const entry of Deno.readDir(Deno.cwd())) {
console.log(entry.name);
}
How can I make this recursive?
Deno's standard library includes a function called walk for this purpose. It's available in std/fs/walk.ts. Here's an example:
/Users/deno/so-74953935/main.ts:
import { walk } from "https://deno.land/std#0.170.0/fs/walk.ts";
for await (const walkEntry of walk(Deno.cwd())) {
const type = walkEntry.isSymlink
? "symlink"
: walkEntry.isFile
? "file"
: "directory";
console.log(type, walkEntry.path);
}
Running in the terminal:
% pwd
/Users/deno/so-74953935
% ls -AF
.vscode/ deno.jsonc deno.lock main.ts
% ls -AF .vscode
settings.json
% deno --version
deno 1.29.1 (release, x86_64-apple-darwin)
v8 10.9.194.5
typescript 4.9.4
% deno run --allow-read main.ts
directory /Users/deno/so-74953935
file /Users/deno/so-74953935/main.ts
file /Users/deno/so-74953935/deno.jsonc
file /Users/deno/so-74953935/deno.lock
directory /Users/deno/so-74953935/.vscode
file /Users/deno/so-74953935/.vscode/settings.json
Since that function returns an async generator, you can make your own generator function that wraps around Deno.readDir:
(Do note that the example provided will join the path and name, giving you strings such as /directory/name.txt)
import { join } from "https://deno.land/std/path/mod.ts";
export async function* recursiveReaddir(
path: string
): AsyncGenerator<string, void> {
for await (const dirEntry of Deno.readDir(path)) {
if (dirEntry.isDirectory) {
yield* recursiveReaddir(join(path, dirEntry.name));
} else if (dirEntry.isFile) {
yield join(path, dirEntry.name);
}
}
}
for await (const entry of recursiveReaddir(Deno.cwd())) {
console.log(entry)
}
OR, you can use recursive_readdir, which is a 3rd party library in Deno made for this purpose.

Is it possible to create subroutine signatures at run time?

I've got a method in a class:
method options(*#opt) {
if !#!valid-options {
my $out = (cmd 'bin/md2html -h').out;
my #matches = $out ~~ m:g/\s'--'(<-[\s]>+)/;
for #matches -> $opt {
push #!valid-options, $opt[0].Str;
}
}
for #opt -> $opt {
when !($opt (elem) #!valid-options) {
warn "'$opt' is not a valid option";
}
push #!options, '--' ~ $opt;
}
}
The method checks that the options to see if they are valid and, if they are, places them into an attribute.
I pass args into the options method like this, as words:
$obj.options: <ftables ftasklists github>;
This works. But it got me wondering if it was possible pass in the options as named flags like this:
$obj.options: :ftables, :ftasklists, :github
But since I don't know all the command's options ahead of time, I'd need to generate the named arguments dynamically. Is this possible? I tried this but had no luck:
# create a signature
my #params = Parameter.new(name => ':$option', type => Bool, :!default);
my $sig = Signature.new(:#params);
my &blah = -> $sig { say 'this works too' } ;
&blah(:option1);
Currently, there is no way to do that, short of using EVAL.
You can add a slurpy hash to any sub signature to catch all unexpected named arguments:
sub foo(*%_) { .say for %_.keys }
foo :bar, :baz; # bar baz
Creating your own signatures at runtime may become possible / easier when the RakuAST has landed.

How to override the NQPMatch.Str function

... Or how to change $<sigil>.Str value from token sigil { ... } idependently from the matched text. Yes I'm asking how to cheat grammars above (i.e. calling) me.
I am trying to write a Slang for Raku without sigil.
So I want the nogil token, matching anything <?> to return NqpMatch that stringifies: $<sigil>.Str to '$'.
Currently, my token sigil look like that
token sigil {
| <[$#%&]>
| <nogil> { say "Nogil returned: ", lk($/, 'nogil').Str; # Here It should print "$"
}
}
token nogil-proxy {
| '€'
| <?>
{log "No sigil:", get-stack; }
}
And the method with that should return a NQPMatch with method Str overwritten
method nogil {
my $cursor := self.nogil-proxy;
# .. This si where Nqp expertise would be nice
say "string is:", $cursor.Str; # here also it should print "$"
return $cursor;
}
Failed try:
$cursor.^cache_add('Str', sub { return '$'; } );
$cursor.^publish_method_cache;
for $cursor.^attributes { .name.say };
for $cursor.^methods { .name.say };
say $cursor.WHAT.Str;
nqp::setmethcacheauth($cursor, 0);
Currently, most of my tests work but I have problems in declarations without my (with no strict) like my-var = 42; because they are considered as method call.
#Arne-Sommer already made a post and an article. This is closely related. But this questions aims:
How can we customize the return value of a compile-time token and not how to declare it.
Intro: The answer, pointed by #JonathanWorthington:
Brief: Use the mixin meta function. (And NOT the but requiring compose method.)
Demo:
Create a NQPMatch object by retrieving another token: here the token sigil-my called by self.sigil-my.
Use ^mixin with a role
method sigil { return self.sigil-my.^mixin(Nogil::StrGil); }
Context: full reproducible code:
So you can see what type are sigil-my and Nogil::StrGil. But I told you: token (more than method) and role (uninstantiable classes).
role Nogil::StrGil {
method Str() {
return sigilize(callsame);
}
}
sub EXPORT(|) {
# Save: main raku grammar
my $main-grammar = $*LANG.slang_grammar('MAIN');
my $main-actions = $*LANG.slang_actions('MAIN');
role Nogil::NogilGrammar {
method sigil {
return self.sigil-my.^mixin(Nogil::StrGil);
}
}
token sigil-my { | <[$#%&]> | <?> }
# Mix
my $grammar = $main-grammar.^mixin(Nogil::NogilGrammar);
my $actions = $main-actions.^mixin(Nogil::NogilActions);
$*LANG.define_slang('MAIN', $grammar, $actions);
# Return empty hash -> specify that we’re not exporting anything extra
return {};
}
Note: This opens the door to mush more problems (also pointed by jnthn question comments) -> -0fun !

Test file structure in groovy(Spock)

How to test created and expected file tree in groovy(Spock)?
Right now I'm using Set where I specify paths which I expect to get and collecting actual paths in this way:
Set<String> getCreatedFilePaths(String root) {
Set<String> createFilePaths = new HashSet<>()
new File(root).eachFileRecurse {
createFilePaths << it.absolutePath
}
return createFilePaths
}
But the readability of the test isn't so good.
Is it possible in groovy to write expected paths as a tree, and after that compare with actual
For example, expected:
region:
usa:
new_york.json
california.json
europe:
spain.json
italy.json
And actual will be converted to this kind of tree.
Not sure if you can do it with the built-in recursive methods. There certainly are powerful ones, but this is standard recursion code you can use:
def path = new File("/Users/me/Downloads")
def printTree(File file, Integer level) {
println " " * level + "${file.name}:"
file.eachFile {
println " " * (level + 1) + it.name
}
file.eachDir {
printTree(it, level + 1)
}
}
printTree(path, 1)
That prints the format you describe
You can either build your own parser or use Groovy's built-in JSON parser:
package de.scrum_master.stackoverflow
import groovy.json.JsonParserType
import groovy.json.JsonSlurper
import spock.lang.Specification
class FileRecursionTest extends Specification {
def jsonDirectoryTree = """{
com : {
na : {
tests : [
MyBaseIT.groovy
]
},
twg : {
sample : {
model : [
PrimeNumberCalculatorSpec.groovy
]
}
}
},
de : {
scrum_master : {
stackoverflow : [
AllowedPasswordsTest.groovy,
CarTest.groovy,
FileRecursionTest.groovy,
{
foo : [
LoginIT.groovy,
LoginModule.groovy,
LoginPage.groovy,
LoginValidationPage.groovy,
User.groovy
]
},
LuceneTest.groovy
],
testing : [
GebTestHelper.groovy,
RestartBrowserIT.groovy,
SampleGebIT.groovy
]
}
}
}"""
def "Parse directory tree JSON representation"() {
given:
def jsonSlurper = new JsonSlurper(type: JsonParserType.LAX)
def rootDirectory = jsonSlurper.parseText(jsonDirectoryTree)
expect:
rootDirectory.de.scrum_master.stackoverflow.contains("CarTest.groovy")
rootDirectory.com.twg.sample.model.contains("PrimeNumberCalculatorSpec.groovy")
when:
def fileList = objectGraphToFileList("src/test/groovy", rootDirectory)
fileList.each { println it }
then:
fileList.size() == 14
fileList.contains("src/test/groovy/de/scrum_master/stackoverflow/CarTest.groovy")
fileList.contains("src/test/groovy/com/twg/sample/model/PrimeNumberCalculatorSpec.groovy")
}
List<File> objectGraphToFileList(String directoryPath, Object directoryContent) {
List<File> files = []
directoryContent.each {
switch (it) {
case String:
files << directoryPath + "/" + it
break
case Map:
files += objectGraphToFileList(directoryPath, it)
break
case Map.Entry:
files += objectGraphToFileList(directoryPath + "/" + (it as Map.Entry).key, (it as Map.Entry).value)
break
default:
throw new IllegalArgumentException("unexpected directory content value $it")
}
}
files
}
}
Please note:
I used new JsonSlurper(type: JsonParserType.LAX) in order to avoid having to quote each single String in the JSON structure. If your file names contain spaces or other special characters, you will have to use something like "my file name", though.
In rootDirectory.de.scrum_master.stackoverflow.contains("CarTest.groovy") you can see how you can nicely interact with the parsed JSON object graph in .property syntax. You might like it or not, need it or not.
Recursive method objectGraphToFileList converts the parsed object graph to a list of files (if you prefer a set, change it, but File.eachFileRecurse(..) should not yield any duplicates, so the set is not needed.
If you do not like the parentheses etc. in the JSON, you can still build your own parser.
You might want to add another utility method to create a JSON string like the given one from a validated directory structure, so you have less work when writing similar tests.
Modified Bavo Bruylandt answer to collect file tree paths, and sort it to not care about the order of files.
def "check directory structure"() {
expect:
String created = getCreatedFilePaths(new File("/tmp/region"))
String expected = new File("expected.txt").text
created == expected
}
private String getCreatedFilePaths(File root) {
List paths = new ArrayList()
printTree(root, 0, paths)
return paths.join("\n")
}
private void printTree(File file, Integer level, List paths) {
paths << ("\t" * level + "${file.name}:")
file.listFiles().sort{it.name}.each {
if (it.isFile()) {
paths << ("\t" * (level + 1) + it.name)
}
if (it.isDirectory()) {
collectFileTree(it, level + 1, paths)
}
}
}
And expected files put in the expected.txt file with indent(\t) in this way:
region:
usa:
new_york.json
california.json
europe:
spain.json
italy.json

Perl6: How to find all installed modules whose filename matches a pattern?

Is it possible in Perl6 to find all installed modules whose file-name matches a pattern?
In Perl5 I would write it like this:
use File::Spec::Functions qw( catfile );
my %installed;
for my $dir ( #INC ) {
my $glob_pattern = catfile $dir, 'App', 'DBBrowser', 'DB', '*.pm';
map { $installed{$_}++ } glob $glob_pattern;
}
There is currently no way to get the original file name of an installed module. However it is possible to get the module names
sub list-installed {
my #curs = $*REPO.repo-chain.grep(*.?prefix.?e);
my #repo-dirs = #curs>>.prefix;
my #dist-dirs = |#repo-dirs.map(*.child('dist')).grep(*.e);
my #dist-files = |#dist-dirs.map(*.IO.dir.grep(*.IO.f).Slip);
my $dists := gather for #dist-files -> $file {
if try { Distribution.new( |%(from-json($file.IO.slurp)) ) } -> $dist {
my $cur = #curs.first: {.prefix eq $file.parent.parent}
take $_ for $dist.hash<provides>.keys;
}
}
}
.say for list-installed();
see: Zef::Client.list-installed()