How can I recursively read a directory in Deno? - file-io

I'm trying to recursively read a file in Deno using Deno.readDir, but the example they provide only does the folder given:
for await (const entry of Deno.readDir(Deno.cwd())) {
console.log(entry.name);
}
How can I make this recursive?

Deno's standard library includes a function called walk for this purpose. It's available in std/fs/walk.ts. Here's an example:
/Users/deno/so-74953935/main.ts:
import { walk } from "https://deno.land/std#0.170.0/fs/walk.ts";
for await (const walkEntry of walk(Deno.cwd())) {
const type = walkEntry.isSymlink
? "symlink"
: walkEntry.isFile
? "file"
: "directory";
console.log(type, walkEntry.path);
}
Running in the terminal:
% pwd
/Users/deno/so-74953935
% ls -AF
.vscode/ deno.jsonc deno.lock main.ts
% ls -AF .vscode
settings.json
% deno --version
deno 1.29.1 (release, x86_64-apple-darwin)
v8 10.9.194.5
typescript 4.9.4
% deno run --allow-read main.ts
directory /Users/deno/so-74953935
file /Users/deno/so-74953935/main.ts
file /Users/deno/so-74953935/deno.jsonc
file /Users/deno/so-74953935/deno.lock
directory /Users/deno/so-74953935/.vscode
file /Users/deno/so-74953935/.vscode/settings.json

Since that function returns an async generator, you can make your own generator function that wraps around Deno.readDir:
(Do note that the example provided will join the path and name, giving you strings such as /directory/name.txt)
import { join } from "https://deno.land/std/path/mod.ts";
export async function* recursiveReaddir(
path: string
): AsyncGenerator<string, void> {
for await (const dirEntry of Deno.readDir(path)) {
if (dirEntry.isDirectory) {
yield* recursiveReaddir(join(path, dirEntry.name));
} else if (dirEntry.isFile) {
yield join(path, dirEntry.name);
}
}
}
for await (const entry of recursiveReaddir(Deno.cwd())) {
console.log(entry)
}
OR, you can use recursive_readdir, which is a 3rd party library in Deno made for this purpose.

Related

How should I bundle a library of text files with my module?

I have the following structure in the resources directory in a module I'm building:
resources
|-- examples
|-- Arrays
| |-- file
|-- Lists
|-- file1
|-- file2
I have the following code to collect and process these files:
use v6.d;
unit module Doc::Examples::Resources;
class Resource {
has Str $.name;
has Resource #.resources;
has Resource %.resource-index;
method resource-names() {
#.resources>>.name.sort
}
method list-resources() {
self.resource-names>>.say;
}
method is-resource(Str:D $lesson) {
$lesson ~~ any self.resource-names;
}
method get-resource(Str:D $lesson) {
if !self.is-resource($lesson) {
say "Sorry, that lesson does not exist.";
return;
}
return %.resource-index{$lesson};
}
}
class Lesson is Resource {
use Doc::Parser;
use Doc::Subroutines;
has IO $.file;
method new(IO:D :$file) {
my $name = $file.basename;
self.bless(:$name, :$file)
}
method parse() {
my #parsed = parse-file $.file.path;
die "Failed parse examples from $.file" if #parsed.^name eq 'Any';
for #parsed -> $section {
my $heading = $section<meta>[0] || '';
my $intro = $section<meta>[1] || '';
say $heading.uc ~ "\n" if $heading && !$intro;
say $heading.uc if $heading && $intro;
say $intro ~ "\n" if $intro;
for $section<code>.Array {
die "Failed parse examples from $.file, check it's syntax." if $_.^name eq 'Any';
das |$_>>.trim;
}
}
}
}
class Topic is Resource {
method new(IO:D :$dir) {
my $files = dir $?DISTRIBUTION.content("$dir");
my #lessons;
my $name = $dir.basename;
my %lesson-index;
for $files.Array -> $file {
my $lesson = Lesson.new(:$file);
push #lessons, $lesson;
%lesson-index{$lesson.name} = $lesson;
}
self.bless(:$name, resources => #lessons, resource-index => %lesson-index);
}
}
class LocalResources is Resource is export {
method new() {
my $dirs = dir $?DISTRIBUTION.content('resources/examples');
my #resources;
my %resource-index;
for $dirs.Array -> $dir {
my $t = Topic.new(:$dir);
push #resources, $t;
%resource-index{$t.name} = $t;
}
self.bless(:#resources, :%resource-index)
}
method list-lessons(Str:D $topic) {
self.get-resource($topic).list-lessons;
}
method parse-lesson(Str:D $topic, Str:D $lesson) {
self.get-resource($topic).get-resource($lesson).parse;
}
}
It works. However, I'm told that this is not reliable and there there is no guarantee that lines like my $files = dir $?DISTRIBUTION.content("$dir"); will work after the module is installed or will continue to work into the future.
So what are better options for bundling a library of text files with my module that can be accessed and found by the module?
Files under the resources directory will always be available as keys to the %?RESOURCES compile-time variable if you declare them in the META6.json file this way:
"resources": [
"examples/Array/file",
]
and so on.
I've settled on a solution. As pointed out by jjmerelo, the META6.json file contains a list of resources and, if you use the comma IDE, the list of resources is automatically generated for you.
From within the module's code, the list of resources can be accessed via the $?DISTRIBUTION variable like so:
my #resources = $?DISTRIBUTION.meta<resources>
From here, I can build up my list of resources.
One note on something I discovered: the $?DISTRIBUTION variable is not accessible from a test script. It has to be placed inside a module in the lib directory of the distribution and exported.

How can I override the jetbrains jdk dependency required by idea-ultimate or idea-community in config.nix?

I’ve been setting up my local nix config as per nixpkgs manual's declarative package management.
I’d like to include idea-ultimate as one of myPackages, but at this time the dependency idea has on the jetbrains jdk is broken, pointing to a non-existing package for macOS.
It’s trying to download jbrsdk-11_0_2-osx-x64-b485.1.tar.gz instead of jbrsdk-11_0_4-osx-x64-b485.1.tar.gz.
I was assuming I could fix that by overriding jetbrainsjdk as follows, but I’m getting: error: attribute 'jetbrainsjdk' missing, at /Users/ldeck/.config/nixpkgs/config.nix:4:20 when I do anything like nix-env -qa ‘jetbrains.*’.
What is the right way to override idea-ultimate so that it uses the fixed jdk?
Here’s my ~./config/nixpkgs/config.nix.
{
allowUnfree = true;
packageOverrides = pkgs: rec {
jetbrainsjdk = pkgs.jetbrainsjdk.override {
version = "520.11";
src = pkgs.fetchurl {
url = "https://bintray.com/jetbrains/intellij-jdk/download_file?file_path=jbrsdk-11_0_4-osx-x64-b${jetbrainsjdk.version}.tar.gz";
sha256 = "0d1qwbssc8ih62rlfxxxcn8i65cjgycdfy1dc1b902j46dqjkq9z";
};
};
myProfile = pkgs.writeText "my-profile" ''
export PATH=$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/sbin:/bin:/usr/sbin:/usr/bin
export MANPATH=$HOME/.nix-profile/share/man:/nix/var/nix/profiles/default/share/man:/usr/share/man
'';
myPackages = with pkgs; buildEnv {
name = "my-packages";
paths = [
(runCommand "profile" {} ''
mkdir -p $out/etc/profile.d
cp ${myProfile} $out/etc/profile.d/my-profile.sh
'')
aspell
bc
coreutils
direnv
emacs
emscripten
ffmpeg
gdb
git
hello
jq
nixops
nox
scala
silver-searcher
];
pathsToLink = [ "/share/man" "/share/doc" "/bin" "/etc" "/Applications" ];
extraOutputsToInstall = [ "man" "doc" ];
};
};
}
UPDATE 1
Thanks to #ChrisStryczynski who suggested I needed with pkgs, I’ve gotten a little further.
But now the problem is when attempting to install idea-ultimate with the custom jdk, it’s still requiring the broken, non-existing, jbrsdk-11_02-osx-x64-b485.1.tar.gz.drv from somewhere.
Updated config and logs below.
{
allowUnfree = true;
packageOverrides = pkgs: **with pkgs;** rec {
myJetbrainsJdk = **pkgs.jetbrains.jdk.overrideAttrs** (oldAttrs: rec {
version = "520.11";
src = pkgs.fetchurl {
url = "https://bintray.com/jetbrains/intellij-jdk/download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz";
sha256 = "0d1qwbssc8ih62rlfxxxcn8i65cjgycdfy1dc1b902j46dqjkq9z";
};
});
myIdeaUltimate = pkgs.jetbrains.idea-ultimate.override {
jdk = myJetbrainsJdk;
};
...
myPackages = with pkgs; buildEnv {
...
myIdeaUltimate
];
...
};
};
}
Logs
nix-channel --update; nix-env -iA nixpkgs.myPackages
unpacking channels...
replacing old 'my-packages'
installing 'my-packages'
these derivations will be built:
/nix/store/9kfi3k9q6hi7z3lwann318hndbah535v-idea-ultimate.desktop.drv
/nix/store/ica1m5yq3f3y05xnw7ln1lnfvp0yjvyf-download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz.drv
/nix/store/bf2hwhrvfl8g77gdiw053rayh06x0120-jetbrainsjdk-520.11.drv
/nix/store/fazsa1a4l70s391rjk9yyi2hvrg0zbmp-download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz.drv
/nix/store/fwwk976sd278zb68zy9wm5pkxss0rnhg-jetbrainsjdk-485.1.drv
/nix/store/s3m2bpcyrnx9dcq4drh95882n0mk1d6m-ideaIU-2019.2.4-no-jbr.tar.gz.drv
/nix/store/9kiajpmmsp3i6ysj4vdqq8dzi84mnr73-idea-ultimate-2019.2.4.drv
/nix/store/jh1ixm54qinv8pk6kypvv6n6cfr4sws8-my-packages.drv
these paths will be fetched (0.02 MiB download, 0.12 MiB unpacked):
/nix/store/hp90sbwznq1msv327f0lb27imvcvi80h-libnotify-0.7.8
building '/nix/store/9kfi3k9q6hi7z3lwann318hndbah535v-idea-ultimate.desktop.drv'...
copying path '/nix/store/hp90sbwznq1msv327f0lb27imvcvi80h-libnotify-0.7.8' from 'https://cache.nixos.org'...
building '/nix/store/fazsa1a4l70s391rjk9yyi2hvrg0zbmp-download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz.drv'...
trying https://bintray.com/jetbrains/intellij-jdk/download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404 Not Found
error: cannot download download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz from any mirror
builder for '/nix/store/fazsa1a4l70s391rjk9yyi2hvrg0zbmp-download_file?file_path=jbrsdk-11_0_2-osx-x64-b485.1.tar.gz.drv' failed with exit code 1
building '/nix/store/ica1m5yq3f3y05xnw7ln1lnfvp0yjvyf-download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz.drv'...
cannot build derivation '/nix/store/fwwk976sd278zb68zy9wm5pkxss0rnhg-jetbrainsjdk-485.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9kiajpmmsp3i6ysj4vdqq8dzi84mnr73-idea-ultimate-2019.2.4.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/jh1ixm54qinv8pk6kypvv6n6cfr4sws8-my-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/jh1ixm54qinv8pk6kypvv6n6cfr4sws8-my-packages.drv' failed
Thanks to How do you discover the override attributes for a derivation, a solution has been found using an overlay.
# ~/config/nixpkgs/overlays/02-jetbrains.nix
self: super:
{
jetbrains = super.jetbrains // {
jdk = super.jetbrains.jdk.overrideAttrs (oldAttrs: rec {
version = "520.11";
src = super.fetchurl {
url = "https://bintray.com/jetbrains/intellij-jbr/download_file?file_path=jbrsdk-11_0_4-osx-x64-b520.11.tar.gz";
sha256 = "3fe1297133440a9056602d78d7987f9215139165bd7747b3303022a6f5e23834";
};
passthru = oldAttrs.passthru // {
home = "${self.jetbrains.jdk}/Contents/Home";
};
});
idea-ultimate = super.jetbrains.idea-ultimate.overrideAttrs (_: {
name = "idea-ultimate.2019.2.4";
src = super.fetchurl {
url = "https://download.jetbrains.com/idea/ideaIU-2019.2.4-no-jbr.tar.gz";
sha256 = "09mz4dx3zbnqw0vh4iqr8sn2s8mvgr7zvn4k7kqivsiv8f79g90a";
};
});
};
}
Install: nix-env -iA 'nixpkgs.jetbrains.idea-ultimate'.
Execute: idea-ultimate.
The crucial part of the puzzle was overriding the passthru.home variable to point to the overlayed JDK rather than the one required by super. Otherwise, you’ll be downloading the old JDK for runtime purposes.
passthru = oldAttrs.passthru // {
home = "${self.jetbrains.jdk}/Contents/Home”;
};
Without /Contents/Home appended, idea won’t startup since self.jetbrains.jdk isn’t a valid home.
Instead of:
nix-env -iA nixpkgs.myPackages
Just do:
nix-env -iA nixpkgs.myIdeaUltimate
The problem being:
myPackages = with pkgs; buildEnv {
...
myIdeaUltimate
];
...
};
Here you are still referencing the old pkgs.myIdeaUltimate.
Nixos seems to do some processing that replaces the pkgs with the appropriate from packageOverrides.

Test file structure in groovy(Spock)

How to test created and expected file tree in groovy(Spock)?
Right now I'm using Set where I specify paths which I expect to get and collecting actual paths in this way:
Set<String> getCreatedFilePaths(String root) {
Set<String> createFilePaths = new HashSet<>()
new File(root).eachFileRecurse {
createFilePaths << it.absolutePath
}
return createFilePaths
}
But the readability of the test isn't so good.
Is it possible in groovy to write expected paths as a tree, and after that compare with actual
For example, expected:
region:
usa:
new_york.json
california.json
europe:
spain.json
italy.json
And actual will be converted to this kind of tree.
Not sure if you can do it with the built-in recursive methods. There certainly are powerful ones, but this is standard recursion code you can use:
def path = new File("/Users/me/Downloads")
def printTree(File file, Integer level) {
println " " * level + "${file.name}:"
file.eachFile {
println " " * (level + 1) + it.name
}
file.eachDir {
printTree(it, level + 1)
}
}
printTree(path, 1)
That prints the format you describe
You can either build your own parser or use Groovy's built-in JSON parser:
package de.scrum_master.stackoverflow
import groovy.json.JsonParserType
import groovy.json.JsonSlurper
import spock.lang.Specification
class FileRecursionTest extends Specification {
def jsonDirectoryTree = """{
com : {
na : {
tests : [
MyBaseIT.groovy
]
},
twg : {
sample : {
model : [
PrimeNumberCalculatorSpec.groovy
]
}
}
},
de : {
scrum_master : {
stackoverflow : [
AllowedPasswordsTest.groovy,
CarTest.groovy,
FileRecursionTest.groovy,
{
foo : [
LoginIT.groovy,
LoginModule.groovy,
LoginPage.groovy,
LoginValidationPage.groovy,
User.groovy
]
},
LuceneTest.groovy
],
testing : [
GebTestHelper.groovy,
RestartBrowserIT.groovy,
SampleGebIT.groovy
]
}
}
}"""
def "Parse directory tree JSON representation"() {
given:
def jsonSlurper = new JsonSlurper(type: JsonParserType.LAX)
def rootDirectory = jsonSlurper.parseText(jsonDirectoryTree)
expect:
rootDirectory.de.scrum_master.stackoverflow.contains("CarTest.groovy")
rootDirectory.com.twg.sample.model.contains("PrimeNumberCalculatorSpec.groovy")
when:
def fileList = objectGraphToFileList("src/test/groovy", rootDirectory)
fileList.each { println it }
then:
fileList.size() == 14
fileList.contains("src/test/groovy/de/scrum_master/stackoverflow/CarTest.groovy")
fileList.contains("src/test/groovy/com/twg/sample/model/PrimeNumberCalculatorSpec.groovy")
}
List<File> objectGraphToFileList(String directoryPath, Object directoryContent) {
List<File> files = []
directoryContent.each {
switch (it) {
case String:
files << directoryPath + "/" + it
break
case Map:
files += objectGraphToFileList(directoryPath, it)
break
case Map.Entry:
files += objectGraphToFileList(directoryPath + "/" + (it as Map.Entry).key, (it as Map.Entry).value)
break
default:
throw new IllegalArgumentException("unexpected directory content value $it")
}
}
files
}
}
Please note:
I used new JsonSlurper(type: JsonParserType.LAX) in order to avoid having to quote each single String in the JSON structure. If your file names contain spaces or other special characters, you will have to use something like "my file name", though.
In rootDirectory.de.scrum_master.stackoverflow.contains("CarTest.groovy") you can see how you can nicely interact with the parsed JSON object graph in .property syntax. You might like it or not, need it or not.
Recursive method objectGraphToFileList converts the parsed object graph to a list of files (if you prefer a set, change it, but File.eachFileRecurse(..) should not yield any duplicates, so the set is not needed.
If you do not like the parentheses etc. in the JSON, you can still build your own parser.
You might want to add another utility method to create a JSON string like the given one from a validated directory structure, so you have less work when writing similar tests.
Modified Bavo Bruylandt answer to collect file tree paths, and sort it to not care about the order of files.
def "check directory structure"() {
expect:
String created = getCreatedFilePaths(new File("/tmp/region"))
String expected = new File("expected.txt").text
created == expected
}
private String getCreatedFilePaths(File root) {
List paths = new ArrayList()
printTree(root, 0, paths)
return paths.join("\n")
}
private void printTree(File file, Integer level, List paths) {
paths << ("\t" * level + "${file.name}:")
file.listFiles().sort{it.name}.each {
if (it.isFile()) {
paths << ("\t" * (level + 1) + it.name)
}
if (it.isDirectory()) {
collectFileTree(it, level + 1, paths)
}
}
}
And expected files put in the expected.txt file with indent(\t) in this way:
region:
usa:
new_york.json
california.json
europe:
spain.json
italy.json

Perl6: How to find all installed modules whose filename matches a pattern?

Is it possible in Perl6 to find all installed modules whose file-name matches a pattern?
In Perl5 I would write it like this:
use File::Spec::Functions qw( catfile );
my %installed;
for my $dir ( #INC ) {
my $glob_pattern = catfile $dir, 'App', 'DBBrowser', 'DB', '*.pm';
map { $installed{$_}++ } glob $glob_pattern;
}
There is currently no way to get the original file name of an installed module. However it is possible to get the module names
sub list-installed {
my #curs = $*REPO.repo-chain.grep(*.?prefix.?e);
my #repo-dirs = #curs>>.prefix;
my #dist-dirs = |#repo-dirs.map(*.child('dist')).grep(*.e);
my #dist-files = |#dist-dirs.map(*.IO.dir.grep(*.IO.f).Slip);
my $dists := gather for #dist-files -> $file {
if try { Distribution.new( |%(from-json($file.IO.slurp)) ) } -> $dist {
my $cur = #curs.first: {.prefix eq $file.parent.parent}
take $_ for $dist.hash<provides>.keys;
}
}
}
.say for list-installed();
see: Zef::Client.list-installed()

How to pass param {overwrite: true} in Brocfile.js

info:
ember-cli: 0.2.5,
ember-cli-less: 1.3.3,
I have the next error:
Merge error: file assets/mobile.css.map exists in mobile/tmp/caching-writer-dest-dir_ZcaW3C.tmp and /mobile/tmp/caching-writer-dest-dir_E4iROO.tmp
Pass option { overwrite: true } to mergeTrees in order to have the latter file win.
I tried what say in this post and this other. But I still without results.
Thanks for your time
Since none of the methods mentioned in the previous links worked. I choose to modify the ember-cli-less package:
node_modules/ember-cli-less/node_modules/broccoli-merge-trees/index.js
I changed the line:
function TreeMerger (inputTrees, options) {
if (!(this instanceof TreeMerger)) return new TreeMerger(inputTrees, options)
if (!Array.isArray(inputTrees)) {
throw new Error('Expected array, got ' + inputTrees)
}
this.inputTrees = inputTrees
this.options = { overwrite: true } // <- This line
}
The mergeTrees module has an option for overwrite.
It's documented here: https://github.com/broccolijs/broccoli-merge-trees#options
Here's an example:
return new MergeTrees([app.toTree(), fonts], { overwrite: true });
If you are using mac, can you try find . -name '.DS_Store' -type f -delete?
I've a similar issue, and I was able to solve by deleting duplicate files.
Error message
Build failed.
Merge error: file .DS_Store exists in /Users/ember-proj/tmp/broccoli_merge_trees-input_base_path-qaWJUMIJ.tmp/0 and /Users/ember-proj/ai/tmp/broccoli_merge_trees-input_base_path-qaWJUMIJ.tmp/1
Pass option { overwrite: true } to mergeTrees in order to have the latter file win.
update ember-cli-less to latest version (now 1.5.3), then pass config like this.
'ember-cli-less': {
mergeTrees: {
overwrite: true
}
}
things would work