I'm trying to start a curl download and then pipe it's stdout to the stdout of my program. Here's what I got (still using old_io):
let process = Command::new("curl")
.arg("-o ").arg(&self.filename).arg(&self.url)
.spawn().ok().expect("Download failed.");
let output = process.wait_with_output().ok().expect("Download failed.");
That works perfectly fine, but the curl download takes quite a long time. So it would be neat to display curl's stdout (=> the progress of the download) while it's still downloading and not just at the end. I tried lot's of things, but can't get this thing working. Any ideas?
Instead of using wait_with_output, just read from the pipes that are part of process. When you can't read anymore, then you can wait for the process. Here's an example with no error handling:
#![feature(old_io)]
use std::old_io::process::Command;
use std::old_io::stdio;
fn main() {
let mut process = Command::new("bash")
.arg("-c")
.arg("for i in 1 2 3 4 5; do echo start-$i; sleep 1; echo stop-$i; done")
.spawn()
.ok()
.expect("Command failed");
let buf = &mut [0; 256];
while let Ok(bytes) = process.stdout.as_mut().unwrap().read(buf) {
stdio::stdout_raw().write_all(&buf[..bytes]).unwrap();
}
let _ = process.wait_with_output().ok().expect("Download failed.");
}
Related
The following code is very simple. Open a file as a write, create a BufWriter using the file, and write a line of string.
The program reports no errors and returns an Ok(10) value, but the file just has no content and is empty.
#[tokio::test]
async fn save_file_async() {
let path = "./hello.txt";
let inner = tokio::fs::OpenOptions::new()
.create(true)
.write(true)
//.truncate(true)
.open(path)
.await
.unwrap();
let mut writer = tokio::io::BufWriter::new(inner);
println!(
"{} bytes wrote",
writer.write("1234567890".as_bytes()).await.unwrap()
);
}
Need an explicit flush:
writer.flush().await.unwrap();
Is it possible to have a macro that generates standalone tests? I have two text files, one with an input and another with an output. Each new line in the text file represents a new test.
Currently, this is how I run my tests:
#[test]
fn it_works() {
let input = read_file("input.txt").expect("failed to read input");
let input = input.split("\n").collect::<Vec<_>>();
let output = read_file("output.txt").expect("failed to read output");
let output = output.split("\n").collect::<Vec<_>>();
input.iter().zip(output).for_each(|(a, b)| {
println!("a: {}, b: {}", a, b);
assert_eq!(b, get_result(a));
})
But, as you can see, if one test fail, all of them fail, since there's a loop inside a single test. And I need each iteration to be a single and isolated test, without having to repeat myself.
So I was wondering if it's possible to achieve that by using macros?
The macro ideally would output something like:
#[test]
fn it_works_1() {
let input = read_file("input.txt").expect("failed to read input");
let input = input.split("\n").collect::<Vec<_>>();
let output = read_file("output.txt").expect("failed to read output");
let output = output.split("\n").collect::<Vec<_>>();
assert_eq!(output[0], get_result(input[0])); // first test
}
#[test]
fn it_works_2() {
let input = read_file("input.txt").expect("failed to read input");
let input = input.split("\n").collect::<Vec<_>>();
let output = read_file("output.txt").expect("failed to read output");
let output = output.split("\n").collect::<Vec<_>>();
assert_eq!(output[1], get_result(input[1])); // second test
}
// ... the N remaining tests: it_works_n()
You can't do this with a declarative macro because a declarative macro cannot generate an identifier to name the test functions. However you can use a crate such as test-case, which can run the same test with different inputs:
use test_case::test_case;
#[test_case(0)]
#[test_case(1)]
#[test_case(2)]
#[test]
fn it_works(index: usize) {
let input = read_file("input.txt").expect("failed to read input");
let input = input.split("\n").collect::<Vec<_>>();
let output = read_file("output.txt").expect("failed to read output");
let output = output.split("\n").collect::<Vec<_>>();
assert_eq!(output[index], get_result(input[index])); // first test
}
If you have a lot of different inputs to test, you could use a declarative macro to generate the code above, which would add all of the #[test_case] annotations.
After Peter Hall answer, I was able to achieve what I wanted. I added the seq_macro crate to generate the repeated #[test_case]'s. Maybe there's a way to loop through all test cases instead of manually defining the amount of tests (like I did), but this is good for now:
macro_rules! test {
( $from:expr, $to:expr ) => {
#[cfg(test)]
mod tests {
use crate::{get_result, read_file};
use seq_macro::seq;
use test_case::test_case;
seq!(N in $from..$to {
#(#[test_case(N)])*
fn it_works(index: usize) {
let input = read_file("input.txt").expect("failed to read input");
let input = input.split("\n").collect::<Vec<_>>();
let output = read_file("output.txt").expect("failed to read output");
let output = output.split("\n").collect::<Vec<_>>();
let res = get_result(input[index]);
assert_eq!(
output[index], res,
"Test '{}': Want '{}' got '{}'",
input[index], output[index], res
);
}
});
}
};
}
test!(0, 82);
I'm attempting to run a series of shell commands in parallel in Perl6, using Perl5's Parallel::ForkManager
This is an almost exact translation of working Perl5 code.
CONTROL {
when CX::Warn {
note $_;
exit 1;
}
}
use fatal;
role KeyRequired {
method AT-KEY (\key) {
die "Key {key} not found" unless self.EXISTS-KEY(key);
nextsame;
}
}
use Parallel::ForkManager:from<Perl5>;
sub run_parallel (#cmd) {
my $manager = Parallel::ForkManager(8).new();
for (#cmd) -> $command {
$manager.start and $manager.next;
my $proc = shell $command, :out, :err;
if $proc.exitcode != 0 {
put "$command failed";
put $proc.out.slurp;
put $proc.err.slurp;
die;
}
$manager.finish;
}
$manager.wait_all_children;#necessary after all lists
}
my #cmd;
my Str $dir = 'A/1';
for dir($dir, test => /\.vcf\.gz$/) -> $vcf {
#cmd.append: "aws s3 cp $vcf s3://s3dir/$dir/"
}
put #cmd.elems;
run_parallel(#cmd);
Basically, I'm trying to parallelize tedious shell commands.
However, this mysterious error comes up:
Cannot invoke this object (REPR: P6opaque; Parallel::ForkManager) in
sub run_parallel at 2.aws_cp.p6 line 18 in block at
2.aws_cp.p6 line 39
Why is Perl6 saying this? what is wrong? how can I get these commands to run?
Perhaps there is a more native/idiomatic way to run shell commands in parallel in Perl6?
You probably want to look at using Proc::Async which runs external commands asynchronously in threads without forking separate instances of the code to do it.
Perl5's Parallel::ForkManager probably won't work in Perl6 because of how Inline::Perl5 is implemented.
Inline::Perl5 embeds the Perl5 compiler/runtime inside of Perl6.
Parallel::ForkManager expects that Perl5 was run by itself.
If you ever did get it to do something other than generate an error it would probably screw up the Perl6 runtime. The main problem is the use of fork. For more information about why fork is a a problem see the article Bart Wiegmans (brrt) wrote about it: “A future for fork(2)”
Perl6 already has a similar feature that is easier to use.
sub run_parallel (#cmd) {
my #children = do for (#cmd) -> $command {
start {
my $proc = shell $command, :out, :err;
if $proc.exitcode != 0 {
put "$command failed";
put $proc.out.slurp;
put $proc.err.slurp;
die;
}
}
}
await #children;
}
start is a prefix that tells the runtime to start running the following code sometime in the near future. It returns a Promise.
await takes a list of Promises and returns a list of their results.
start basically calls Promise.start which is similar to:
sub start ( &code ) {
my $promise = Promise.new;
my $vow = $promise.vow;
$*SCHEDULER.cue(
{ $vow.keep(code(|c)) },
:catch(-> $ex { $vow.break($ex); }) );
$promise
}
So it will use the globally available thread pool in $*SCHEDULER. If you want to use a separate one you could.
sub run_parallel (#cmd) {
my $*SCHEDULER = ThreadPoolScheduler.new(max_threads => 8);
my #children = do for (#cmd) -> $command {
start {
my $proc = shell $command, :out, :err;
if $proc.exitcode != 0 {
put "$command failed";
put $proc.out.slurp;
put $proc.err.slurp;
die;
}
}
}
await #children;
}
It would make more sense to use Proc::Async for this though.
In my Perl 6 script, I want to do a (preferably non-blocking) check of standard input to see if data is available. If this is the case, then I want to process it, otherwise I want to do other stuff.
Example (consumer.p6):
#!/usr/bin/perl6
use v6.b;
use fatal;
sub MAIN() returns UInt:D {
while !$*IN.eof {
if some_fancy_check_for_STDIN() { #TODO: this needs to be done.
for $*IN.lines -> $line {
say "Process '$line'";
}
}
say "Do something Else.";
}
say "I'm done.";
return 0;
}
As a STDIN-Generator I wrote another Perl6 script (producer.p6):
#!/usr/bin/perl6
use v6.b;
use fatal;
sub MAIN() returns UInt:D {
$*OUT.say("aaaa aaa");
sleep-until now+2;
$*OUT.say("nbbasdf");
sleep-until now+2;
$*OUT.say("xxxxx");
sleep-until now+2;
return 0;
}
If consumer.p6 works as expected, it should produce the following output, if called via ./producer.p6 | ./consumer.p6:
Process 'aaaa aaa'
Do something Else.
Process 'nbbasdf'
Do something Else.
Process 'xxxxx'
Do something Else.
I'm done.
But actually, it produces the following output (if the if condition is commented out):
Process 'aaaa aaa'
Process 'nbbasdf'
Process 'xxxxx'
Do something Else.
I'm done.
You are using an old version of Perl 6, as v6.b is from before the official release of the language.
So some of what I have below may need a newer version to work.
Also why are you using sleep-until now+2 instead of sleep 2?
One way to do this is to turn the .lines into a Channel, then you can use .poll.
#!/usr/bin/env perl6
use v6.c;
sub MAIN () {
# convert it into a Channel so we can poll it
my $lines = $*IN.Supply.lines.Channel;
my $running = True;
$lines.closed.then: {$running = False}
while $running {
with $lines.poll() -> $line {
say "Process '$line'";
}
say "Do something Else.";
sleep ½;
}
say "I'm done.";
}
Note that the code above blocks at the my $lines = … line currently; so it doesn't start doing something until the first line comes in. To get around that you could do the following
my $lines = supply {
# unblock the $*IN.Supply.lines call
whenever start $*IN.Supply {
whenever .lines { .emit }
}
}.Channel;
I am attempting to set up a small test environment (homelab) using CentOS 6.6, Rancid 3.1, Looking Glass, and some Cisco Switches/Routers, with httpd acting as the handler. I have picked up a little perl by means of this endeavor, but python (more 2 than 3) is my background. Right now, everything on the rancid side of things works without issue: bin/clogin successfully logs into all of the equipment in the router.db file, and logging of the configs is working as expected. All switches/routers to be accessed are available and online, verified by ssh connection to devices as well as using bin/clogin.
Right now, I have placed the lg.cgi and lgform.cgi files into var/www/cgi-bin/ which allows the forms to be run as cgi scripts. I had to modify the files to split on ';' instead of ':' due to the change in the .db file in Rancid 3.1:#record = split('\:', $_); was replaced with: #record = split('\;', $_); etc. Once that change was made, I was able to load the lgform.cgi with the proper router.db parsing. At this point, it seemed like everything should be good to go. When I attempt to ping from one of those devices out to 8.8.8.8, the file correctly redirects to lg.cgi, and the page loads, but with
main is unavailable. Try again later.
as the error, where 'main' is the router hostname. Using this output, I was able to find the function responsible for this output. Here it is before I added anything:
sub DoRsh
{
my ($router, $mfg, $cmd, $arg) = #_;
my($ctime) = time();
my($val);
my($lckobj) = LockFile::Simple->make(-delay => $lock_int,
-max => $max_lock_wait, -hold => $max_lock_hold);
if ($pingcmd =~ /\d$/) {
`$pingcmd $router`;
} else {
`$pingcmd $router 56 1`;
}
if ($?) {
print "$router is unreachable. Try again later.\n";
return(-1);
}
if ($LG_SINGLE) {
if (! $lckobj->lock("$cache_dir/$router")) {
print "$router is busy. Try again later.\n";
return(-1);
}
}
$val = &DoCmd($router, $mfg, $cmd, $arg);
if ($LG_SINGLE) {
$lckobj->unlock("$cache_dir/$router");
}
return($val);
}
In order to dig in a little deeper, I peppered that function with several print statements. Here is the modified function, followed by the output from the loaded lg.cgi page:
sub DoRsh
{
my ($router, $mfg, $cmd, $arg) = #_;
my($ctime) = time();
my($val);
my($lckobj) = LockFile::Simple->make(-delay => $lock_int,
-max => $max_lock_wait, -hold => $max_lock_hold);
if ($pingcmd =~ /\d$/) {
`$pingcmd $router`;
} else {
`$pingcmd $router 56 1`;
}
print "About to test the ($?) branch.\n";
print "Also who is the remote_user?:' $remote_user'\n";
print "What about the ENV{REMOTE_USER} '$ENV{REMOTE_USER}'\n";
print "Here is the ENV{HOME}: '$ENV{HOME}'\n";
if ($?) {
print "$lckobj is the lock object.\n";
print "#_ something else to look at.\n";
print "$? whatever this is suppose to be....\n";
print "Some variables:\n";
print "$mfg is the mfg.\n";
print "$cmd was the command passed in with $arg as the argument.\n";
print "$pingcmd $router\n";
print "$cloginrc - Is the cloginrc pointing correctly?\n";
print "$LG_SINGLE the next value to be tested.\n";
print "$router is unreachable. Try again later.\n";
return(-1);
}
if ($LG_SINGLE) {
if (! $lckobj->lock("$cache_dir/$router")) {
print "$router is busy. Try again later.\n";
return(-1);
}
}
$val = &DoCmd($router, $mfg, $cmd, $arg);
if ($LG_SINGLE) {
$lckobj->unlock("$cache_dir/$router");
}
return($val);
}
OUTPUT:
About to test the (512) branch.
Also who is the remote_user?:' '
What about the ENV{REMOTE_USER} ''
Here is the ENV{HOME}: '.'
LockFile::Simple=HASH(0x1a13650) is the lock object.
main cisco ping 8.8.8.8 something else to look at.
512 whatever this is suppose to be....
Some variables:
cisco is the mfg.
ping was the command passed in with 8.8.8.8 as the argument.
/bin/ping -c 1 main
./.cloginrc - Is the cloginrc pointing correctly?
1 the next value to be tested.
main is unreachable. Try again later.
I can provide the code for when DoRsh is called, if necessary, but it looks mostly like this:&DoRsh($router, $mfg, $cmd, $arg);.
From what I can tell the '$?' special variable (or at least according to
this reference it is a special var) is returning the 512 value, which is causing that fork to test true. The problem is I don't know what that 512 means, nor where it is coming from. Using the ref site's description ("The status returned by the last pipe close, backtick (``) command, or system operator.") and the formation of the conditional tree above, I can see that it is some error of some kind, but I don't know how else to proceed with this inspection. I'm wondering if maybe it is in response to some permission issue, since the remote_user variable is null, when I didn't expect it to be. Any guidance anyone may be able to provide would be helpful. Furthermore, if there is any information that I may have skipped over, that I didn't think to include, or that may prove helpful, please ask, and I will provide to the best of my ability
May be you put in something like
my $pingret=$pingcmd ...;
print 'Ping result was:'.$pingret;
And check the returned strings?