I'm trying to run a series of shell commands with Perl6 to the variable $cmd, which look like
databricks jobs run-now --job-id 35 --notebook-params '{"directory": "s3://bucket", "output": "s3://bucket/extension", "sampleID_to_canonical_id_map": "s3://somefile.csv"}'
Splitting the command by everything after notebook-params
my $cmd0 = 'databricks jobs run-now --job-id 35 --notebook-params ';
my $args = "'{\"directory\": \"$in-dir\", \"output\": \"$out-dir\",
\"sampleID_to_canonical_id_map\": \"$map\"}'"; my $run = run $cmd0,
$args, :err, :out;
Fails. No answer given either by Databricks or the shell. Stdout and stderr are empty.
Splitting the entire command by white space
my #cmd = $cmd.split(/\s+/);
my $run = run $cmd, :err, :out
Error: Got unexpected extra arguments ("s3://bucket", "output":
"s3://bucket/extension",
"sampleID_to_canonical_id_map":
"s3://somefile.csv"}'
Submitting the command as a string
my $cmd = "$cmd0\"$in-dir\", \"output\": \"$out-dir\", \"sampleID_to_canonical_id_map\": \"$map\"}'";
again, stdout and stderr are empty. Exit code 1.
this is something about how run can only accept arrays, and not strings (I'm curious why)
If I copy and paste the command that was given to Perl6's run, it works when given from the shell. It doesn't work when given through perl6. This isn't good, because I have to execute this command hundreds of times.
Perhaps Perl6's shell https://docs.perl6.org/routine/shell would be better? I didn't use that, because the manual suggests that run is safer. I want to capture both stdout and stderr inside a Proc class
EDIT: I've gotten this running with shell but have encountered other problems not related to what I originally posted. I'm not sure if this qualifies as being answered then. I just decided to use backticks with perl5. Yes, backticks are deprecated, but they get the job done.
I'm trying to run a series of shell commands
To run shell commands, call the shell routine. It passes the positional argument you provide it, coerced to a single string, to the shell of the system you're running the P6 program on.
For running commands without involving a shell, call the run routine. The first positional argument is coerced to a string and passed to the operating system as the filename of the program you want run. The remaining arguments are concatenated together with a space in between each argument to form a single string that is passed as a command line to the program being run.
my $cmd0 = 'databricks jobs run-now --job-id 35 --notebook-params ';
That's wrong for both shell and run:
shell only accepts one argument and $cmd0 is incomplete.
The first argument for run is a string interpreted by the OS as the filename of a program to be run and $cmd0 isn't a filename.
So in both cases you'll get either no result or nonsense results.
Your other two experiments are also invalid in their own ways as you discovered.
this is something about how run can only accept arrays, and not strings (I'm curious why)
run can accept a single argument. It would be passed to the OS as the name of the program to be run.
It can accept two arguments. The first would be the program name, the second the command line passed to the program.
It can accept three or more arguments. The first would be the program name, the rest would be concatenated to form the command line passed to the program. (There are cases where this is more convenient coding wise than the two argument form.)
run can also accept a single array. The first element would the program name and the rest the command line passed to it. (There are cases where this is more convenient.)
I just decided to use backticks with perl5. Yes, backticks are deprecated, but they get the job done.
Backticks are subject to code injection and shell interpolation attacks and errors. But yes, if they work, they work.
P6 has direct equivalents of most P5 features. This includes backticks. P6 has two variants:
The safer P6 alternative to backticks is qx. The qx quoting construct calls the shell but does not interpolate P6 variables so it has the same sort of level of danger as using shell with a single quoted string.
The qqx variant is the direct equivalent of P5 backticks or using shell with a double quoted string so it suffers from the same security dangers.
Two mistakes:
the simplistic split cuts up the last, single parameter into multiple arguments
you are passing $cmd to run, not #cmd
use strict;
my #cmd = ('/tmp/dummy.sh', '--param1', 'param2 with spaces');
my $run = run #cmd, :err, :out;
print(#cmd ~ "\n");
print("EXIT_CODE:\t" ~ $run.exitcode ~ "\n");
print("STDOUT:\t" ~ $run.out.slurp ~ "\n");
print("STDERR:\t" ~ $run.err.slurp ~ "\n");
output:
$ cat /tmp/dummy.sh
#!/bin/bash
echo "prog: '$0'"
echo "arg1: '$1'"
echo "arg2: '$2'"
exit 0
$ perl6 dummy.pl
/tmp/dummy.sh --param1 param2 with spaces
EXIT_CODE: 0
STDOUT: prog: '/tmp/dummy.sh'
arg1: '--param1'
arg2: 'param2 with spaces'
STDERR:
If you can avoid generating $cmd as single string, I would generate it into #cmd directly. Otherwise you'll have to implement complex split operation that handles quoting.
Related
I am trying to pull all the jdk packages installed on set of hosts by sending a sql select statement to osquery on linux shell via pssh .
Here is the query:
pssh -h myhosts -i 'echo "SELECT name FROM rpm_packages where name like '%jdk%';"| osqueryi --json'
but usage of "%" is giving me below error.
Error: near line 1: near "%": syntax error
I tried to escape % ,but the error remains same. Any ideas how to overcome this error?
You aren't getting this error from your shell but from the query parser, and it's not actually caused by the % character, but to the ' that immediately precedes it. Look at where you have quotes:
'echo "SELECT name FROM rpm_packages where name like '%jdk%';"| osqueryi --json'
^----------------------------------------------------^ ^-------------------^
These quotes are consumed by the shell when it parses the argument. Single quotes tell the shell to ignore any otherwise-special characters inside and treat what is within the quotes as part of the argument -- but not the quotes themselves.
After shell parsing finishes, the actual, verbatim argument that gets sent to pssh looks like this:
echo "SELECT name FROM rpm_packages where name like %jdk%;"| osqueryi --json
Note that all of the single quotes have been erased. The result is that your query tool sees the % (presumably modulus) operator in a place that it doesn't expect -- right after another operator (like) which makes about as much sense to the parser as name like * jdk. The parser doesn't understand what it means to have two consecutive binary operators, so it complains about the second one: %.
In order to get a literal ' there, you need to jump through this hoop:
'\''
^^^^- start quoting again
|||
|\+-- literal '
|
\---- stop quoting
So, to fix this, replace all ' instances inside the string with '\'':
pssh -h myhosts -i 'echo "SELECT name FROM rpm_packages where name like '\''%jdk%'\'';"| osqueryi --json'
osqueryi accepts a single statement on the command line. Eliminating the echo can make quoting a bit simpler:
osqueryi --json "SELECT * FROM users where username like '%jdk%'"
You will, however, need the quotes to pass through your pssh command line.
While osqueryi is great for short simple things, if you're building a frequent polling service, osqueryd with scheduled queries is generally simpler.
Let's say I have this script:
# prog.p6
my $info = run "uname";
When I run prog.p6, I get:
$ perl6 prog.p6
Linux
Is there a way to store a stringified version of the returned value and prevent it from being output to the terminal?
There's already a similar question but it doesn't provide a specific answer.
You need to enable the stdout pipe, which otherwise defaults to $*OUT, by setting :out. So:
my $proc = run("uname", :out);
my $stdout = $proc.out;
say $stdout.slurp;
$stdout.close;
which can be shortened to:
my $proc = run("uname", :out);
say $proc.out.slurp(:close);
If you want to capture output on stderr separately than stdout you can do:
my $proc = run("uname", :out, :err);
say "[stdout] " ~ $proc.out.slurp(:close);
say "[stderr] " ~ $proc.err.slurp(:close);
or if you want to capture stdout and stderr to one pipe, then:
my $proc = run("uname", :merge);
say "[stdout and stderr] " ~ $proc.out.slurp(:close);
Finally, if you don't want to capture the output and don't want it output to the terminal:
my $proc = run("uname", :!out, :!err);
exit( $proc.exitcode );
The solution covered in this answer is concise.
This sometimes outweighs its disadvantages:
Doesn't store the result code. If you need that, use ugexe's solution instead.
Doesn't store output to stderr. If you need that, use ugexe's solution instead.
Potential vulnerability. This is explained below. Consider ugexe's solution instead.
Documentation of the features explained below starts with the quote adverb :exec.
Safest unsafe variant: q
The safest variant uses a single q:
say qx[ echo 42 ] # 42
If there's an error then the construct returns an empty string and any error message will appear on stderr.
This safest variant is analogous to a single quoted string like 'foo' passed to the shell. Single quoted strings don't interpolate so there's no vulnerability to a code injection attack.
That said, you're passing a single string to the shell which may not be the shell you're expecting so it may not parse the string as you're expecting.
Least safe unsafe variant: qq
The following line produces the same result as the q line but uses the least safe variant:
say qqx[ echo 42 ]
This double q variant is analogous to a double quoted string ("foo"). This form of string quoting does interpolate which means it is subject to a code injection attack if you include a variable in the string passed to the shell.
By default run just passes the STDOUT and STDERR to the parent process's STDOUT and STDERR.
You have to tell it to do something else.
The simplest is to just give it :out to tell it to keep STDOUT. (Short for :out(True))
my $proc = run 'uname', :out;
my $result = $proc.out.slurp(:close);
my $proc = run 'uname', :out;
for $proc.out.lines(:close) {
.say;
}
You can also effectively tell it to just send STDOUT to /dev/null with :!out. (Short for :out(False))
There are more things you can do with :out
{
my $file will leave {.close} = open :w, 'test.out';
run 'uname', :out($file); # write directly to a file
}
print slurp 'test.out'; # Linux
my $proc = run 'uname', :out;
react {
whenever $proc.out.Supply {
.print
LAST {
$proc.out.close;
done; # in case there are other whenevers
}
}
}
If you are going to do that last one, it is probably better to use Proc::Async.
I want to save the output of a program to a variable.
I use the following approach ,but fail.
$ PIPE RUN TEST | DEFINE/JOB VALUE #SYS$PIPE
$ x = f$logical("VALUE")
I got an error:%DCL-W-MAXPARM, too many parameters - reenter command with fewer parameters
\WORLD\
reference :
How to assign the output of a program to a variable in a DCL com script on VMS?
The usual way to do this is to write the output to a file and read from the file and put that into a DCL symbol (or logical). Although not obvious, you can do this with the PIPE command was well:
$ pipe r 2words
hello world
$ pipe r 2words |(read sys$pipe line ; line=""""+line+"""" ; def/job value &line )
$ sh log value
"VALUE" = "hello world" (LNM$JOB_85AB4440)
$
IF you are able to change the program, add some code to it to write the required values into symbols or logicals (see LIB$ routines)
If you can modify the program, using LIB$SET_SYMBOL in the program defines a DCL symbol (what you are calling a variable) for DCL. That's the cleanest way to do this. If it really needs to be a logical, then there are system calls that define logicals.
Im having an issue writing a DCL in OpenVMS in that I need the DCL to call a command and capture its output (but not output the output to the screen) Later on in the DCL I then need to print that output I stored.
Heres an example:
ICE SET FASTER !This command sets my environment to the "Faster" environment.
The above command outputs this if executed directly in OpenVMS:
Initialising TEST Environment to FASTER
--------------------------------------------------------------------------------
Using Test Search rules FASTER
Using Test Search rules FASTER
--------------------------------------------------------------------------------
dcl>
So I created a DCL in an attempt to wrap this output in order to display a more simplified output. Heres my code so far:
!************************************************************************
$ !* Wrapper for setting ICE account. Outputs Environment
$ !************************************************************************
$ on error then goto ABORT_PROCESS
$ICE_DCL_MAIN:
$ ice set 'P1'
$ ICE SHOW
$ EXIT
$ABORT_PROCESS:
$ say "Error ICING to: " + P1
$ EXIT 2
[End of file]
In the lines above ICE SET 'P1' is setting the ice environment, but I dont want this output to be echoed to VMS. But what I do want is to write the output of $ICE SHOW into a variable and then echo that out later on in the DCL (most of which ive omitted for simplification purposes)
So what should be outputted should be:
current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]
Instead of:
Initialising TEST Environment to FASTER
--------------------------------------------------------------------------------
Using Test Search rules FASTER
Using Test Search rules FASTER
--------------------------------------------------------------------------------
current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]
Ive had a look through the manual and im getting a bit confused so I figured I tried here. Id appreciate any pointers. Thanks.
EDIT
Here is what ive come up with after the comments, the problem im having is when I connect to VMS using an emulator such as SecureCRT the correct output is echoed. But when I run the DCL via my SSH2 library in .NET it doesnt output anything. I guess thats because it closes the SYS$OUTPUT stream temporarily or something?
$ !************************************************************************
$ !* Wrapper for setting ICE account. Outputs Environment
$ !************************************************************************
$ on error then goto ABORT_PROCESS
$ICE_DCL_MAIN:
$ DEFINE SYS$OUTPUT NL:
$ ice set 'P1'
$ DEASSIGN SYS$OUTPUT
$ ice show
$ EXIT
$ABORT_PROCESS:
$ say "Error ICING to: " + P1
$ EXIT 2
[End of file]
EDIT 2
So I guess really I need to clarify what im trying to do here. Blocking the output doesnt so matter so much, im merely trying to capture it into a Symbol for example.
So in C# for example you can have a method that returns a string. So you'd have string myResult = vms.ICETo("FASTER"); and it would return that and store it in the variable.
I guess im looking for a similar thing in VMS so that once ive iced to the environment I can call:
$ environment == $ICE SHOW
But I of course get errors with that statement
The command $ assign/user_mode Thing Sys$Output will cause output to be redirected to Thing until you $ deassign/user_mode Sys$Output or next executable image exits. An assignment without the /USER_MODE qualifier will persist until deassigned.
Thing can be a logical name, a file specification (LOG.TXT) or the null device (NLA0:) if you simply want to flush the output.
When a command procedure is executed the output can be redirected using an /OUTPUT qualifier, e.g. $ #FOO/output=LOG.TXT.
And then there is piping ... .
You can redirect the output to a temp file and then print its content later:
$ pipe write sys$output "hi" > tmp.tmp
$ ty tmp.tmp
VMS is not Unix, DCL is not Bash: you can not easily set a DCL symbol from the output of a command.
Your ICE SHOW prints one line, correct? The first word is always "current", correct?
So you can create a hack.
First let me fake your ICE command:
$ create ice.com
$ write sys$output "current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]"
^Z
$
and I define a dcl$path pointing to the directory where this command procedure is
so that I can use/fake the command ICE
$ define dcl$path sys$disk[]
$ ice show
current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]
$
Now what you need, create a command procedure which sets a job logical
$ cre deflog.com
$ def/job/nolog mylog "current''p1'"
^Z
$
And I define a command "current" to run that command procedure:
$ current="#deflog """
Yes, you need three of the double quotes at the end of the line!
And finally:
$ pipe (ice show | #sys$pipe) && mysym="''f$log("mylog")'"
$ sh symb mysym
MYSYM = "current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]"
$
On the other hand, I don't know what you are referring to C# and Java. Can you elaborate on that and tell us what runs where?
You can try using: DEFINE /USER SYS$OUTPUT NL:.
It works only for the next command and you dont need to deassign.
Sharing some of my experience here. I used below methods to redirect outputs to files.
Define/Assign the user output and then execute the required command/script afterwards. Output will be written to .
$define /user sys$output <file_path>
execute your command/script
OR
assign /user <file_path> sys$output
execute your command/script
deassign sys$output
To re-direct in to null device like in Unix (mentioned in above answers), you can use 'nl:' instead of
define /user sys$output nl:
or
assign /user nl: sys$output
I want to write a pig script that takes a filter condition as a command line parameter. From the command line I want to type something like:
pig -p "MY_FILTER=field1 == 0 and field2 == 5" myscript.pig
In my script I have a line:
my_filtered_data = filter my_data by $MY_FILTER;
This works as expected when MY_FILTER has no spaces and I pass quotes around my value; So if I type MY_FILTER=\"field1==0\" at the command line the shell will pass the quotes with the value and pig does the expansion I want. However, the parameter will fail to expand if I supply it like MY_FILTER=\"field1 == 0\"
I've tried a bunch of different quoting techniques and even tried running the command directly from python's subprocess module to ensure my shell wasn't doing something weird.
Which version of Pig do you use? I use 0.9.2 and the following command works for me:
pig -p "F='field1 == 3 AND field2 == 5'" test.pig
But it doesn't work with 0.8.1.