The OpenMPI 4.0.5 on our cluster is built with CUDA support, but I want to benchmark pnetcdf without needing CUDA for that. Since I want to do a number of test runs that I can start on like 1/4th of a node and my tests won't make use of the GPUs I wanted to ask if there is a way to suppress the MPI check for CUDA devices. Because when I simply obtain a SLURM allocation without GPUs, I get lots of errors from that alone.
These errors come from hwloc, and can be suppressed with HWLOC_HIDE_ERRORS=1 but I'd like to know if there is a more specific method.
Steps to reproduce:
frontend$ salloc -n 16 -t 8:00:00 -A k20200
node$ exec bash -l
node$ module load gcc openmpi
node$ mpicc -o /tmp/hello ~/usr/src/helloworld_mpi.c
node$ srun -n 1 /tmp/hello
CUDA: Failed to get number of devices with cudaGetDeviceCount(): no CUDA-capable device is detected
Hello world!, I'm rank 0 of 1!
node$ HWLOC_HIDE_ERRORS=1 srun -n 1 /tmp/hello
Hello world!, I'm rank 0 of 1!
node$ logout
The example code used above is the following but any program without CUDA use is equally useful in this exercise
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#define xmpi(rc) \
do { \
int err = (rc); \
if (err != MPI_SUCCESS) { \
char msg[MPI_MAX_ERROR_STRING + 1]; \
int msg_len; \
\
if (MPI_Error_string(err, msg, &msg_len) \
== MPI_SUCCESS){ \
msg[msg_len] = '\0'; \
\
fprintf(stderr, \
"Problem in MPI call: %d = %s\n", \
err, msg); \
MPI_Abort(MPI_COMM_WORLD, 1); \
} \
} \
} while (0)
int
main(int argc, char *argv[])
{
xmpi(MPI_Init(&argc, &argv));
int rank, size;
xmpi(MPI_Comm_rank(MPI_COMM_WORLD, &rank));
xmpi(MPI_Comm_size(MPI_COMM_WORLD, &size));
printf("Hello world!, I'm rank %d of %d!\n", rank, size);
xmpi(MPI_Finalize());
return EXIT_SUCCESS;
}
I want to write robot framework test case for below situation:
How to check that an element contains child tag(<br>)or not?
First <div>, doesn't contain <br> tag
Second <div>, contains <br> as child tag
Example:
<div class="heading parbase section">
<h6 class="h3 text-success text-left " id="this-heading-is-on-a-single-line" data-qe="qa-heading-line-break-single">
This Heading Is On A Single Line
</h6>
</div>
<div class="heading parbase section">
<h6 class="h3 text-success text-left " id="this-heading-is-on-two-lines" data-qe="qa-heading-line-break-two">
This Heading Is <br>
On Two Lines
</h6>
</div>
Here are two approaches, in the same test case. The first one gets the html value of the parent element, and then you can check for the presence of whatever substring.
The 2nd one uses xpath - it checks does the parent element has a direct child of br type - I'd recommend it, cause it doesn't depend on string parsing, with all its negatives (case, whitespace, etc).
*** Variables ***
${parent element} xpath=//h6[#id="this-heading-is-on-a-single-line"]
*** Test Cases ***
A testcase
${value}= Get Element Attribute ${parent element}#innerHTML
# will be ${true} if there's <br>, ${false} otherwise
${string presence check}= Run Keyword And Return Status Should Contain ${value} <br>
${xpath presence check}= Run Keyword And Return Status Element Should Be Visible ${parent element}/br
# again - ${true} if there's <br>; if it's not guaranteed to be a direct child of the <h6>, use // - e.g. ${parent element}//br
Here is a complete possible solution:
*** Settings ***
Library String
*** Test Cases ***
Negative Test
${myteststring}= Set Variable <div class="heading parbase section"> \ \ \ \ \ <h6 class="h3 text-success text-left " id="this-heading-is-on-a-single-line" data-qe="qa-heading-line-break-single"> \ \ \ \ \ \ \ \ \ This Heading Is On A Single Line \ \ \ \ \ \ \ </h6> \ \ \ \ </div>
${result}= Contains <br>? ${myteststring}
Should Not Be True ${result}
Positive Test
${myteststring}= Set Variable <div class="heading parbase section"> \ \ \ \ \ <h6 class="h3 text-success text-left \ \ \ \ \ \ " id="this-heading-is-on-two-lines" data-qe="qa-heading-line-break-two"> \ \ \ \ \ \ \ \ \ This Heading Is \ <br> \ \ \ \ \ \ \ \ \ On Two Lines \ \ \ \ \ \ \ </h6> </div>
${result}= Contains <br>? ${myteststring}
Should Be True ${result}
*** Keywords ***
Contains <br>?
[Arguments] ${arg1}
${result}= Get Regexp Matches ${arg1} (?im)<br>
${result}= Evaluate len(${result}) > 0
[Return] ${result}
I'm writing a bash script and need to retrieve the process list from apache for a specific user where the C value (processor utilization) is zero. I then want to kill just those processes. my script currently looks like this:
process_user=myuser
max_instances=10
poll_interval=60
while true; do
count=$(ps -u $process_user | wc -l)
echo "count: $count"
if [[ $count > $max_instances ]]; then
killall "$process_user"
echo "Found $count $process_user processes. Killed."
fi
sleep "$poll_interval"
done
The above works fine for identifying the processes for a specific user and killing them. But I don't know how to further limit by whether processor utilization is 0.
Here's a solution:
process_user=myuser
max_instances=10
pool_interval=60
while sleep $pool_interval;do
ps -o pid,c -u $process_user --no-headers \
| awk ' \
$2 > 0{top=top " " $1} \
{count++} \
END { \
if(count > '$max_instances' && top){ \
system("kill " top); \
print "killed: " top \
} \
}'
done
Some explanations:
ps -o pid,c -u $user --no-headers
Show pid and processor utilization (c) of processes owned by $user. Skip the header (PID C)
$2 > 0 - this {} block will be executed only for lines where the second field (processor utilization) is greated then 0.
{ top = top ' ' $1 } append pid (first field - $1) to variable top separating with space
{count++} count all the lines = user processes
END { this block will be executed after all the lines were processes
ok so i have have this
{"status":0,"id":"7aceb216d02ecdca7ceffadcadea8950-1","hypotheses":[{"utterance":"hello how are you","confidence":0.96311796}]}
and at the moment i'm using this shell command to decode it to get the string i need,
echo $x | grep -Po '"utterance":.*?[^\\]"' | sed -e s/://g -e s/utterance//g -e 's/"//g'
but this only works when you have a grep compiled with perl and plus the script i use to get that JSON string is written in perl, so is there any way i can do this same decoding in a simple perl script or a simpler unix command, or better yet, c or objective-c?
the script i'm using to get the json is here, http://pastebin.com/jBGzJbMk and if you want a file to use then download http://trevorrudolph.com/a.flac
How about:
perl -MJSON -nE 'say decode_json($_)->{hypotheses}[0]{utterance}'
in script form:
use JSON;
while (<>) {
print decode_json($_)->{hypotheses}[0]{utterance}, "\n"
}
Well, I'm not sure if I can deduce what you are after correctly, but this is a way to decode that JSON string in perl.
Of course, you'll need to know the data structure in order to get the data you need. The line that prints the "utterance" string is commented out in the code below.
use strict;
use warnings;
use Data::Dumper;
use JSON;
my $json = decode_json
q#{"status":0,"id":"7aceb216d02ecdca7ceffadcadea8950-1","hypotheses":[{"utterance":"hello how are you","confidence":0.96311796}]}#;
#print $json->{'hypotheses'}[0]{'utterance'};
print Dumper $json;
Output:
$VAR1 = {
'status' => 0,
'hypotheses' => [
{
'utterance' => 'hello how are you',
'confidence' => '0.96311796'
}
],
'id' => '7aceb216d02ecdca7ceffadcadea8950-1'
};
Quick hack:
while (<>) {
say for /"utterance":"?(.*?)(?<!\\)"/;
}
Or as a one-liner:
perl -lnwe 'print for /"utterance":"(.+?)(?<!\\)"/g' inputfile.txt
The one-liner is troublesome if you happen to be using Windows, since " is interpreted by the shell.
Quick hack#2:
This will hopefully go through any hash structure and find keys.
my $json = decode_json $str;
say find_key($json, 'utterance');
sub find_key {
my ($ref, $find) = #_;
if (ref $ref) {
if (ref $ref eq 'HASH' and defined $ref->{$find}) {
return $ref->{$find};
} else {
for (values $ref) {
my $found = find_key($_, $find);
if (defined $found) {
return $found;
}
}
}
}
return;
}
Based on the naming, it's possible to have multiple hypotheses. The prints the utterance of each hypothesis:
echo '{"status":0,"id":"7aceb216d02ecdca7ceffadcadea8950-1","hypotheses":[{"utterance":"hello how are you","confidence":0.96311796}]}' | \
perl -MJSON::XS -n000E'
say $_->{utterance}
for #{ JSON::XS->new->decode($_)->{hypotheses} }'
Or as a script:
use feature qw( say );
use JSON::XS;
my $json = '{"status":0,"id":"7aceb216d02ecdca7ceffadcadea8950-1","hypotheses":[{"utterance":"hello how are you","confidence":0.96311796}]}';
say $_->{utterance}
for #{ JSON::XS->new->decode($json)->{hypotheses} };
If you don't want to use any modules from CPAN and try a regex instead there are multiple variants you can try:
# JSON is on a single line:
$json = '{"other":"stuff","hypo":[{"utterance":"hi, this is \"bob\"","moo":0}]}';
# RegEx with negative look behind:
# Match everything up to a double quote without a Backslash in front of it
print "$1\n" if ($json =~ m/"utterance":"(.*?)(?<!\\)"/)
This regex works if there is only one utterance. It doesn't matter what else is in the string around it, since it only searches for the double quoted string following the utterance key.
For a more robust version you could add whitespace where necessary/possible and make the . in the RegEx match newlines: m/"utterance"\s*:\s*"(.*?)(?<!\\)"/s
If you have multiple entries for the utterance confidence hash/object, changing case and weird formatting of the JSON string try this:
# weird JSON:
$json = <<'EOJSON';
{
"status":0,
"id":"an ID",
"hypotheses":[
{
"UtTeraNcE":"hello my name is \"Bob\".",
"confidence":0.0
},
{
'utterance' : 'how are you?',
"confidence":0.1
},
{
"utterance"
: "
thought
so!
",
"confidence" : 0.9
}
]
}
EOJSON
# RegEx with alternatives:
print "$1\n" while ( $json =~ m/["']utterance["']\s*:\s*["'](([^\\"']|\\.)*)["']/gis);
The main part of this RegEx is "(([^\\"]|\\.)*)". Description in detail as extended regex:
/
["'] # opening quotes
( # start capturing parentheses for $1
( # start of grouping alternatives
[^\\"'] # anything that's not a backslash or a quote
| # or
\\. # a backslash followed by anything
) # end of grouping
* # in any quantity
) # end capturing parentheses
["'] # closing quotes
/xgs
If you have many data sets and speed is a concern you can add the o modifier to the regex and use character classes instead of the i modifier. You can suppress the capturing of the alternatives to $2 with clustering parenthesis (?:pattern). Then you get this final result:
m/["'][uU][tT][tT][eE][rR][aA][nN][cC][eE]["']\s*:\s*["']((?:[^\\"']|\\.)*)["']/gos
Yes, sometimes perl looks like a big explosion in a bracket factory ;-)
Just stubmled upon another nice method of doing this, i finaly found how to acsess the Mac OS X JavaScript engine form commandline, heres the script,
alias jsc='/System/Library/Frameworks/JavaScriptCore.framework/Versions/A/Resources/jsc'
x='{"status":0,"id":"7aceb216d02ecdca7ceffadcadea8950-1","hypotheses":[{"utterance":"hello how are you","confidence":0.96311796}]}'
jsc -e "print(${x}['hypotheses'][0]['utterance'])"
Ugh, yes i came up with another answer, im strudying python and it reads arrays in both its python format and the same format as a json so, i jsut made this one liner when your variable is x
python -c "print ${x}['hypotheses'][0]['utterance']"
figured it out for unix but would love to see your perl and c, objective-c answers...
echo $X | sed -e 's/.*utterance//' -e 's/confidence.*//' -e s/://g -e 's/"//g' -e 's/,//g'
:D
shorter copy of the same sed:
echo $X | sed -e 's/.*utterance//;s/confidence.*//;s/://g;s/"//g;s/,//g'
This is similar to another issue, but I only want make to prompt for a value if I'm running a specific target and a mandatory variable has not been specified.
The current code:
install-crontab: PASSWORD ?= "$(shell read -p "Password: "; echo "$$REPLY")"
install-crontab: $(SCRIPT_PATH)
#echo "#midnight \"$(SCRIPT_PATH)\" [...] \"$(PASSWORD)\""
This just results in the following output and no prompt:
Password: read: 1: arg count
#midnight [...] ""
The important point here is that I have to ask only when running this target, and only if the variable has not been defined. I can't use a configure script, because obviously I shouldn't store passwords in a config script, and because this target is not part of the standard installation procedure.
Turns out the problem was that Makefiles don't use Dash / Bash-style quotation, and that Dash's read built-in needs a variable name, unlike Bash. Resulting code:
install-crontab-delicious: $(DELICIOUS_TARGET_PATH)
#while [ -z "$$DELICIOUS_USER" ]; do \
read -r -p "Delicious user name: " DELICIOUS_USER;\
done && \
while [ -z "$$DELICIOUS_PASSWORD" ]; do \
read -r -p "Delicious password: " DELICIOUS_PASSWORD; \
done && \
while [ -z "$$DELICIOUS_PATH" ]; do \
read -r -p "Delicious backup path: " DELICIOUS_PATH; \
done && \
( \
CRONTAB_NOHEADER=Y crontab -l || true; \
printf '%s' \
'#midnight ' \
'"$(DELICIOUS_TARGET_PATH)" ' \
"\"$$DELICIOUS_USER\" " \
"\"$$DELICIOUS_PASSWORD\" " \
"\"$$DELICIOUS_PATH\""; \
printf '\n') | crontab -
Result:
$ crontab -r; make install-crontab-delicious && crontab -l
Delicious user name: a\b c\d
Delicious password: e f g
Delicious backup path: h\ i
no crontab for <user>
#midnight "/usr/local/bin/export_Delicious" "a\b c\d" "e f g" "h\ i"
$ DELICIOUS_PASSWORD=foo make install-crontab-delicious && crontab -l
Delicious user name: bar
Delicious backup path: baz
#midnight "/usr/local/bin/export_Delicious" "a\b c\d" "e f g" "h\ i"
#midnight "/usr/local/bin/export_Delicious" "bar" "foo" "baz"
This code:
treats all input characters as literals, so it works with spaces and backslashes,
avoids problems if the user presses Enter without writing anything,
uses environment variables if they exist, and
works whether crontab is empty or not.
l0b0's answer helped me with a similar problem where I wanted to exit if the user doesn't input 'y'. I ended up doing this:
#while [ -z "$$CONTINUE" ]; do \
read -r -p "Type anything but Y or y to exit. [y/N] " CONTINUE; \
done ; \
if [ ! $$CONTINUE == "y" ]; then \
if [ ! $$CONTINUE == "Y" ]; then \
echo "Exiting." ; exit 1 ; \
fi \
fi
I hope that helps someone. It's hard to find more info about using user input for an if/else in a makefile.