Is it possible to time out a user input for the c shell? My code so far is :
#!/bin/csh -f
set COUNT = 5
printf "INFO: Start ok (0/1)? "
set INPUT = 0
while ($COUNT > 0 && $INPUT == 0)
printf "\b%d" "$COUNT"
set INPUT = <$
sleep 1
# COUNT --
end
if ($INPUT == 1) then
./execute.sh
end
If no input is given, I want to execute a shell script; if not i want to skip this part.
Unfortunately, the skript does not skip the input part but waits for the input. Any solutions for this problem?
Thanks a lot guys!!!
try this for non-blocking user input in tcsh shell:
set TMPFILE = `mktemp`
set COUNT = 5
printf "INFO: Start ok (0/1)? "
stty -F /dev/tty -icanon
while ($COUNT > 0 && -z $TMPFILE)
printf "\b%d" "$COUNT"
sleep 1
(dd bs=1 count=1 iflag=nonblock > $TMPFILE) >& /dev/null
set INPUT = `cat $TMPFILE`
# COUNT--
end
stty -F /dev/tty icanon
echo ""
if ("$INPUT" == "1") then
./execute.sh
endif
Related
Using gawk 5, we are able to launch shell commands with something like:
command = "echo toto"
command | getline
However, this hangs until the command is complete, for example:
BEGIN {
command = "{ echo start >> test.log ; sleep 10 ; echo stop >> test.log ;}"
command | getline
print "terminated"
}
I tried using a & at the end of the command, without success:
BEGIN {
command = "{ echo start >> test.log ; sleep 10 ; echo stop >> test.log ;} &"
command | getline
print "terminated"
}
I would like to be able to launch shell commands as new processes and "forget" about them. For example:
BEGIN {
command1 = "dosomething.sh &"
command1 | getline
command2 = "dosomething2.sh &"
command2 | getline
print "terminated"
}
Can this be achieved using gawk 5 only ?
| getline waits for something to read. So output something so it can read it.
$ awk 'BEGIN { cmd="{ echo something; ( echo start >&2 ; sleep 1; echo stop >&2 ) & }"; cmd | getline; print "Awk end"; }' <<<'' ; echo "Awk end"; sleep 2
Awk end
start
After awk
stop
I'm trying to check if a multi-line string exists in a file using common bash commands (grep, awk, ...).
I want to have a file with a few lines, plain lines, not patterns, that should exists in another file and create a command (sequence) that checks if it does. If grep could accept arbitrary multiline patterns, I'd do it with something similar to
grep "`cat contentfile`" targetfile
As with grep I'd like to be able to check the exit code from the command. I'm not really interested in the output. Actually no output would be preferred since then I don't have to pipe to /dev/null.
I've searched for hints, but can't come up with a search that gives any good hits. There's How can I search for a multiline pattern in a file?, but that is about pattern matching.
I've found pcre2grep, but need to use "standard" *nix tools.
Example:
contentfile:
line 3
line 4
line 5
targetfile:
line 1
line 2
line 3
line 4
line 5
line 6
This should match and return 0 since the sequence of lines in the content file is found (in the exact same order) in the target file.
EDIT: Sorry for not being clear about the "pattern" vs. "string" comparison and the "output" vs. "exit code" in the previous versions of this question.
You didn't say if you wanted a regexp match or string match and we can't tell since you named your search file "patternfile" and a "pattern" could mean anything and at one point you imply you want to do a string match (check if a multi-line _string_ exists) but then you're using grep and pcregpre with no stated args for string rather than regexp matches.
In any case, these will do whatever it is you want using any awk (which includes POSIX standard awk and you said you wanted to use standard UNIX tools) in any shell on every UNIX box:
For a regexp match:
$ cat tst.awk
NR==FNR { pat = pat $0 ORS; next }
{ tgt = tgt $0 ORS }
END {
while ( match(tgt,pat) ) {
printf "%s", substr(tgt,RSTART,RLENGTH)
tgt = substr(tgt,RSTART+RLENGTH)
}
}
$ awk -f tst.awk patternfile targetfile
line 3
line 4
line 5
For a string match:
$ cat tst.awk
NR==FNR { pat = pat $0 ORS; next }
{ tgt = tgt $0 ORS }
END {
lgth = length(pat)
while ( beg = index(tgt,pat) ) {
printf "%s", substr(tgt,beg,lgth)
tgt = substr(tgt,beg+lgth)
}
}
$ awk -f tst.awk patternfile targetfile
line 3
line 4
line 5
Having said that, with GNU awk you could do the following if you're OK with a regexp match and backslash interpretation of the patternfile contents (so \t is treated as a literal tab):
$ awk -v RS="$(cat patternfile)" 'RT!=""{print RT}' targetfile
line 3
line 4
line 5
or with GNU grep:
$ grep -zo "$(cat patternfile)" targetfile | tr '\0' '\n'
line 3
line 4
line 5
There are many other options depending on what kind of match you're really trying to do and which tools versions you have available.
EDIT: Since OP needs outcome of command in form of true or false(yes or no), so edited command in that manner now(created and tested in GNU awk).
awk -v message="yes" 'FNR==NR{a[$0];next} ($0 in a){if((FNR-1)==prev){b[++k]=$0} else {delete b;k=""}} {prev=FNR}; END{if(length(b)>0){print message}}' patternfile targetfile
Could you please try following, tested with given samples and it should print all continuous lines from pattern file if they are coming in same order in target file(count should be at least 2 for continuous lines in this code).
awk '
FNR==NR{
a[$0]
next
}
($0 in a){
if((FNR-1)==prev){
b[++k]=$0
}
else{
delete b
k=""
}
}
{
prev=FNR
}
END{
for(j=1;j<=k;j++){
print b[j]
}
}' patternfile targetfile
Explanation: Adding explanation for above code here.
awk ' ##Starting awk program here.
FNR==NR{ ##FNR==NR will be TRUE when first Input_file is being read.
a[$0] ##Creating an array a with index $0.
next ##next will skip all further statements from here.
}
($0 in a){ ##Statements from here will will be executed when 2nd Input_file is being read, checking if current line is present in array a.
if((FNR-1)==prev){ ##Checking condition if prev variable is equal to FNR-1 value then do following.
b[++k]=$0 ##Creating an array named b whose index is variable k whose value is increment by 1 each time it comes here.
}
else{ ##Mentioning else condition here.
delete b ##Deleting array b here.
k="" ##Nullifying k here.
}
}
{
prev=FNR ##Setting prev value as FNR value here.
}
END{ ##Starting END section of this awk program here.
for(j=1;j<=k;j++){ ##Starting a for loop here.
print b[j] ##Printing value of array b whose index is variable j here.
}
}' patternfile targetfile ##mentioning Input_file names here.
another solution in awk:
echo $(awk 'FNR==NR{ a[$0]; next}{ x=($0 in a)?x+1:0 }x==length(a){ print "OK" }' patternfile targetfile )
This returns "OK" if there is a match.
a one-liner:
$ if [ $(diff --left-column -y patternfile targetfile | grep '(' -A1 -B1 | tail -n +2 | head -n -1 | wc -l) == $(cat patternfile | wc -l) ]; then echo "ok"; else echo "error"; fi
explanation:
first is to compare the two files using diff:
diff --left-column -y patternfile targetfile
> line 1
> line 2
line 3 (
line 4 (
line 5 (
> line 6
then filter to show only interesting lines, which are the lines the '(', plus extra 1-line before, and after match, to check if lines in patternfile match without a break.
diff --left-column -y patternfile targetfile | grep '(' -A1 -B1
> line 2
line 3 (
line 4 (
line 5 (
> line 6
Then leave out the first, and last line:
diff --left-column -y patternfile targetfile | grep '(' -A1 -B1 | tail -n +2 | head -n -1
line 3 (
line 4 (
line 5 (
add some code to check if the number of lines match the number of lines in the patternfile:
if [ $(diff --left-column -y patternfile targetfile | grep '(' -A1 -B1 | tail -n +2 | head -n -1 | grep '(' | wc -l) == $(cat patternfile | wc -l) ]; then echo "ok"; else echo "error"; fi
ok
to use this with a return-code, a script could be created like this:
#!/bin/bash
patternfile=$1
targetfile=$2
if [ $(diff --left-column -y $patternfile $targetfile | grep '(' -A1 -B1 | tail -n +2 | head -n -1 | grep '(' | wc -l) == $(cat $patternfile | wc -l) ];
then
exit 0;
else
exit 1;
fi
The test (when above script is named comparepatterns):
$ comparepatterns patternfile targgetfile
echo $?
0
The easiest way to do this is to use a sliding window. First you read the pattern file, followed by file to search.
(FNR==NR) { a[FNR]=$0; n=FNR; next }
{ b[FNR]=$0 }
(FNR >= n) { for(i=1; i<=n;++i) if (a[i] != b[FNR-n+i]) { delete b[FNR-n+1]; next}}
{ print "match at", FNR-n+1}
{ r=1}
END{ exit !r}
which you call as
awk -f script.awk patternFile searchFile
Following up on a comment from Cyrus, who pointed to How to know if a text file is a subset of another, the following Python one-liner does the trick
python -c "content=open('content').read(); target=open('target').read(); exit(0 if content in target else 1);"
Unless you're talking about 10 GB+, here's an awk-based solution that's fast and clean :
mawk '{ exit NF==NR }' RS='^$' FS="${multiline_pattern}"
The pattern exists only in the file "${m2p}"
which is embedded within multi-file pipeline of 1st test,
but not 2nd one
This solution, for now, doesn't auto handle instances where regex meta-character escaping is needed. Alter it as you see fit.
Unless the pattern occurs far too often, it might even save time to do it all at once instead of having to check line-by-line, including saving lines along the way in some temp pattern space.
NR is always 1 there since RS is forced to the tail end of the input. NF is larger than 1 only when the pattern is found. By evaluating exit NF == NR, it inverts the match, thus matching structure of posix exit codes.
% echo; ( time ( \
\
echo "\n\n multi-line-pattern :: \n\n " \
"-------------\n${multiline_pattern}\n" \
" -----------\n\n " \
"$( nice gcat "${m2m}" "${m3m}" "${m3l}" "${m2p}" \
"${m3r}" "${m3supp}" "${m3t}" | pvE0 \
\
| mawk2 '{ exit NF == NR
}' RS = '^$' \
FS = "${multiline_pattern}" \
\
) exit code : ${?} " ) ) | ecp
in0: 3.10GiB 0:00:01 [2.89GiB/s] [2.89GiB/s] [ <=> ]
( echo ; ) 0.77s user 1.74s system 110% cpu 2.281 total
multi-line-pattern ::
-------------
77138=1159=M
77138=1196=M
77138=1251=M
77138=1252=M
77138=4951=M
77138=16740=M
77138=71501=M
-----------
exit code : 0
% echo; ( time ( \
\
echo "\n\n multi-line-pattern :: \n\n " \
"-------------\n${multiline_pattern}\n" \
" -----------\n\n " \
"$( nice gcat "${m2m}" "${m3m}" "${m3l}" \
"${m3r}" "${m3supp}" "${m3t}" | pvE0 \
\
| mawk2 '{ exit NF == NR
}' RS = '^$' \
FS = "${multiline_pattern}" \
\
) exit code : ${?} " ) ) | ecp
in0: 2.95GiB 0:00:01 [2.92GiB/s] [2.92GiB/s] [ <=> ]
( echo ; ) 0.64s user 1.65s system 110% cpu 2.074 total
multi-line-pattern ::
-------------
77138=1159=M
77138=1196=M
77138=1251=M
77138=1252=M
77138=4951=M
77138=16740=M
77138=71501=M
-----------
exit code : 1
If your pattern is the full file, then something like this - even when using the full file as a single gigantic 153 MB pattern, it finished in less than 2.4 secs against ~3 GB input.
echo
( time ( nice gcat "${m2m}" "${m3m}" "${m3l}" "${m3r}" "${m3supp}" "${m3t}" | pvE0 \
\
| mawk2 -v pattern_file="${m2p}" '
BEGIN {
RS = "^$"
getline FS < pattern_file
close(pattern_file)
} END {
exit NF == NR }' ; echo "\n\n exit code :: $?\n\n" ))|ecp;
du -csh "${m2p}" ;
( time ( nice gcat "${m2m}" "${m3m}" "${m3l}" \
"${m2p}" "${m3r}" "${m3supp}" "${m3t}" | pvE0 \
\
| mawk2 -v pattern_file="${m2p}" '
BEGIN {
RS = "^$"
getline FS < pattern_file
close(pattern_file)
} END {
exit NF == NR }' ; echo "\n\n exit code :: $?\n\n" ))|ecp;
in0: 2.95GiB 0:00:01 [2.58GiB/s] [2.58GiB/s] [ <=> ]
( nice gcat "${m2m}" "${m3m}" "${m3l}" "${m3r}" "${m3supp}" "${m3t}" | pvE 0.)
0.82s user 1.71s system 111% cpu 2.260 total
exit code :: 1
153M /Users/************/m2map_main.txt
153M total
in0: 3.10GiB 0:00:01 [2.56GiB/s] [2.56GiB/s] [ <=> ]
( nice gcat "${m2m}" "${m3m}" "${m3l}" "${m2p}" "${m3r}" "${m3supp}" "${m3t}")
0.83s user 1.79s system 112% cpu 2.339 total
exit code :: 0
Found a portable solution using patch command. The idea is to create a diff/patch in remove direction and check if it could be applied to the source file. Sadly there is no option for a dry-run (in my old patch version). So we've to do the patch and remove the temporary files.
The shell part around is optimized for my ksh usage:
file_in_file() {
typeset -r vtmp=/tmp/${0%.sh}.$$.tmp
typeset -r vbasefile=$1
typeset -r vcheckfile=$2
typeset -ir vlines=$(wc -l < "$vcheckfile")
{ echo "1,${vlines}d0"; sed 's/^/< /' "$vcheckfile"; } |
patch -fns -F0 -o "$vtmp" "$vbasefile" >/dev/null 2>&1
typeset -ir vrc=$?
rm -f "$vtmp"*
return $vrc
}
Explanation:
set variables for local usage (on newer bash you should use declare instead)
count lines of input file
create a patch/diff file in-memory (the line with the curly brackets)
use patch with strict settings patch -F0
cleanup (also eventually created reject files: rm -f "$vtmp"*)
return RC of patch
In POSIX awk, how do I get the exit status (return code) from command after processing its output via command | getline var? I want my awk script to exit 1 if command exited with a non-zero exit status.
For example, suppose I had an awk script named foo.awk that looks like this:
function close_and_get_exit_status(cmd) {
# magic goes here...
}
BEGIN {
cmd = "echo foo; echo bar; echo baz; false"
while ((cmd | getline line) > 0)
print "got a line of text: " line
if (close_and_get_exit_status(cmd) != 0) {
print "ERROR: command '" cmd "' failed" | "cat >&2"
exit 1
}
print "command '" cmd "' was successful"
}
then I want the following to happen:
$ awk -f foo.awk
got a line of text: foo
got a line of text: bar
got a line of text: baz
ERROR: command 'echo foo; echo bar; echo baz; false' failed
$ echo $?
1
According to the POSIX specification for awk, command | getline returns 1 for successful input, zero for end-of-file, and -1 for an error. It's not an error if command exits with a non-zero exit status, so this can't be used to see if command is done and has failed.
Similarly, close() can't be used for this purpose: close() returns non-zero only if the close fails, not if the associated command returns a non-zero exit status. (In gawk, close(command) returns the exit status of command. This is the behavior I'd like, but I think it violates the POSIX spec and not all implementations of awk behave this way.)
The awk system() function returns the exit status of the command, but as far as I can tell there's no way to use getline with it.
The simplest thing to do is just echo the exit status from shell after the command executes and then read that with getline. e.g.
$ cat tst.awk
BEGIN {
cmd = "echo foo; echo bar; echo baz; false"
mod = cmd "; echo \"$?\""
while ((mod | getline line) > 0) {
if (numLines++)
print "got a line of text: " prev
prev = line
}
status = line
close(mod)
if (status != 0) {
print "ERROR: command '" cmd "' failed" | "cat >&2"
exit 1
}
print "command '" cmd "' was successful"
}
$ awk -f tst.awk
got a line of text: foo
got a line of text: bar
got a line of text: baz
ERROR: command 'echo foo; echo bar; echo baz; false' failed
$ echo $?
1
In case anyone's reading this and considering using getline, make sure you read http://awk.freeshell.org/AllAboutGetline and FULLY understand all the caveats and implications of doing so first.
Not an ideal solution, but you can do:
"command || echo failure" | getline var; ... if( var == "failure" ) exit;
There is some ambiguity in that you have to select the string "failure" in such a way that command can never generate the same string, but perhaps this is an adequate workaround.
The following is horrifically complicated, but it:
is POSIX conformant (mostly -- fflush() isn't yet in the POSIX standard, but it will be and it's widely available)
is general (it works no matter what kind of output is emitted by the command)
does not introduce any processing delay. The accepted answer to this question makes a line available only after the next line has been printed by the command. If the command slowly outputs lines and responsiveness is important (e.g., occasional events printed by an IDS system that should trigger a firewall change or email notification), this answer might be more appropriate than the accepted answer.
The basic approach is to echo the exit status/return value after the command completes. If this last line is non-zero, exit the awk script with an error. To prevent the code from mistaking a line of text output by the command for the exit status, each line of text output by the command is prepended with a letter that is later stripped off.
function stderr(msg) { print msg | "cat >&2"; }
function error(msg) { stderr("ERROR: " msg); }
function fatal(msg) { error(msg); exit 1; }
# Wrap cmd so that each output line of cmd is prefixed with "d".
# After cmd is done, an additional line of the format "r<ret>" is
# printed where "<ret>" is the integer return code/exit status of the
# command.
function safe_cmd_getline_wrap(cmd) {
return \
"exec 3>&1;" \
"ret=$(" \
" exec 4>&1;" \
" { ( "cmd" ) 4>&-; echo $? >&4; } 3>&- |" \
" awk '{print\"d\"$0;fflush()}' >&3 4>&-;" \
");" \
"exec 3>&-;" \
"echo r${ret};"
}
# like "cmd | getline line" except:
# * if getline fails, the awk script exits with an error
# * if cmd fails (returns non-zero), the awk script exits with an
# error
# * safe_cmd_getline_close(cmd) must be used instead of close(cmd)
function safe_cmd_getline(cmd, wrapped_cmd,ret,type) {
wrapped_cmd = safe_cmd_getline_wrap(cmd)
ret = (wrapped_cmd | getline line)
if (ret == -1) fatal("failed to read line from command: " cmd)
if (ret == 0) return 0
type = substr(line, 1, 1)
line = substr(line, 2)
if (type == "d") return 1
if (line != "0") fatal("command '" cmd "' failed")
return 0
}
function safe_cmd_getline_close(cmd) {
if (close(safe_cmd_getline_wrap(cmd))) fatal("failed to close " cmd)
}
You use the above like this:
cmd = "ls no-such-file"
while (safe_cmd_getline(cmd)) {
print "got a line of text: " line
}
safe_cmd_getline_close(cmd)
If you have mktemp command, you could store the exit status in a temporary file:
#!/bin/sh
set -e
file=$(mktemp)
finish() {
rm -f "$file"
}
trap 'finish' EXIT
trap 'finish; trap - INT; kill -s INT $$' INT
trap 'finish; trap - TERM; kill $$' TERM
awk -v file="$file" 'BEGIN{
o_cmd="echo foo; echo bar; echo baz; false"
cmd = "("o_cmd "); echo $? >\""file"\""
print cmd
while ((cmd | getline) > 0) {
print "got a line of text: " $0
}
close(cmd)
getline ecode <file; close(file)
print "exit status:", ecode
if(ecode)exit 1
}'
I'm writing a bash script and need to retrieve the process list from apache for a specific user where the C value (processor utilization) is zero. I then want to kill just those processes. my script currently looks like this:
process_user=myuser
max_instances=10
poll_interval=60
while true; do
count=$(ps -u $process_user | wc -l)
echo "count: $count"
if [[ $count > $max_instances ]]; then
killall "$process_user"
echo "Found $count $process_user processes. Killed."
fi
sleep "$poll_interval"
done
The above works fine for identifying the processes for a specific user and killing them. But I don't know how to further limit by whether processor utilization is 0.
Here's a solution:
process_user=myuser
max_instances=10
pool_interval=60
while sleep $pool_interval;do
ps -o pid,c -u $process_user --no-headers \
| awk ' \
$2 > 0{top=top " " $1} \
{count++} \
END { \
if(count > '$max_instances' && top){ \
system("kill " top); \
print "killed: " top \
} \
}'
done
Some explanations:
ps -o pid,c -u $user --no-headers
Show pid and processor utilization (c) of processes owned by $user. Skip the header (PID C)
$2 > 0 - this {} block will be executed only for lines where the second field (processor utilization) is greated then 0.
{ top = top ' ' $1 } append pid (first field - $1) to variable top separating with space
{count++} count all the lines = user processes
END { this block will be executed after all the lines were processes
I want to run the system command in an awk script and get its output stored in a variable. I've been trying to do this, but the command's output always goes to the shell and I'm not able to capture it. Any ideas on how this can be done?
Example:
$ date | awk --field-separator=! {$1 = system("strip $1"); /*more processing*/}
Should call the strip system command and instead of sending the output to the shell, should assign the output back to $1 for more processing. Rignt now, it's sending output to shell and assigning the command's retcode to $1.
Note: Coprocess is GNU awk specific.
Anyway another alternative is using getline
cmd = "strip "$1
while ( ( cmd | getline result ) > 0 ) {
print result
}
close(cmd)
Calling close(cmd) will prevent awk to throw this error after a number of calls :
fatal: cannot open pipe `…' (Too many open files)
To run a system command in awk you can either use system() or cmd | getline.
I prefer cmd | getline because it allows you to catch the value into a variable:
$ awk 'BEGIN {"date" | getline mydate; close("date"); print "returns", mydate}'
returns Thu Jul 28 10:16:55 CEST 2016
More generally, you can set the command into a variable:
awk 'BEGIN {
cmd = "date -j -f %s"
cmd | getline mydate
close(cmd)
}'
Note it is important to use close() to prevent getting a "makes too many open files" error if you have multiple results (thanks mateuscb for pointing this out in comments).
Using system(), the command output is printed automatically and the value you can catch is its return code:
$ awk 'BEGIN {d=system("date"); print "returns", d}'
Thu Jul 28 10:16:12 CEST 2016
returns 0
$ awk 'BEGIN {d=system("ls -l asdfasdfasd"); print "returns", d}'
ls: cannot access asdfasdfasd: No such file or directory
returns 2
Figured out.
We use awk's Two-way I/O
{
"strip $1" |& getline $1
}
passes $1 to strip and the getline takes output from strip back to $1
gawk '{dt=substr($4,2,11); gsub(/\//," ",dt); "date -d \""dt"\" +%s"|getline ts; print ts}'
You can use this when you need to process a grep output:
echo "some/path/exex.c:some text" | awk -F: '{ "basename "$1"" |& getline $1; print $1 " ==> " $2}'
option -F: tell awk to use : as field separator
"basename "$1"" execute shell command basename on first field
|& getline $1 reads output of previous shell command in substream
output:
exex.c ==> some text
I am using macOS's awk and I also needed exit status of the command. So I extended #ghostdog74's solution to get the exit status too:
Exit if non-zero exit status:
cmd = <your command goes here>
cmd = cmd" ; printf \"\n$?\""
last_res = ""
value = ""
while ( ( cmd | getline res ) > 0 ) {
if (value == "") {
value = last_res
} else {
value = value"\n"last_res
}
last_res = res
}
close(cmd)
# Now `res` has the exit status of the command
# and `value` has the complete output of command
if (res != 0) {
exit 1
} else {
print value
}
So basically I just changed cmd to print exit status of the command on a new line. After the execution of the above while loop, res would contain the exit status of the command and
value would contain the complete output of the command.
Honestly not a very neat way and I myself would like to know if there is some better way.