awk with egrep filter unable to return null value with condition - awk

I new in awk, my command as below. When there is no row return need print pass, else print fail. But when there is no value, the pass is unable to display
egrep -v "^\+" /etc/passwd | awk -F: '($1!="root" && $1!="sync" && $1!="shutdown" && $1!="halt" && $3<500 && $7!="/sbin/nologin") {print}' | awk '{if(NR==0||NR<=0||'null') print "pass"; else print "fail"}'
The result should return pass but there is noting print, please advice on this.


consolidate all into one, for example
$ awk -F: '!/^+/ && $1!="root" && ... {f=1; exit}
END {print (f?"fail":"pass")}' /etc/passwd
perhaps better if you set the exit code
$ awk -F: '!/^+/ && $1!="root" && ... {exit 1}' /etc/passwd


This MAY be what you're trying to do:
awk -F: '/^+/ || $1~/^(root|sync|shutdown|halt)$/ || $3>=500 || $7=="/sbin/nologin"{next} {f=1; exit} END{print (f ? "pass" : "fail")}'

Related

Awk syntax error with time layout from logs

I'm getting an syntax error with awk when I'm doing this one liner :
awk '{ if ($3 == '16' && $4 == '23:59:44') {print $0} }' /var/log/radius/radius.log
it gives me a syntax error from the time field. However, when I'm doing:
awk '{ print $4 }' /var/log/radius/radius.log
this gives me the proper format for the time hh:mm:ss so I don't understand why it doesn't work from my one liner ?
Cheers!

Single quotes ' should be used to start and end the awk's main program, you should use " instead here for comparison.
OP's code fix:
awk '{ if ($3 == 16 && $4 == "23:59:44") {print $0} }' Input_file
OR above could be shorten to(awk sh way to do):
awk '($3 == 16 && $4 == "23:59:44")' Input_file

If one string matches at the beginning of last line in a specific file then replace the other string from same line.using regex groups?

I have a file "test"
Below is the content
235788###235788###20200724_103122###SUCCESS
235791###235791###20200724_105934###SUCCESS
235833###235833###20200724_130652###FAILURE
235842###235842###20200724_132721###FAILURE
235852###235852###20200724_134607###FAILURE
235791###235791###20200724_105934###SUCCESS
if last line of this file begin with 235791 then replace string "SUCCESS" to "FAILURE" on just that line.
Expected Output
235788###235788###20200724_103122###SUCCESS
235791###235791###20200724_105934###SUCCESS
235833###235833###20200724_130652###FAILURE
235842###235842###20200724_132721###FAILURE
235852###235852###20200724_134607###FAILURE
235791###235791###20200724_105934###FAILURE
Below is the sample code
id = 235791
last_build_id = `tail -1 test | awk -F'###' '{print \$1}'`
if (id == last_build_id ){
sed -i '$s/SUCCESS/FAILURE/' test
}
I would like to avoid these many lines and use one line shell command using regex groups or in any other simple way.

sed might be easier here
$ sed -E '$s/(^235791#.*)SUCCESS$/\1FAILURE/' file
you can add -i for in place update.
To pass id as a variable
$ id=235791; sed -E '$s/(^'$id'#.*)SUCCESS$/\1FAILURE/' file
you should double quote "$id" ideally, but if you're sure about the contents you may get away without.

With GNU sed
sed -E '${/^235791\>/ s/SUCCESS$/FAILURE/}' file
Or with the BSD sed on MacOS
sed -E '${/^235791#/ s/SUCCESS$/FAILURE/;}' file
When working with "the last X in the file", it's often easier to reverse the file and work with "the first X":
tac file | awk '
BEGIN {FS = OFS = "###"}
NR == 1 && $1 == 235791 && $NF == "SUCCESS" {$NF = "FAILURE"}
1
' | tac

Could you please try following, written and tested with shown samples in GNU awk. You need not to use many commands for this one, we could do this in a single awk itself.
One liner form of code:
awk -v id="$your_shell_variable" 'BEGIN{ FS=OFS="###" } NR>1{print prev} {prev=$0} END{if($1==id && $NF=="SUCCESS"){$NF="FAILURE"}; print}' Input_file > temp && mv temp Input_file
Explanation: Adding detailed explanation for above.
awk -v id="$your_shell_variable"' ##Starting awk program from here.
NR>1{ ##Checking condition if prev is NOT NULL then do following.
print prev ##Printing prev here.
}
{
prev=$0 ##Assigning current line to prev here.
}
END{ ##Starting END block of this program from here.
if($1==id && $NF=="SUCCESS"){ ##Checking condition if first field is 235791 and last field is SUCCESS then do following.
$NF="FAILURE" ##Setting last field FAILURE here.
}
print ##Printing last line here.
}
' Input_file > temp && mv temp Input_file ##Mentioning Input_file name here.
2nd solution: As per Ed sir's comment some awk's don't support $1, $NF in END sections so if above doesn't work for someone please try more generic solution as follows.
One liner form of solution(since specifically asking it):
awk -v id="$your_shell_variable" 'BEGIN{ FS=OFS="###" } NR>1{print prev} {prev=$0} END{num=split(prev,array,"###");if(array[1]==id && array[num]=="SUCCESS"){array[num]="FAILURE"};for(i=1;i<=num;i++){val=(val?val OFS:"")array[i]};print val}' Input_file > temp && mv temp Input_file
Detailed level(non-one liner code):
awk -v id="$your_shell_variable" '
BEGIN{ FS=OFS="###" }
NR>1{
print prev
}
{
prev=$0
}
END{
num=split(prev,array,"###")
if(array[1]==id && array[num]=="SUCCESS"){
array[num]="FAILURE"
}
for(i=1;i<=num;i++){
val=(val?val OFS:"")array[i]
}
print val
}
' Input_file > temp && mv temp Input_file

$ awk -v val='235791' '
BEGIN { FS=OFS="###" }
NR>1 { print prev }
{ prev=$0 }
END {
$0=prev
if ($1 == val) {
$NF="FAILURE"
}
print
}
' file
235788###235788###20200724_103122###SUCCESS
235791###235791###20200724_105934###SUCCESS
235833###235833###20200724_130652###FAILURE
235842###235842###20200724_132721###FAILURE
235852###235852###20200724_134607###FAILURE
235791###235791###20200724_105934###FAILURE

awk to match fields condition from two files

Would like to match Input.csv file based on field $2 (Country) And field $4 (Type) with Ref.csv
If it matches, then need to print entire line from Input.csv and corresponding Amount field from Ref.csv
If it is not match, then print entire line from Input.csv and "NotFound"
For example: Country= "Angola" and Type="Voice" then need to print the value from $3 (Amount=10) Ref.csv
Country= "Angola" and Type="Data" then need to print the value from $4 (Amount=20) Ref.csv
Country= "Angola" and Type="SMS" then need to print the value from $5 (Amount=30) Ref.csv
Country= "Yemen" is not available in the Ref.csv , so need to print as "NotFound"
Input.csv
Month,Country,Desc,Type
Nov'2017,Angola,abc,Voice
Nov'2017,Angola,xxx,Voice
Nov'2017,Angola,abc,Data
Nov'2017,Angola,abc,SMS
Nov'2017,Brazil,def,Data
Nov'2017,Brazil,def,Voice
Nov'2017,Yemen,yyy,Data
Nov'2017,Zambia,ghi,SMS
Nov'2017,Zambia,xxx,SMS
Nov'2017,Zimbabwe,yyy,Voice
Ref.csv
Country,Desc,Voice,Data,SMS
Albania,abc,5,10,15
Angola,def,10,20,30
Bahrain,ghi,2,4,6
Brazil,jkl,45,30,15
Zambia,mno,5,7,9
Expected Output.csv
Month,Country,Desc,Type,Amount
Nov'2017,Angola,abc,Voice,10
Nov'2017,Angola,xxx,Voice,10
Nov'2017,Angola,abc,Data,20
Nov'2017,Angola,abc,SMS,30
Nov'2017,Brazil,def,Data,30
Nov'2017,Brazil,def,Voice,45
Nov'2017,Yemen,yyy,Data,NotFound
Nov'2017,Zambia,ghi,SMS,9
Nov'2017,Zambia,xxx,SMS,9
Nov'2017,Zimbabwe,yyy,Voice,NotFound
Looking for your suggestions ...!!!
Edit Command #1
awk 'BEGIN { FS=OFS="," }
NR==1{print $0,"Amount"; next} {
map[$1,"Voice"]=$3
map[$1,"Data"]=$4
map[$1,"SMS"]=$5
(($2","$4) in map ) { print $0,map[f2,f4] } else { print $0,"NotFound" }
}' Ref.csv Input.csv

awk -F, -v OFS=, '
NR == FNR {
for (i=3; i<=NF; i++)
if (FNR == 1)
head[i] = $i
else
ref[$1,head[i]] = $i
next
}
FNR == 1 {print $0, "Amount"; next}
{
amount = (($2,$4) in ref) ? ref[$2,$4] : "NotFound"
print $0, amount
}
' Ref.csv Input.csv

Run this script within the same directory where your input files are located.
#!/bin/sh
cat Input.csv | while read line
do
country=`echo $line | awk -F ',' '{print $2}'`
type=`echo $line | awk -F ',' '{print $4}'`
ref=`grep -i $country Ref.csv`
if [ -z "$ref" ];
then
echo $line,NotFound;
else
case $type in
"Voice")
data=`echo $ref | awk -F ',' '{print $3}'`
;;
esac
case $type in
"Data")
data=`echo $ref | awk -F ',' '{print $4}'`
;;
esac
case $type in
"SMS")
data=`echo $ref | awk -F ',' '{print $5}'`
;;
esac
echo $line,$data;
fi;
done;
I tested it and it outputs exactly what you want.

Find "complete cases" with awk

Using awk, how can I output the lines of a file that have all fields non-null without manually specifying each column?
foo.dat
A||B|X
A|A|1|
|1|2|W
A|A|A|B
Should return:
A|A|A|B
In this case we can do:
awk -F"|" -v OFS="|" '$1 != "" && $2 != "" && $3 != "" && $4 != "" { print }' foo.dat
But is there a way to do this without specifying each column?

You can loop over all fields and skip the record if any of the fields are empty:
$ awk -F'|' '{ for (i=1; i<=NF; ++i) { if (!$i) next } }1' foo.dat
A|A|A|B
if (!$i) is "if field i is not non-empty", and 1 is short for "print the line", but it is only hit if next was not executed for any of the fields of the current line.

Another in awk:
$ awk -F\| 'gsub(/[^|]+(\||$)/,"&")==NF' file
A|A|A|B
print record if there are NF times | terminating (non-empty, |-excluding) strings.

awk '!/\|\|/&&!/\|$/&&!/^\|/' file
A|A|A|B

awk beginner question about putting some commands in a program file like abc.awk

i need to write this for counting lines between NR= 0-500, 500-1000,1000-1500 etc. till the end, and which start with 1.
awk "{NR<=500 && "/^1/" && sum+=1}; END {print sum}" 1.txt
awk "{NR>500 && NR<=1000 && "/^1/" && sum+=1}; END {print sum}" 1.txt
awk "{NR>1000 && NR<=1500 && "/^1/" && sum+=1}; END {print sum}" 1.txt
.....
here is my question, how can i put these in to a file(*.awk) to run at the same time..

you can try put these commands in a file eg myscript.awk
NR<=500 && /^1/ { sum1 += 1 }
NR>500 && NR<=1000 && /^1/ { sum2 += 1}
NR>1000 && NR<=1500 && /^1/ { sum3+=1 }
END { print sum1,sum2.sum3 }
then run on command line
awk -f myscript.awk 1.txt
you do not need to call awk on the same file 3 times! Its an example of inefficient coding.
you can out a shebang #!/usr/bin/awk -f in the first line of your script only if you want to run it like ./myscript.awk, otherwise , you can leave it out

If you want the script to be pure awk, you need a shebang line with a -f. Make this line one of the file:
#!/usr/bin/awk -f
(or adjust according to the location of the awk you want to use)
One big drawback of that approach is that you must hard code the path of awk in the script. The typical solution to that problem (to use /usr/bin/env) does not work with awk or sed, because they require the -f in the shebang. If you just want to use a shell script wrapper around your awk, you can do something like:
#!/bin/sh
file=${1-1.txt}
awk "{NR<=500 && "/^1/" && sum+=1}; END {print sum}" $file
awk "{NR>500 && NR<=1000 && "/^1/" && sum+=1}; END {print sum}" $file
awk "{NR>1000 && NR<=1500 && "/^1/" && sum+=1}; END {print sum}" $file

Untested, but this should replace all those duplicated lines:
awk '/^1/ {sum++} NR % 500 == 0 {print sum; sum=0} END {print sum}' 1.txt