Not extracting data between two patterns - awk

I have tried this awk command, but for some reason it is not printing out the data between two patterns
This is my entire awk command
for file in `cat out.txt`
do
awk -v ff="$file" 'BEGIN {print "Start Parsing for"ff} /ff-START/{flag=1; next}/ff-END/{flag=0}flag; END{print "End Parsing"ff}' data.txt
done
This is the content of data.txt
JOHN SMITH-START
Device,Number
TV,1
Washing Machine,1
Phones, 5
JOHN SMITH-END
MARY JOE-START
Device,Number
TV,3
Washing Machine,1
Phones, 2
MARY JOE-END
and there are 100 more similar lines here the patterns is NAME-START and NAME-END. So for eg JOHN SMITH-START is the first pattern and then JOHN SMITH-END is the second pattern, and I want to extract the data between these two which is
Device,Number
TV,1
Washing Machine,1
Phones, 5
But the output I get is
Start Parsing forJOHN SMITH
End ParsingJOHN SMITH
Content of out.txt is
JOHN SMITH
MARY JOE

With your shown samples, could you please try following.
awk '/JOHN SMITH-END/ && found{exit} /JOHN SMITH-START/{found=1;next} found' Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
/JOHN SMITH-END/ && found{ ##Checking condition if line contains JOHN SMITH-END and found is SET then do following.
exit ##exiting from program from here.
}
/JOHN SMITH-START/{ ##Checking condition if line contains JOHN SMITH-START then do following.
found=1 ##Setting found to 1 here.
next ##next will skip all further statements from here.
}
found ##If found is set then print that line.
' Input_file ##Mentioning Input_file name here.
NOTE: In case you want to use variables in awk for search then try following:
awk -v start="JOHN SMITH-START" -v end="JOHN SMITH-END" '$0 ~ end && found{exit} $0 ~ start{found=1;next} found' Input_file

Related

multiple condition store in variable and use as if condition in awk

table1.csv:
33622|AAA
33623|AAA
33624|BBB
33625|CCC
33626|DDD
33627|AAA
33628|BBB
33629|EEE
33630|FFF
Aims:
33622|AAA
33623|AAA
33624|BBB
33625|CCC
33626|DDD
33627|AAA
33628|BBB
Using command:
awk 'BEGIN{FS="|";OFS="|"} {if($2=="AAA" && $2=="BBB" && $2=="CCC" && $2=="DDD"){print $1,$2}}' table1.csv
However, trying to be more automatic, since the categories may increase.
list1.csv:
AAA BBB CCC DDD
list=`cat list1.csv`
awk -v list=$list 'BEGIN{FS="|";OFS="|"} {if($2==list){print $1,$2}}' table1.csv
Which means, can I stored $2=="AAA" && $2=="BBB" ....... into a variable by using list1.csv?
Expected output:
33622|AAA
33623|AAA
33624|BBB
33625|CCC
33626|DDD
33627|AAA
33628|BBB
So, any suggestion on storing the multiple condition in one variable?
Thanks all!
$ awk 'NR==FNR{for(i=1;i<=NF;i++)a[$i];next}FNR==1{FS="|";$0=$0}($2 in a)' list table
Output:
33622|AAA
33623|AAA
33624|BBB
33625|CCC
33626|DDD
33627|AAA
33628|BBB
Explained:
$ awk '
NR==FNR { # process list
for(i=1;i<=NF;i++) # hash all items in file
a[$i]
next # possibility for multiple lines
}
FNR==1 { # changing FS in the beginning of table file
FS="|"
$0=$0
}
($2 in a)' list table
Almost same logic Like James Brown's nice answer, just adding here a small variant which is setting field separator in Input_file places itself.
awk 'FNR==NR{for(i=1;i<=NF;i++){arr[$i]};next} ($2 in arr)' list FS="|" table
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
FNR==NR{ ##Checking condition which will be TRUE when list is being read.
for(i=1;i<=NF;i++){ ##Going through all fields here.
arr[$i] ##Creating arr with index of current column value here.
}
next ##next will skip all further statements from here.
}
($2 in arr) ##Checking condition if 2nd field is present in arr then print that line from table file.
' list FS="|" table ##mentioning Input_file(s) here and setting FS as | before table file.

Using awk for vlookup functionality in bash script

Here are the input:
files1.csv
21|AAAAA|1023
21|BBBBB|1203
21|CCCCC|2533
22|DDDDD|1294
22|EEEEE|1249
22|FFFFF|4129
22A|GGGGG|4121
22A|HHHHH|1284
31B|IIIII|5403
31B|JJJJJ|1249
file2.csv
21|A800
22|B900
22A|C1000
31B|D1000
expect output:
files3.csv
21|A800|AAAAA|1023
21|A800|BBBBB|1203
21|A800|CCCCC|2533
22|B900|EEEEE|1249
22|B900|FFFFF|4129
22A|C1000|GGGGG|4121
22A|C1000|HHHHH|1284
31B|D1000|IIIII|5403
31B|D1000|JJJJJ|1249
currently tried using join,
join -a1 -t '|' -1 1 -2 1 -o 1.1,2.2,1.2,1.3 file1.csv file2.csv > file3.csv
But it found that some rows missed matching, so i turn my concept to use most likely vlookup functionality for this two files. Please help.
Thanks all
Could you please try following with awk, written and tested with GNU awk with shown samples.
awk '
BEGIN{
FS=OFS="|"
}
FNR==NR{
arr[$1]=$2
next
}
($1 in arr){
$1=($1 OFS arr[$1])
}
1
' file2.csv file1.csv
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section from here of this program.
FS=OFS="|" ##Setting | as field separator and output field separator.
}
FNR==NR{ ##Checking condition if FNR==NR which will be TRUE when file2.csv is being read.
arr[$1]=$2 ##Creating arr with index of 1st field and value of 2nd field.
next ##next will skip all further statements from here.
}
($1 in arr){ ##checking condition if $1 is present in arr then do following.
$1=($1 OFS arr[$1]) ##Saving current $1 OFS and value of arr with index of $1 in $1.
}
1 ##1 will print the current line.
' file2.csv file1.csv ##Mentioning Input_file names here.
I tested the join command you provided and I think it produces the intended output on my machine (FreeBSD 12.2-RELEASE):
21|A800|AAAAA|1023
21|A800|BBBBB|1203
21|A800|CCCCC|2533
22|B900|DDDDD|1294
22|B900|EEEEE|1249
22|B900|FFFFF|4129
22A|C1000|GGGGG|4121
22A|C1000|HHHHH|1284
31B|D1000|IIIII|5403
31B|D1000|JJJJJ|1249
It's possible that you need to sort both files first on the columns (or in this case where you join the first columns the whole line should also work) that you intend to join, i.e. join -a1 -t '|' -1 1 -2 1 -o 1.1,2.2,1.2,1.3 <(sort file1.csv) <(sort file2.csv) > file3.csv

How to compare two columns of two csv files with awk?

I have two csv files I need to compare against one column.
My member.csv file looks like:
ID|lastName|firstName
01|Lastname01|Firstname01
02|Lastname02|Firstname02
The second file check-ID.csv looks like:
Lastname01|Name01|pubID01|Hash01
Lastname02|Name02|pubID02|Hash02a
Lastname03|Name03|pubID03|Hash03
Lastname02|Name02|pubID02|Hash02b
Lastname01|Name01|pubID01|Hash01b
--> Lastname03 is not in my member.csv !
What I want is to check if the value of the first column of check-ID.csv is equal to value of second column in member.csv.
My attempt with script.awk is
NR==FNR{a[$1]=$1; b[$1]=$0; next}
$2==a[$1]{ delete b[$1]}
END{for (i in b ) print b[i]}
executing with
awk -f script.awk check-ID.csv member.csv
The problem is that the result is not filtered.
I like to get a filtered and sorted output so only members are listed like this:
Lastname01|Name01|pubID01|Hash01
Lastname01|Name01|pubID01|Hash01b
Lastname02|Name02|pubID02|Hash02a
Lastname02|Name02|pubID02|Hash02b
Any help appreciated!
Could you please try following. I think you were close only thing is you could change your Input_files reading sequence. Where I am reading members Input_file first and then check-ID.csv because later Input_file has all details in it which needs to be printed and we need to only check for 2nd field from members Input_file.
awk '
BEGIN{
FS="|"
}
FNR==NR{
a[$2]
next
}
($1 in a)
' members.csv check-ID.csv |
sort -t'|' -k1
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section of this program from here.
FS="|" ##Setting field separator as | here.
}
FNR==NR{ ##Checking condition if FNR==NR which will be TRUE when first Input_file named members.csv is being read.
a[$2] ##Creating array a with index 2nd field here.
next ##next will skip all further statements from here.
}
($1 in a) ##Checking condition if 1st field is preent in a then print that line.
' members.csv check-ID.csv | ##Mentioning Input_file names here and sending its output to sort command.
sort -t'|' -k1 ##Sorting output(which we got from awk command above) by setting separator as | and by first field.

matching columns one separate files and appending matches to file

I'm trying to merge two files filtered on a single column using awk. What I'd then like to do is append the relevant columns from file2 into file 1.
Easier to explain with dummy example.
File1
name fruit animal
bob apple dog
jim orange cat
gary mango snake
daisy peach mouse
File 2:
animal number shape
cat eight square
dog nine circle
mouse eleven sphere
Desired output:
name fruit animal shape
bob apple dog circle
jim orange cat square
gary mango snake NA
daisy peach mouse sphere
Step 1: Need to filter on column 3 in file1 and column 1 in file2
awk -F'\t' 'NR==FNR{c[$3]++;next};c[$1] > 0' file1 file2
This gives me output:
cat eight square
dog nine circle
mouse eleven sphere
This helps me somewhat, however I can't simply cut the third column (shape) from the output above and append it to to file1 since there is no entry for 'snake' in file2. I need to be able to append column 3 of output to file 1 where a match is successful, and where it is not to put 'NA'. It's essential that all the lines in file1 are retained so I can't just omit them. This is where I'm stuck!
I'd appreciate any help please....
E
Could you please try following, written and tested based on shown samples in GNU awk.
awk '
BEGIN{
OFS="\t"
}
FNR==NR{
a[$1]=$NF
next
}
{
print $0,($3 in a?a[$3]:"NA")
}' Input_file2 Input_file1
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section from here.
OFS="\t" ##Setting TAB as output field separator here.
}
FNR==NR{ ##Checking condition FNR==NR which will be TRUE when first Input_file file2 is being read.
a[$1]=$NF ##Creating array a with index $1 and value is $NF for current line.
next ##next will skip all further statements from here.
}
{
print $0,($3 in a?a[$3]:"NA") ##Printing current line and checking if 3rd field is present in array a then print its value OR print NA.
}' file2 file1 ##Mentioning Input_file names here.

bash compare two columns with exact match

I am comparing columns between two files for exact match but I am ending up with inaccurate result. Example as follows.
File1 File2
adam sunny
jhon adam
kelly adam
matt kevin
stuart adam
Gary Gary
When we look at the files there is only match i.e. Garry. My output should be following.
Emptyline
Emptyline
Emptyline
Emptyline
Emptyline
Gary
In order to achieve requirement. I am running the following command
awk 'NR==FNR { n[$1]=$0;next } ($1 in n) { print n[$1],$2 }' file1 file2
and I am getting output as follows
adam
adam
adam
Garry
You should be tracking line numbers, not just line contents:
$ awk 'NR==FNR { lines[NR]=$0; next }
{ if ($0 == lines[FNR]) print; else print "" }' file1.txt file2.txt
Gary
1st solution: With simple awk.
awk 'FNR==NR{a[FNR]=$0;next} a[FNR]==$0{print;next} {print ""}' file1 file2
OR as per anubhava sir's comment:
awk 'FNR==NR{a[FNR]=$0;next} a[FNR]!=$0{$0=""} 1' file1 file2
Explanation: Adding detailed explanation for above code.
awk ' ##Starting awk program from here.
FNR==NR{ ##Checking condition FNR==NR which will be TRUE when first file Input_file1 is being read.
a[FNR]=$0 ##Creating an array a with index FNR and value of current line here.
next ##next will skip all further statements from here.
}
a[FNR]==$0{ ##Checking condition if value of array a with FNR index and current line is equal then do following.
print $0,a[FNR] ##Printing current line and value array a with index FNR here.
}
' file1 file2 ##Mentioning Input_file names here
2nd solution: Considering that your actual Input_file(s) have only 2 columns as per shown samples, could you please try following then.
paste Input_file1 Input_file2 | awk '$1==$2{print $1};$1!=$2{print ""}'
This code will only print lines whose values are equal in Input_file1 and Input_file2.