Here are the input:
files1.csv
21|AAAAA|1023
21|BBBBB|1203
21|CCCCC|2533
22|DDDDD|1294
22|EEEEE|1249
22|FFFFF|4129
22A|GGGGG|4121
22A|HHHHH|1284
31B|IIIII|5403
31B|JJJJJ|1249
file2.csv
21|A800
22|B900
22A|C1000
31B|D1000
expect output:
files3.csv
21|A800|AAAAA|1023
21|A800|BBBBB|1203
21|A800|CCCCC|2533
22|B900|EEEEE|1249
22|B900|FFFFF|4129
22A|C1000|GGGGG|4121
22A|C1000|HHHHH|1284
31B|D1000|IIIII|5403
31B|D1000|JJJJJ|1249
currently tried using join,
join -a1 -t '|' -1 1 -2 1 -o 1.1,2.2,1.2,1.3 file1.csv file2.csv > file3.csv
But it found that some rows missed matching, so i turn my concept to use most likely vlookup functionality for this two files. Please help.
Thanks all
Could you please try following with awk, written and tested with GNU awk with shown samples.
awk '
BEGIN{
FS=OFS="|"
}
FNR==NR{
arr[$1]=$2
next
}
($1 in arr){
$1=($1 OFS arr[$1])
}
1
' file2.csv file1.csv
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section from here of this program.
FS=OFS="|" ##Setting | as field separator and output field separator.
}
FNR==NR{ ##Checking condition if FNR==NR which will be TRUE when file2.csv is being read.
arr[$1]=$2 ##Creating arr with index of 1st field and value of 2nd field.
next ##next will skip all further statements from here.
}
($1 in arr){ ##checking condition if $1 is present in arr then do following.
$1=($1 OFS arr[$1]) ##Saving current $1 OFS and value of arr with index of $1 in $1.
}
1 ##1 will print the current line.
' file2.csv file1.csv ##Mentioning Input_file names here.
I tested the join command you provided and I think it produces the intended output on my machine (FreeBSD 12.2-RELEASE):
21|A800|AAAAA|1023
21|A800|BBBBB|1203
21|A800|CCCCC|2533
22|B900|DDDDD|1294
22|B900|EEEEE|1249
22|B900|FFFFF|4129
22A|C1000|GGGGG|4121
22A|C1000|HHHHH|1284
31B|D1000|IIIII|5403
31B|D1000|JJJJJ|1249
It's possible that you need to sort both files first on the columns (or in this case where you join the first columns the whole line should also work) that you intend to join, i.e. join -a1 -t '|' -1 1 -2 1 -o 1.1,2.2,1.2,1.3 <(sort file1.csv) <(sort file2.csv) > file3.csv
Related
table1.csv:
33622|AAA
33623|AAA
33624|BBB
33625|CCC
33626|DDD
33627|AAA
33628|BBB
33629|EEE
33630|FFF
Aims:
33622|AAA
33623|AAA
33624|BBB
33625|CCC
33626|DDD
33627|AAA
33628|BBB
Using command:
awk 'BEGIN{FS="|";OFS="|"} {if($2=="AAA" && $2=="BBB" && $2=="CCC" && $2=="DDD"){print $1,$2}}' table1.csv
However, trying to be more automatic, since the categories may increase.
list1.csv:
AAA BBB CCC DDD
list=`cat list1.csv`
awk -v list=$list 'BEGIN{FS="|";OFS="|"} {if($2==list){print $1,$2}}' table1.csv
Which means, can I stored $2=="AAA" && $2=="BBB" ....... into a variable by using list1.csv?
Expected output:
33622|AAA
33623|AAA
33624|BBB
33625|CCC
33626|DDD
33627|AAA
33628|BBB
So, any suggestion on storing the multiple condition in one variable?
Thanks all!
$ awk 'NR==FNR{for(i=1;i<=NF;i++)a[$i];next}FNR==1{FS="|";$0=$0}($2 in a)' list table
Output:
33622|AAA
33623|AAA
33624|BBB
33625|CCC
33626|DDD
33627|AAA
33628|BBB
Explained:
$ awk '
NR==FNR { # process list
for(i=1;i<=NF;i++) # hash all items in file
a[$i]
next # possibility for multiple lines
}
FNR==1 { # changing FS in the beginning of table file
FS="|"
$0=$0
}
($2 in a)' list table
Almost same logic Like James Brown's nice answer, just adding here a small variant which is setting field separator in Input_file places itself.
awk 'FNR==NR{for(i=1;i<=NF;i++){arr[$i]};next} ($2 in arr)' list FS="|" table
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
FNR==NR{ ##Checking condition which will be TRUE when list is being read.
for(i=1;i<=NF;i++){ ##Going through all fields here.
arr[$i] ##Creating arr with index of current column value here.
}
next ##next will skip all further statements from here.
}
($2 in arr) ##Checking condition if 2nd field is present in arr then print that line from table file.
' list FS="|" table ##mentioning Input_file(s) here and setting FS as | before table file.
I have two csv files I need to compare against one column.
My member.csv file looks like:
ID|lastName|firstName
01|Lastname01|Firstname01
02|Lastname02|Firstname02
The second file check-ID.csv looks like:
Lastname01|Name01|pubID01|Hash01
Lastname02|Name02|pubID02|Hash02a
Lastname03|Name03|pubID03|Hash03
Lastname02|Name02|pubID02|Hash02b
Lastname01|Name01|pubID01|Hash01b
--> Lastname03 is not in my member.csv !
What I want is to check if the value of the first column of check-ID.csv is equal to value of second column in member.csv.
My attempt with script.awk is
NR==FNR{a[$1]=$1; b[$1]=$0; next}
$2==a[$1]{ delete b[$1]}
END{for (i in b ) print b[i]}
executing with
awk -f script.awk check-ID.csv member.csv
The problem is that the result is not filtered.
I like to get a filtered and sorted output so only members are listed like this:
Lastname01|Name01|pubID01|Hash01
Lastname01|Name01|pubID01|Hash01b
Lastname02|Name02|pubID02|Hash02a
Lastname02|Name02|pubID02|Hash02b
Any help appreciated!
Could you please try following. I think you were close only thing is you could change your Input_files reading sequence. Where I am reading members Input_file first and then check-ID.csv because later Input_file has all details in it which needs to be printed and we need to only check for 2nd field from members Input_file.
awk '
BEGIN{
FS="|"
}
FNR==NR{
a[$2]
next
}
($1 in a)
' members.csv check-ID.csv |
sort -t'|' -k1
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section of this program from here.
FS="|" ##Setting field separator as | here.
}
FNR==NR{ ##Checking condition if FNR==NR which will be TRUE when first Input_file named members.csv is being read.
a[$2] ##Creating array a with index 2nd field here.
next ##next will skip all further statements from here.
}
($1 in a) ##Checking condition if 1st field is preent in a then print that line.
' members.csv check-ID.csv | ##Mentioning Input_file names here and sending its output to sort command.
sort -t'|' -k1 ##Sorting output(which we got from awk command above) by setting separator as | and by first field.
I have two CSV files in the form of
file1
A,44
A,21
B,65
C,79
file2
A,7
B,4
C,11
I used awk as
awk -F, 'NR==FNR{a[$1]=$0;next} ($1 in a){print a[$1]","$2 }' file1.csv file2.csv
producing
A,44,7
A,21,7
B,65,4
C,79,11
a[$1] prints the entire line from file1. How can I omit the first columns in both files (the first column is only used to match the second columns) to produce:
44,7
21,7
65,4
79,11
In other words, how can I pass the columns from the first file to the print block, as $2 does for the second file?
Could you please try following, tested and written on shown samples only.
awk 'BEGIN{FS=OFS=","} FNR==NR{a[$1]=$2;next} ($1 in a){print $2,a[$1]}' file2 file1
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section from here.
FS=OFS="," ##Setting field and output field separator as comma here.
}
FNR==NR{ ##Checking condition FNR==NR which will be TRUE when file2 is being read.
a[$1]=$2 ##Creating array a with index $1 and value is $2 from current line.
next ##next will skip all further statement from here.
}
($1 in a){ ##Statements from here will be executed when file1 is being read and it's checking if $1 is present in array a then do following.
print $2,a[$1] ##Printing 2nd field and value of array a with index $1 here.
}
' file2 file1 ##Mentioning Input_file names here.
Output will be as follows for shown samples.
44,7
21,7
65,4
79,11
2nd solution: More Generic solution, where considering that your both Input_files could have duplicates in that case it will print 1st value of A in Input_file1 to first value of Input_file2 and so on.
awk '
BEGIN{
FS=OFS=","
}
FNR==NR{
a[$1]
b[$1,++c[$1]]=$2
next
}
($1 in a){
print $2,b[$1,++d[$1]]
}
' file2 file1
You can join them using the join command and chose which fields you want to have in the output:
kent$ join -t',' -o 1.2,2.2 file1 file2
44,7
21,7
65,4
79,11
My input is:
group;|line1_1|line2_2|line3_3
I want to extract the inforamtion line1,line2,line3 as below output:
line1,line2,line3
I have tried by using the following command but not executable:
LINE="group;|line1_1|line2_2|line3_3"; echo $LINE | awk -F ";" '{print $2}' | awk -F "|" '{for(i=1;i<=NF;++i){print $i | system("cut -d _ -f1")}}'
Considering that your actual Input_file is same as shown samples. Could you please try following.
awk -F'|' '{for(i=2;i<=NF;i++){sub(/_.*/,"",$i);val=val?val OFS $i:$i};print val;val=""}' OFS="," Input_file
2nd Solution: Only using sub and gsub of awk here.
awk '{sub(/^[^|]*\|/,"");gsub(/_[0-9]\|/,",");sub(/_[0-9]$/,"")} 1' Input_file
OR
awk '{gsub(/^[^|]*\||_[0-9]$/,"");gsub(/_[0-9]\|/,",")} 1' Input_file
Output will be as follows.
line1,line2,line3
Explanation: Adding explanation for above code now.
awk -F'|' ' ##Setting field separator as | here for all lines.
{ ##Starting block here.
for(i=2;i<=NF;i++){ ##Starting for loop starting form i=2 to till value of NF here.
sub(/_.*/,"",$i) ##Using sub for substitution from _ to till everything with NULL in value of $i.
val=val?val OFS $i:$i ##Creating variable val and concatenate its own value if it is having NON ZERO value else save in it for 1st time.
} ##Close for loop block here.
print val ##Printing value of val here.
val="" ##Nullifying value of variable val here.
}' OFS="," Input_file ##Setting value of OFS to comma here and mentioning Input_file name here.
Need to vlookup from two different files having multiple entries:
cat file1.csv
aaaaaaa;24/09/2018;06/09/2018;1;89876768
bbbbbbb;15/09/2018;03/09/2018;2;76958489
ccccccc;10/09/2018;28/08/2018;3;57848472
ddddddd;22/09/2018;08/09/2018;4;17929730
eeeeeee;19/09/2018;30/08/2018;5;18393770
cat file2.csv
20180901;abc;1
20180901;sdf;2
20180904;jhh;2
20180905;skf;3
20180911;asf;2
20180923;ghf;4
20180925;asb;4
20180918;mnj;3
In addition for file1.csv, the fourth column is the identifier of the third colunm into file2.csv.
Output required is:
aaaaaaa;24/09/2018;06/09/18;1;89876768;20180901
bbbbbbb;15/09/2018;03/09/18;2;76958489;20180901;20180904;20180911
ccccccc;10/09/2018;28/08/18;3;57848472;20180905;20180918
ddddddd;22/09/2018;08/09/18;4;17929730;20180923;20180925
eeeeeee;19/09/2018;30/08/18;5;18393770;unknown
Could you please try following.
awk 'BEGIN{FS=OFS=";"}FNR==NR{a[$NF]=a[$NF]?a[$NF] OFS $1:$1;next} {print ($4 in a)?$0 OFS a[$4]:$0 OFS "unknown"}' file2.csv file1.csv
Output will be as follows.
aaaaaaa;24/09/2018;06/09/2018;1;89876768;20180901
bbbbbbb;15/09/2018;03/09/2018;2;76958489;20180901;20180904;20180911
ccccccc;10/09/2018;28/08/2018;3;57848472;20180905;20180918
ddddddd;22/09/2018;08/09/2018;4;17929730;20180923;20180925
eeeeeee;19/09/2018;30/08/2018;5;18393770;unknown
Explanation of code:
awk '
BEGIN{ ##Starting BEGIN section for awk here.
FS=OFS=";" ##Setting values for FS and OFS as semi colon.
} ##Closing block for BEGIN section here.
FNR==NR{ ##Checking condition FNR==NR which will be TRUE when first Input_file named file2.csv is being read.
a[$NF]=a[$NF]?a[$NF] OFS $1:$1 ##Creating an array named a whose index is $NF and value is $1 and concatenating its own value with same index.
next ##next will skip all further statements from here.
} ##Closing block for FNR==NR condition here.
{
print ($4 in a)?$0 OFS a[$4]:$0 OFS "unknown" ##These statements will execute when 2nd Input_file is being read and printing value of $0 with condition if $4 is present in array a then concatenate its value with current line else concatenate unknown with it.
}' file2.csv file1.csv ##Mentioning Input_file names here.