awk: look for duplicated fields in multiple columns, print new column under condition - awk

Question

I would like your help with awk.
I am trying to look for lines where column $1and $2are duplicated in the file and where at least one of the duplicate has the value refin column $3. If so, print a "1"else print "2" in new column.
An example of input file would be:
a 123 exp_a
a 123 ref
b 146 exp_a
c 156 ref
d 205 exp_a
d 205 exp_b
And the output file would be:
a 123 exp_a 1
a 123 ref 1
b 146 exp_a 2
c 156 ref 2
d 205 exp_a 2
d 205 exp_b 2
Here, a 123 is duplicated with one line having ref at $3so it gets a 1. In contrast, the others are either not duplicated at $1and $2or duplicated but with no ref at $3, so they get a 2.
After some fiddling around, I manage to put a 1at lines where $1and $2are duplicated but it does not take the ref at $3 into account and I cannot tell awk to print a 2 otherwise... SPOILERS: my code is probably very ugly.
awk 'BEGIN {FS=OFS="\t"} {i=$1FS$2} {a[i]=!a[i]?$3:a[i]FS"1\n" i"\t"$3FS"1"} END {for (l in a) {print l,a[l]}}' infile > outfile
The output I get is:
d 205 exp_a 1
d 205 exp_b 1
a 123 exp_a 1
a 123 ref 1
b 146 exp_a
c 156 ref

Answer 1

$ cat tst.awk
BEGIN { OFS="\t" }
NR==FNR {
cnt2[$1,$2]++
cnt3[$1,$2,$3]++
next
}
{ print $0, (cnt2[$1,$2]>1 && cnt3[$1,$2,"ref"]>0 ? 1 : 2) }
$ awk -f tst.awk file file
a 123 exp_a 1
a 123 ref 1
b 146 exp_a 2
c 156 ref 2
d 205 exp_a 2
d 205 exp_b 2

Answer 2

Could you please try following.
awk 'FNR==NR{a[$1,$2]++;b[$1,$2]=$3;next} {$NF=(b[$1,$2]=="ref" && a[$1,$2]>1?$NF OFS "1":$NF OFS "2")} 1' OFS="\t" Input_file Input_file
Adding a non-one liner form of solution too here.
awk '
FNR==NR{
a[$1,$2]++
b[$1,$2]=$3
next
}
{
$NF=(b[$1,$2]=="ref" && a[$1,$2]>1?$NF OFS "1":$NF OFS "2")
}
1
' OFS="\t" Input_file Input_file

Answer 3

This one works in one go of the data but expects the file to be ordered by $1 $2, the "key". Records within each "key" group are outputed in random order (for(i in a)):
awk '
BEGIN { FS=OFS="\t" }
{
if((p!=$1 OFS $2) && NR>1) { # when the $1 $2 changes from previous
for(i=1;i<=a[0];i++) { # iterate and output buffered records
print p,a[i],2-(a[-1]&&a[0]>1) # more than one record in buffer and ...
} # ... ref for $4=1
delete a # empty buffer after output
}
if($3=="ref") # if there is a match in $3
a[-1]++ # increase counter
a[++a[0]]=$3 # buffer records to a, a[0] counter
p=$1 OFS $2 # p is for previous "key"
}
END {
for(i=1;i<=a[0];i++) # duplicate code from above if
print p,a[i],2-(a[-1]&&a[0]>1)
}' file
Outputs:
a 123 exp_a 1
a 123 ref 1
b 146 exp_a 2
c 156 ref 2
d 205 exp_a 2
d 205 exp_b 2
Record counter a[0] and ref counter a[-1] are in a[] to reset them with a single delete a.