Filtering file according to the highest value in a column of each line - awk

Question

I have the following file:
gene.100079.0.5.p3 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 84.9
gene.100079.0.3.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 84.9
gene.100079.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 86.7
gene.100080.0.3.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
gene.100080.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
chr11_pilon3.g3568.t1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 74.9
chr11_pilon3.g3568.t2 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 76.7
The above file has some IDs which are similar
gene.100079.0.5.p3
gene.100079.0.3.p1
gene.100079.0.0.p1
By remaining only gene.100079 the IDs become identically. I would like to filter the above file in the following way:
chr11_pilon3.g3568.t1 = 74.9. IDs starting with chr get excluded from the comparison and they end up straight in the output.
gene.100079.0.0.p1 = 86.7 && gene.100079.0.5.p3 = 84.9 == gene.100079.0.3.p1 = 84.9. gene.100079.0.0.p1 has the highest value and therefore it should be in the output.
gene.100080.0.3.p1 = 99.9 == gene.100080.0.0.p1 = 99.9. Both IDs have the same value and therefore both should be in the output.
However, this awk script from #RavinderSingh13 and #anubhava returns the wrong results.
awk '{
if (/^gene\./) {
split($1, a, /\./)
k = a[1] "." a[2]
}
else
k = $1
}
!(k in max) || $13 >= max[k] {
if(!(k in max))
ord[++n] = k
else if (max[k] == $13) {
print
next
}
max[k] = $13
rec[k] = $0
}
END {
for (i=1; i<=n; ++i)
print rec[ord[i]]
}' file
Wrong output with the above script:
chr11_pilon3.g3568.t1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 74.9
chr11_pilon3.g3568.t2 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 76.7
gene.100079.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 86.7
gene.100079.0.3.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 84.9
gene.100080.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
gene.100080.0.3.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
As output I would like to get:
chr11_pilon3.g3568.t1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 74.9
chr11_pilon3.g3568.t2 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 76.7
gene.100079.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 86.7
gene.100080.0.3.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
gene.100080.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
I also tried to fix as show below but it didn't work:
awk '{
if (/^gene\./) {
split($1, a, /\./)
k = a[1] "." a[2]
}
else
k = $1
}
!(k in max) || $13 > max[k] {
max[k]=$13;
line[k]=$0
}
END {
for(i in line)
print line[i]
}'
Thank you in advance,

Answer 1

This seems to work correctly, assuming that the data is ordered so that all the lines with the same first two name components are grouped together in the data file. The order of those lines within the group doesn't matter.
As revised, the question now wants lines starting chr transferred to the output without any filtering. That is easily achieved — the rule matching /^chr/ provides that functionality.
#!/bin/sh
awk '
function dump_memo()
{
if (memo_num > 0)
{
for (i = 0; i < memo_num; i++)
print memo_line[i]
}
}
/^chr/ { print; next } # Do not process lines starting chr specially
{
split($1, a, ".")
key = a[1] "." a[2]
val = $NF
# print "# " key " = " val " (memo_key = " memo_key ", memo_val = " memo_val ")"
if (memo_key == key)
{
if (memo_val == val)
{
memo_line[memo_num++] = $0
}
else if (memo_val < val)
{
memo_val = val
memo_num = 0
memo_line[memo_num++] = $0
}
}
else
{
dump_memo()
memo_num = 0
memo_line[memo_num++] = $0
memo_key = key
memo_val = val
}
}
END { dump_memo() }' "$#"
When run on the data file shown in the question, the original output from the unrevised script was:
gene.100079.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 86.7
gene.100080.0.3.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
gene.100080.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
chr11_pilon3.g3568.t2 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 76.7
The main difference between this and what was requested is the sort order. If you need the data in sorted order, pipe the output of the script through sort.
With the revised script (with the /^chr/ rule) is:
gene.100079.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 86.7
chr11_pilon3.g3568.t1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 74.9
chr11_pilon3.g3568.t2 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 76.7
gene.100080.0.3.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
gene.100080.0.0.p1 transcript:OIS96097 82.2 169 30 0 1 169 4 172 1.3e-75 283.1 99.9
Again, if you want the data in some specific order, apply a sort to the output.