How to print columns that have values satisfying a range? - awk

Question

I have a big table with thousands of columns and rows. But for the sake of simplification, let say that I have a table consisting 11 rows and 100 columns. The table cells contain values between 0 and 1. The table looks like below:
Sample1 Sample2 Sample3 Sample4
1 0 0.001 0.002
0.74 0.52 0.654 0.75
0.65 0.64 0.455 0.72
0.24 0.51 0.512 0.78
0.25 0.555 0.557 0.25
0.003 0.454 0.532 0.23
0.02 0.56 0.643 0.22
1 0.495 0.555 0.99
0.992 1 0.999 0.98
0.12 0 0.968 1
Now I would like to scan all the cells for all values that are within a specific range of 0.80 >= value >= 0.70. Any cells that contain values as such, the whole column will be printed, including the headers.
The intended output would be as below:
Sample1 Sample4
1 0.002
0.74 0.75
0.65 0.72
0.24 0.78
0.25 0.25
0.003 0.23
0.02 0.22
1 0.99
0.992 0.98
0.12 1
Commands using awk would be preferable, but I do not know whether it is the best for this kind of extraction.
Please kindly advise me on how to do this. Any help will be very much appreciated. Thank you.

Answer 1

awk to the rescue!
$ awk 'NR==FNR && NR>1{for(i=1; i<=NF; i++)
if(0.7<=$i && $i<=0.8) col[i]=1; next}
{for(i=1 ;i<=NF; i++)
if(col[i]) printf "%s", $i OFS; print ""}' file{,} | column -t
Sample1 Sample4
1 0.002
0.74 0.75
0.65 0.72
0.24 0.78
0.25 0.25
0.003 0.23
0.02 0.22
1 0.99
0.992 0.98
0.12 1
a double scan algorithm, marks the filtered columns in the first round and prints them in the second round.

Answer 2

Not sure whether this will work with the size of table you're dealing with, but any solution is likely going to have to store something in an array, or be multi-pass.
My first thought was that given rotate.awk like this:
{
for (i=1; i<=NF; i++) {
d[i,NR]=$i
}
}
END {
for (i=1; i<=NF; i++) {
tab=""
for (j=1; j<=NR; j++) {
printf "%s%s", tab, d[i,j]
tab="\t"
}
printf "\n"
}
}
You could analyse your results between two rotations:
$ awk -f rotate.awk file.tsv | awk -v n=0.7 -v m=0.8 '{x=0; for (i=2; i<=NF; i++) if ($i >= n && $i <= m) x=1} x' | awk -f rotate.awk
Sample1 Sample4
1 0.002
0.74 0.75
0.65 0.72
0.24 0.78
0.25 0.25
0.003 0.23
0.02 0.22
1 0.99
0.992 0.98
0.12 1
Again, you may be limited by the amount of memory your system can allocate to awk to contain the array required for rotation.
An alternative which doesn't use gobs of memory to store an array would be a multi-pass approach that identifies column numbers, then uses them as input to a print script:
$ awk -v n=0.7 -v m=0.8 '{for (i=1; i<=NF; i++) if ($i >= n && $i <= m) print i}' file.tsv |
awk 'NR==FNR{c[$1];next} {tab="";for (i=1; i<=NF; i++) if (i in c) {printf "%s%s",tab,$i;tab="\t"} printf "\n" }' - file.tsv
The idea here is that the FIRST awk script selects the columns to be printed, and prints just those column numbers. The SECOND awk script takes two inputs; first, it reads the list of column numbers from stdin (-) and populates an array with them. Then it steps through the input file, printing columns whose numbers are within the array.

Answer 3

$ cat tst.awk
BEGIN { FS=OFS="\t" }
NR==FNR {
if (FNR > 1) {
for (i=1; i<=NF; i++) {
if ( ($i >= 0.7) && ($i <= 0.8) ) {
good[i]
}
}
}
next
}
{
c=0
for (i=1; i<=NF; i++) {
if (i in good) {
printf "%s%s", (c++ ? OFS : ""), $i
}
}
print ""
}
$ awk -f tst.awk file file
Sample1 Sample4
1 0.002
0.74 0.75
0.65 0.72
0.24 0.78
0.25 0.25
0.003 0.23
0.02 0.22
1 0.99
0.992 0.98
0.12 1