Calculate sequential average and median from file using awk - awk

Question

This is my input file (there are thousands of rows):
$ cat file.txt
1 495.03
2 503.76
3 512.28
4 520.75
5 529.17
I'd like to use awk to calculate the median of the first column over X (let's say 1-100) number of rows and the average of the corresponding values of the second column. awk would then move the the next set of rows (101-201) and do the same, i.e. median of the first column and average of the second column and so on. Needless to say, I'm trying to learn awk and have tried several previous solutions but couldn't quite make it work.
From a previous post, I found that I can calculate the average this way:
awk '{sum+=$1} NR%3==0 {print sum/3; sum=0}'
How does this work exactly (i.e. what does this {sum+=$1} expression mean?) and how can I adapt this for median? Btw, the first column will always be sorted.
Thanks in advance,
TP

Answer 1

If the records are sorted, the median will be just the average of 50th and 51st values.
$ awk '{r=NR%100; sum+=$2}
r==50 {m=$1}
r==51 {m=(m+$1)/2}
r==0 {print m, sum/100; sum=0}' file
this will work if number of records is a multiple of 100, otherwise you need to handle the last group which will have a different size.
There are other definitions for "median" for even number of records but that's something you should specify.
Explanation define r to be the remainder by mod 100, in essence the relative position in each block of 100 records. For the median we take the average of 50th and 51st records, sum aggregates the second field value for each 100 block. When the remainder is 0, we complete each block, print median and average (sum/100) values; reset sum for the next block.

Answer 2

note: This contains a bit more information wrt running means and medians for unsorted data. This should be seen as an addendum to the original question.
If you want to compute the running average over the last n terms (assume n = 100) then you have to take care of how you handle the first m records with m < n. A way to handle this is to place the values in an array where the index is the modulo of n. This way you always have the last n terms in your array :
running average of $i:
awk '{a[NR%100] = $i; s=0; for(j in a) { s+=a[j] }; print "avg:" s/length(a) }'
You can, however, remove the for-loop by keeping track of s:
awk '{s+=$i; if (NR%100 in a) s-=a[NR%100]; a[NR%100]=$i; print "avg:" s/length(a) }'
running median of $i:
A way to compute the median can be done with gawk in which we assume that the array is sorted for array-traversal by value
awk 'BEGIN{ PROCINFO["sorted_in"]="#val_num_asc" }
{ a[NR%100] = $i }
{ k=0; m=0;
for(j in a) { k++
if (k >= length(a)/2 ) m+=a[j]
if (k <= length(a)/2+1) {m+=a[j]; break }
}
print "med:", m/2
}'
or if you want it a bit lighter on the if-conditions
awk 'BEGIN{ PROCINFO["sorted_in"]="#val_num_asc" }
{ a[NR%100] = $i }
{ k=0; m=0;
for(j in a) { k++
if (k < length(a)/2 ) continue
if (k > length(a)/2+1) break
m+=a[j]
}
print "med:", (length(a)%2==0 ? m/2 : m)
}'
If you don't want to use the pre-sorted concept, then the computation of the median becomes much more difficult. A possible way would be making use of selection algorithm as explained here.