awk - Rounding to 2 decimal places in subtotals - awk

Short version:
Is there a way to tell awk to round to 2 decimal places during the consolidation, not during the printing?
Long version:
I have an incoming file in the format below. I should get the net balances per currency and if the net is NOT zero, print the result in two columns: net balances less than zero go to neg_bal column and positive balances go to pos_bal column. For some reason, the USD column is still being printed despite netting to zero
JPY||170
JPY||40
USD|-42.61|
USD|-166.27
USD||42.61|
GBP|-20|
EUR||18.7
USD||174.6|
USD|-8.33||
EUR|-30.6|
GBP||100
JPY|-210|
Here is the code am using:
#!/bin/awk -f
BEGIN {
FS="|";
}
{
bal[$1]+=$2+$3
ccy[$1]=$1
}
END {
for (i in ccy)
{
if (bal[i] >0 )
{
pos_bal = bal[i]
neg_bal = 0
}
else
{
neg_bal = bal[i]
pos_bal = 0
}
if (bal[i] != 0 )
{
printf "%s|%.2f|%.2f\n",ccy[i],neg_bal,pos_bal
}
}
}
Result (notice JPY is not displayed since it nets to zero):
awk]$ ./scr1 file1
EUR|-11.90|0.00
USD|0.00|0.00
GBP|0.00|80.00
If I increase the decimal places to say, 20, I see that the USD net amount is not really zero. (Why is this, btw? Even excel gives a net of -1.59872E-14)
awk]$ ./scr1 file1
EUR|-11.90000000000000213163|0.00000000000000000000
USD|0.00000000000000000000|0.00000000000001243450
GBP|0.00000000000000000000|80.00000000000000000000

Is there a way to tell awk to round to 2 decimal places during the
consolidation, not during the printing?
Yes: multiply by 100 and convert to int. Then divide by 100 when you're ready to print.
(In other words, count pennies instead of dollars.)

Related

sum 4th field data between the pattern

suppose my data is :
*dnet *1234 1.2
1 port *12 2.3
3 port1 *34 0.2
7 *15 0.1
*dnet *234 0.2
2 *12 0.1
4 *123 *234 1.2
fields are separated by space.
In this I want to get the sum of 4th fields of data present inside each *dnet. Some fields have 4th field data some has not. I want 4th field sum value for each *dnet seperate.
I tried using awk but could not get. It will be thankful if someone helps.
the output for above will look like
*dnet *1234 1.2 2.5
*dnet *234 0.2 1.2
Commented, slightly simplified, version of the comment...
awk '
# look for header line
$1=="*dnet" {
# print any previously calculated sum
if (header) print header, sum
# reset sum for next block of lines
sum = 0
# save new header line
header = $0
# skip remaining actions
next
}
# if we get here, we know this is not a header line
# if there is a 4th field, add it to the sum
$4 {
sum += $4
}
END {
# print the final sum
if (header) print header, sum
}
' datafile

normalize column data with average value of that column with awk

I have 3 columns in a data file look like below and continues up to 250 rows:
0.9967 0.7765 0.5798
0.9955 0.7742 0.5767
0.9942 0.7769 0.5734
I want to normalise each column based on the average value of that column.
I am using the code below (e.g. for column 1) but it does not print my desired output.
The results should be very close to 1
awk 'NR==FNR{sum+= $1; next}{avg=(NR/sum)}FNR>1{print($1/avg)}' f.dat f.dat
expected output for first column.
1.003
1.001
0.9988
You need separate placeholders for storing the sum and the count of columns. Recommend using an array for storing it for each column.
awk '
NR==FNR {
for (col=1; col<=NF; col++) {
avg[col] += $col
len[col] += 1
}
next
}
{
for (col=1; col<=NF; col++) {
colAvg = avg[col]/len[col]
printf "%.3f%s", $col/colAvg, (col<NF ? FS : ORS)
}
}
' file file
Or if you want to update the entire table with the new normalized values, drop the FNR==1 from the above snippet. If you want to increase the precision of the averaged value, change %.2f to how many digits you want as preferable

Reading fields in previous lines for moving average

Main Question
What is the correct syntax for recursively calling AWK inside of another AWK program, and then saving the output to a (numeric) variable?
I want to call AWK using 2/3 variables:
N -> Can be read from Bash or from container AWK script.
Linenum -> Read from container AWK program
J -> Field that I would like to read
This is my attempt.
Container AWk program:
BEGIN {}
{
...
# Loop in j
...
k=NR
# Call to other instance of AWK
var=(awk -f -v n="$n_steps" linenum=k input-file 'linenum-n {printf "%5.4E", $j}'
...
}
END{}
Background for more general questions:
I have a file for which I would like to calculate a moving average of n (for example 2280) steps.
Ideally, for the first n rows the average is of the values 1 to k,
where k <= n.
For rows k > n the average would be of the last n values.
I will eventually execute the code in many large files, with several columns, and thousands to millions of rows, so I'm interested in streamlining the code as much as possible.
Code Excerpt and Description
The code I'm trying to develop looks something like this:
NR>1
{
# Loop over fields
for (j in columns)
{
# Rows before full moving average is done
if ( $1 <= n )
{
cumsum[j]=cumsum[j]+$j #Cumulative sum
$j=cumsum[j]/$1 # Average
}
#moving average
if ( $1 > n )
{
k=NR
last[j]=(awk -f -v n="$n_steps" ln=k input-file 'ln-n {printf "%5.4E", $j}') # Obtain value that will get ubstracted from moving average
cumsum[j]=cumsum[j]+$j-last[j] # Cumulative sum adds last step and deleted unwanted value
$j=cumsum[j]/n # Moving average
}
}
}
My input file contains several columns. The first column contains the row number, and the other columns contain values.
For the cumulative sum of the moving average: If I am in row k, I want to add it to the cumulative sum, but also start subtracting the first value that I don't need (k-n).
I don't want to have to create an array of cumulative sums for the last steps, because I feel it could impact performance. I prefer to directly select the values that I want to substract.
For that I need to call AWK once again (but on a different line). I attempt to do it in this line:
k=NR
last[j]=(awk -f -v n="$n_steps" ln=k input-file 'ln-n {printf "%5.4E", $j}'
I am sure that this code cannot be correct.
Discussion Questions
What is the best way to obtain information about a field in a previous line to the one that AWK is working on? Can it be then saved into a variable?
Is this recursive use of AWK allowed or even recommended?
If not, what could be the most efficient way to update the cumulative sum values so that I get an efficient enough code?
Sample input and Output
Here is a sample of the input (second column) and the desired output (third column). I'm using 3 as the number of averaging steps (n)
N VAL AVG_VAL
1 1 1
2 2 1.5
3 3 2
4 4 3
5 5 4
6 6 5
7 7 6
8 8 7
9 9 8
10 10 9
11 11 10
12 12 11
13 13 12
14 14 13
14 15 14
If you want to do a running average of a single column, you can do it this way:
BEGIN{n=2280; c=7}
{ s += $c - a[NR%n]; a[NR%n] = $c }
{ print $0, s /(NR < n : NR ? n) }
Here we store the last n values in an array a and keep track of the cumulative sum s. Every time we update the sum we correct by first removing the last value from it.
If you want to do this for a couple of columns, you have to be a bit handy with keeping track of your arrays
BEGIN{n=2280; c[0]=7; c[1]=8; c[2]=9}
{ for(i in c) { s[i] += $c[i] - a[n*i + NR%n]; a[n*i + NR%n] = $c[i] } }
{ printf $0
for(i=0;i<length(c);++i) printf OFS (s[i]/(NR < n : NR ? n))
printf ORS
}
However, you mentioned that you have to add millions of entries. That is where it becomes a bit more tricky. Summing a lot of values will introduce numeric errors as you loose precision bit by bit (when you add floats). So in this case, I would suggest implementing the Kahan summation.
For a single column you get:
BEGIN{n=2280; c=7}
{ y = $c - a[NR%n] - k; t = s + y; k = (t - s) - y; s = t; a[NR%n] = $c }
{ print $0, s /(NR < n : NR ? n) }
or a bit more expanded as:
BEGIN{n=2280; c=7}
{ y = $c - k; t = s + y; k = (t - s) - y; s = t; }
{ y = -a[NR%n] - k; t = s + y; k = (t - s) - y; s = t; }
{ a[NR%n] = $c }
{ print $0, s /(NR < n : NR ? n) }
For a multi-column problem, it is now straightforward to adjust the above script. All you need to know is that y and t are temporary values and k is the compensation term which needs to be stored in memory.

Explain why storing the value of printf in a variable and then printing it gives an extra value?

int d;
d=printf("\n%d%d%d%d",1,2,3,4);
printf("%d",d);
The code gives the output as 1,2,3,4,5.
I don't understand why an integer greater than the last one is being printed.
printf returns the total number of characters written. In the first printf call that is 4 digits from the 4 variables and the newline character which adds up to 5. So the return value is 5 which is what you get in the second call.

VB.Net About dividing Currency by an X amount of months

im trying to learn how to made stuff with currency.
For example:
I divide 10.000$ by 12 Months, rounding with 2 decimals i have 833,33 $.
If i multiply 833,33 $ * 12 i got 9999,96 $, so there is 0.04 of possible loss.
Rounding the 9999.96 with 2 decimals of presition i got 10.000 $ but that's what i don't want since 0.04 is a loss.
Im using SQL Compact 4.0 as database, the price_month table is decimal(18,2)
Here is my code:
Dim price as Decimal = 10000
Dim pricemonth as Decimal = Math.round((price/12),2) ' 833.33
Console.Writeline(pricemonth*12) ' 9999.96
Console.Writeline(Math.round((pricemonth*12),2)) ' 10000
Any advice how to increase accuracy with currency? Thanks and have a nice day!
Don't round your calculation. Leave the original numbers untouched but when you display the answer round it so that it looks nice.