Cacluate the sum/avg of an attribute in Apache pig - apache-pig

Question

How can I calculate the avg or sum of an attribute in Apache pig (vertically not horizontally). Lots of example are available for doing this horizontally but not vertically.
This is my code
f1 = LOAD '/user/maria_dev/flightdelays/flight_delays1.csv' USING
PigStorage(',') AS (Year:int, ArrDelay:chararray);
f123 = f1;
ff123 = FILTER f123 BY something;
grp = GROUP ff123 ALL;
cnt = FOREACH grp GENERATE COUNT(ff123);-- this counts the number of rows and works fine
DUMP cnt;
-- The below code is the problem
DESCRIBE grp;
cntsum = FOREACH grp GENERATE FLATTEN(ff123.ArrDelay);
DESCRIBE cntsum;
and the output:
(2008,30)
(2009,60)
(2)
grp: {group: chararray,ff123: {(Year: int,ArrDelay: chararray)}}
cntsum: {null::ArrDelay: chararray}
But this throw me an error:
cntsum = FOREACH grp GENERATE SUM((int)FLATTEN(ff123.ArrDelay));
DESCRIBE cntsum;
I need to get 90 as output (30+60)
By the way, what is this schema as the output:
cntsum: {null::ArrDelay: chararray}
using pig Apache Pig version 0.16.0.2.6.5.0-292

Answer 1

You should load ArrDelay as int column
f1 = LOAD '/user/maria_dev/flightdelays/flight_delays1.csv' USING PigStorage(',') AS (Year:int, ArrDelay:int);
ff123 = FILTER f1 BY something;
grp = GROUP ff123 ALL;
total = FOREACH grp GENERATE SUM(ff123.ArrDelay);
DUMP total;
If that is not an option then load ArrDelay into chararray and then cast it before the group all and sum
f1 = LOAD '/user/maria_dev/flightdelays/flight_delays1.csv' USING PigStorage(',') AS (Year:int, ArrDelay:chararray);
ff123 = FILTER f1 BY something;
f2 = FOREACH ff123 GENERATE Year, (int)ArrDelay as ArrDelay;
grp = GROUP f2 ALL;
total = FOREACH grp GENERATE SUM(f2.ArrDelay);
DUMP total;