Pig Grouping Functions - apache-pig

Question

I would like to get ,what item was bought very recently by each person. Assume that a same person can buy many items.
below are the input details
kumar,2014-09-30,television
kumar,2014-07-27,smartphone
Andrew,2014-06-21,camera
Andrew,2014-05-20,car
I need the output as below
kumar,2014-09-30,television
Andrew,2014-06-21,camera
I wrote a Pig script upto this, but after that i dont know how to proceed,can somebody help me
A = LOAD 'records.txt' USING PigStorage(',') AS(name:chararray,date:chararray,item:chararray);
B = GROUP A BY name;
C = FOREACH B GENERATE group,MAX(A.date);
But i need to get the item that was purchased recently by each person. How do i get that. If i apply GROUP then i am supposed to use only aggregate function in Pig.
How do i get the recepective item that was purchased?

Answer 1

Use bags and order by in a nested foreach, it will use only 1 MR job and is more in Apache Pig style.
A = LOAD 'input.txt' USING PigStorage(',') AS(name:chararray,date:chararray,item:chararray);
B = GROUP A BY name;
C = FOREACH B {
ordered = ORDER A BY date DESC; -- this will cause secondary sort to optimise the execution
latest = LIMIT ordered 1;
GENERATE FLATTEN(latest); - advantage of PIG, that all columns are preserved and not dropped as on SQL group by
};
DUMP C;
Also use of $0, $1 etc is convenient, but imagine you have a script with hundreds of lines and tens of group by and join operations that project using '$', it is nightmare to understand the flow of information/columns though such scripts. Time wasted in maintenance and making changes to such scripts is huge.

Answer 2

I hope this works for you.
input.txt
kumar,2014-09-30,television
kumar,2014-07-27,smartphone
Andrew,2014-06-21,camera
Andrew,2014-05-20,car
PigScript:
A = LOAD 'input.txt' USING PigStorage(',') AS(name:chararray,date:chararray,item:chararray);
B = GROUP A BY name;
C = FOREACH B GENERATE group,FLATTEN(MAX($1.date));
D = JOIN A BY date,C BY $1;
E = FOREACH D GENERATE $0,$1,$2;
DUMP E;
Output:
(Andrew,2014-06-21,camera)
(kumar,2014-09-30,television)