Reuse Pig Groups in nested FOREACH statement - apache-pig

Question

I'm trying to group records together, calculate the average of SCORE1, filter out the lower half of the scores, and compute their average of SCORE2. Obviously I can calculate the summary statistics, and rejoin them to the original dataset, but I'd prefer to use the intermediate grouped values.
Example Input
ID,GROUPBY,SCORE1,SCORE2
1,A,58.8,67.3
2,A,85.2,76.3
3,B,49.1,90.7
4,B,78.3,99.8
Pig Script
records = load 'example.csv' Using PigStorage(',') AS (ID,GROUPBY,SCORE1,SCORE2);
grouped = group records by GROUPBY;
avgscore = foreach grouped GENERATE group AS GROUPBY, AVG(records.SCORE1) AS AVGSCORE;
joined = join grouped BY group, avgscore BY GROUPBY USING 'replicated';
results = foreach joined {
scores = foreach records generate SCORE1,SCORE2;
low = FILTER scores by SCORE1 < avgscore.AVGSCORE;
GENERATE GROUPBY, AVG(low.SCORE2);
};
dump results;
Desired Output
A 67.3
B 90.7
However this gives me a result of java.lang.Exception: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Scalar has more than one row in the output. 1st : (A,72.0), 2nd :(B,63.7)

Answer 1

You are essentially grouping two different data structures in line 4.
You are joining grouped (which is grouped) with avgscore (which should be flattened).
You should be doing:
joined = join records BY GROUPBY, avgscore BY GROUPBY USING 'replicated';
edit:
I would rewrite like this to avoid confusion (since there will be two GROUPBYs)
records = load 'example.csv' Using PigStorage(',') AS (ID,GROUPBY,SCORE1,SCORE2);
grouped = group records by GROUPBY;
avgscore = foreach grouped GENERATE group AS GROUPBY, AVG(records.SCORE1) AS AVGSCORE;
joined = join records BY GROUPBY, avgscore BY GROUPBY USING 'replicated';
joined_reduced = foreach joined generate ID, records::GROUPBY as GROUPBY, AVGSCORE, SCORE1, SCORE2;
filter_joined = filter joined_reduced by (SCORE1 > AVGSCORE);
grouped2 = group filter_joined by GROUPBY;
result = foreach grouped2 generate flatten (group), AVG(filter_joined.SCORE2) as low_avg;
dump result;