I have a data set that looks like this:
foo,R
foo,Y
bar,C
foo,R
baz,Y
foo,R
baz,Y
baz,R
...
I'd like to generate a report that sums up the number of 'R', 'Y' and 'C' records for each unique value in the first column. For this data set, it would look like:
foo,3,1,0
bar,0,0,1
baz,1,2,0
Where the 2nd column is the number of 'R' records, the third is the number of 'Y' records and the last is the number of 'C' records.
I know I can first filter by record type, group and aggregate, but that leads to an expensive join of the three sub-reports. I would much rather group once and GENERATE each of the {R, Y, C} columns in my group.
How can I convert the Boolean result of comparing the second column in my data set to 'R', 'Y' or 'C' to a numeric value I can aggregate? Ideally I want 1 for a match and 0 for a non-match for each of the three columns.
Apache PIG is perfectly adapted for such type of problems. It can be solved with one GROUP BY and one nested FOREACH
inpt = load '~/pig/data/group_pivot.csv' using PigStorage(',') as (val : chararray, cat : chararray);
grp = group inpt by (val);
final = foreach grp {
rBag = filter inpt by cat == 'R';
yBag = filter inpt by cat == 'Y';
cBag = filter inpt by cat == 'C';
generate flatten(group) as val, SIZE(rBag) as R, SIZE(yBag) as Y, SIZE(cBag) as C;
};
dump final;
--(bar,0,0,1)
--(baz,1,2,0)
--(foo,3,1,0)
bool = foreach final generate val, (R == 0 ? 0 : 1) as R, (Y == 0 ? 0 : 1) as Y, (C == 0 ? 0 : 1) as C;
dump bool;
--(bar,0,0,1)
--(baz,1,1,0)
--(foo,1,1,0)
I have tried it on your example and got the expected result. The idea is that after GROUP BY each value has a BAG that contains all rows with R, Y, C categories. Using FILTER within FOREACH we create 3 separate BAGs (one per R, Y, C) and SIZE(bag) in GENERATE counts the number of rows in each bag.
The only problem you might encounter is when there are too many rows with the same value in val column, as nested FOREACH relies on in memory operations and resulting intermidiate BAGs could get quite large. If you start getting memory related exceptions, then you can inspire from How to handle spill memory in pig. The idea would be to use 2 GROUP BY operations, first one to get counts per (val, cat) and second to pivot R, Y, C around val, thus avoiding expensive JOIN operation (see Pivoting in Pig).
Regarding the question with BOOLEAN: I have used bincond operator.
If you do not need the counts, you could use IsEmpty(bag) instead of SIZE(bag), it would be slightly faster and bincond to get your 0 and 1 conversions.
Related
how can i find the continuity of a field and starting position
The input is like
A-1
B-2
B-3
B-4
C-5
C-6
The output i want is
A,1,1
B,3,2
C,2,5
Thanks.
Assuming you do not have discontinuous data with respect to a value, you can get the desired results by first grouping on value and using COUNT and MIN to get continuous_counts and start_index respectively.
A = LOAD 'data' USING PigStorage('-') AS (value:chararray;index:int);
B = FOREACH (GROUP A BY value) GENERATE
group as value,
COUNT(A) as continuous_counts,
MIN(A.value) as start_index;
STORE B INTO 'output' USING PigStorage(',');
If your data does have the possibility of discontinuous data, the solution is not longer trivial in native pig and you might need to write a UDF for that purpose.
Group and count the number of values for continous_counts. i.e.
A,1
B,3
C,2
Get the top row for each value. i.e.
A,1
B,2
C,5
Join the above two relations and get the desired output.
A = LOAD 'data.txt' USING PigStorage('-') AS (value:chararray;index:int);
B = GROUP A BY value;
C = FOREACH B GENERATE group as value,COUNT(A.value) as continuous_counts;
D = FOREACH B {
ordered = ORDER B BY index;
first = LIMIT ordered 1;
GENERATE first.value,first.index;
}
E = JOIN C BY value,D BY value;
F = FOREACH E GENERATE C::value,C::continuous_counts,D::index;
DUMP F;
Relatively new to Pig scripts. I have below script to derive the error details grouped by Error Code, Name and their respective count.
A = LOAD 'traffic_error_details.txt' USING
PigStorage(',') as (id:int, error_code:chararray,error_name:chararray, error_status:int);
B = FOREACH A GENERATE A.error_code as errorCode,A.error_name as
errorName,A.error_status as errorStatus;
C = GROUP B by ($0,$1,$2);
F = FOREACH C GENERATE group, COUNT(B) as count;
Dump F;
Above would give results as below :
INVALID_PARAM,REQUEST_ERROR,10
INTERNAL_ERROR,SERVER_ERROR,15
NOT_ALLOWED,ACCESS_ERROR,4
UNKNOWN_ERR,UNKNOWN_ERROR,10
NIL,NIL,11
I would want to display percentage of errors as well. So as below :
INVALID_PARAM,REQUEST_ERROR,10,20%
INTERNAL_ERROR,SERVER_ERROR,15,30%
NOT_ALLOWED,ACCESS_ERROR,4,9%
UNKNOWN_ERR,UNKNOWN_ERROR,10,20%
NIL,NIL,11,21%
Here total number of requests considered is 50. Out of which 21% are successful. Remaining are splitup of Error %.
So how to calculate the total as well in the same script and in the same tuple ? so that % could be calculated as (count/total)*100.
Total refers to the count of all records error_details.txt.
After you've gotten counts for each error code, you would need to do a GROUP ALL to find the total number of errors and add that field to every row. Then you can divide the error code counts by the total count to find percent. Make sure you convert the counts variables from type long to type double to avoid any integer division problems.
This is the code:
A = LOAD 'traffic_error_details.txt' USING PigStorage(',') as
(id:int, errorCode:chararray, errorName:chararray, errorStatus:int);
B = FOREACH A GENERATE errorCode, errorName, errorStatus;
C = GROUP B BY (errorCode, errorName, errorStatus);
D = FOREACH C GENERATE
FLATTEN(group) AS (errorCode, errorName, errorStatus),
COUNT(B) AS num;
E = GROUP D ALL;
F = FOREACH E GENERATE
FLATTEN(D) AS (errorCode, errorName, errorStatus, num),
SUM(D.num) AS num_total;
G = FOREACH F GENERATE
errorCode,
errorName,
errorStatus,
num,
(double)num/(double)num_total AS percent;
You'll notice I modified your code slightly. I grouped by (errorCode, errorName, errorStatus) instead of ($0,$1,$2). It's safer to refer to the field names themselves instead of their positions in case you modify your code in the future and the positions aren't the same.
I am trying to characterize fractions of rows having certain properties using Apache Pig.
For example, if the data looks like:
a,15
a,16
a,17
b,3
b,16
I would like to get:
a,0.6
b,0.4
I am trying to do the following:
A = LOAD 'my file' USING PigStorage(',');
total = FOREACH (GROUP A ALL) GENERATE COUNT(A);
which gives me total = (5), but then when I attempt to use this 'total':
fractions = FOREACH (GROUP A by $0) GENERATE COUNT(A)/total;
I get an error.
Clearly COUNT() returns some kind of projection and both projections (in computing total and fractions) should be consistent. Is there a way to make this work? Or perhaps just to cast total to be a number and avoid this projection consistency requirement?
One more way to do the same:
test = LOAD 'test.txt' USING PigStorage(',') AS (one:chararray,two:int);
B = GROUP test by $0;
C = FOREACH B GENERATE group, COUNT(test.$0);
D = GROUP test ALL;
E = FOREACH D GENERATE group,COUNT(test.$0);
F = CROSS C,E;
G = FOREACH F GENERATE $0,$1,$3,(double)($1*100/$3);
Output:
(a,3,5,0.6)
(b,2,5,0.4)
You will have to project and cast it to double:
total = FOREACH (GROUP A ALL) GENERATE COUNT(A);
rows = FOREACH (GROUP A by $0) GENERATE group,COUNT(A);
fractions = FOREACH rows GENERATE rows.$0,(double)rows.$1/(double)total.$0;
For some reason the following modification of what #inquisitive-mind suggested works:
total = FOREACH (GROUP A ALL) GENERATE COUNT(A);
rows = FOREACH (GROUP A by $0) GENERATE group as colname, COUNT(A) as cnt;
fractions = FOREACH rows GENERATE colname, cnt/(double)total.$0;
I am using Apache DataFu
(http://datafu.incubator.apache.org/docs/datafu/1.3.0/datafu/pig/stats/Quantile.html)
to compute the quantiles of several variables in the data.
My undestanding is that the data needs to be sorted before calling quantile.
However, what if I need to compute quantiles for several variables within the same GROUP BY sequence? Given that the creation of the quantile variable occurs after the GENERATE, only one sorting (the last one) will be taken into account, as in this simple example:
-- input: 9,10,2,3,5,8,1,4,6,7
input = LOAD 'input' AS (val:int);
grouped = GROUP input ALL;
-- produces: (1,5.5,10)
quantiles = FOREACH grouped {
sorted = ORDER input BY val;
GENERATE Quantile(sorted);
}
works for one variable, but how can I modify this code if I want the quantiles for two different variables in each groupby?
say, if the data looks like
group col1 col2
A 1 2
A 3 1
B 1 0
B 9 -2
for each group, I want the quantiles of col1 and col2?
You can have multiple ORDER statements in a nested FOREACH, for example:
x = FOREACH grouped {
sorted_by_col1 = ORDER input BY col1;
sorted_by_col2 = ORDER input by col2;
GENERATE
group,
Quantile(sorted_by_col1.col1),
Quantile(sorted_by_col2.col2);
}
I'm trying to run simple word counter in pig latin as follows:
lines = LOAD 'SOME_FILES' using PigStorage('#') as (line:chararray);
word = FILTER lines BY (line matches '.*SOME_VALUE.*');
I want to count how many SOME_VALUEs found searching SOME_FILES, so the expected output should be something like:
(SOME_VALUE,xxxx)
Where xxxx, is the total number of SOME_VALUE found.
How can I search for multiple values and print each one as above ?
What you should do is split each line into a bag of tokens, then FLATTEN it. Then you can do a GROUP on the words to pull all occurrences of each word into it's own line. Once you do a COUNT of the resulting bag you'll have the total count for all words in the document.
This will look something like:
B = FOREACH lines GENERATE FLATTEN(TOKENIZE(line)) ;
C = GROUP B BY $0 ;
D = FOREACH C GENERATE group AS word, COUNT(B) AS count ;
If you aren't sure what each step is doing, then you can use DESCRIBE and DUMP to help visualize what is happening.
Update: If you want to filter the results to contain only the couple of strings you want you can do:
E = FILTER D BY (word == 'foo') OR
(word == 'bar') OR
(word == 'etc') ;
-- Another way...
E = FILTER D BY (word matches 'foo|bar|etc') ;
However, you can also do this between B and C so you don't do any COUNTs you don't need to.