My input file is below
a1,1,on,400
a1,2,off,100
a1,3,on,200
I need to add $3 only if $2 is equal to "on".I have written script as below, after that I don't know how to proceed. For adding $3 only I need to apply some filter. for adding $1 there is no filter at all
Can someone help me on finishing this.
myinput = LOAD 'file' USING PigStorage(',') AS(id:chararray,flag:chararray,amt:int)
grouped = GROUP myinput BY id
I need output as below
a1, 6,600
Here is a possible solution,
You could do something like this (not tested) :
myinput = LOAD 'file' USING PigStorage(',');
A = FOREACH myinput GENERATE $0 as id, $1 as first_sum, (($2 == 'on') ? $3 : 0) as second_sum;
grouped = GROUP A BY id;
RESULT = FOREACH grouped GENERATE group as id, SUM($1.first_sum), SUM($1.second_sum);
That should do the trick
Try this
myinput = LOAD '/home/gopalkrishna/PIGPRAC/pig-sum.txt' using PigStorage(',') as (name:chararray,num:int,stat:chararray,amt:int);
A = GROUP myinput BY name;
B = FOREACH A GENERATE group, SUM(myinput.num),SUM(myinput.amt);
STORE B INTO 'SUMOUT';
Related
I have data set like below :
"column,1A",column2A,column3A
"column,1B",column2B,column3B
"column,1C",column2C,column3C
"column,1D",column2D,column3D
What separator I should be using in this case to separate out above 3 columns.
First column value is => Column,1A
Second column value is => Column2A
Third column value is => Column3A
Let be try my code:
a = LOAD '/home/hduser/pig_ex' USING PigStorage(',') AS (col1,col2,col3,col4);
b = FOREACH a GENERATE REGEX_EXTRACT(col1, '^\\"(.*)', 1) AS (modfirstcol),REGEX_EXTRACT(col2, '^(.*)\\"', 1) AS (modsecondcol),col3,col4;
c = foreach b generate CONCAT($0, CONCAT(', ', $1)), $2 , $3;
dump c;
I am able to resolve it using the below steps:
Input:-
"column,1A",column2A,column3A
"column,1B",column2B,column3B
"column,1C",column2C,column3C
"column,1D",column2D,column3D
PiG Script :-
A = load '/home/hduser/pig_ex' AS line;
B = FOREACH A GENERATE FLATTEN(STRSPLIT(line,'\\,',4)) AS (firstcol:chararray,secondcol:chararray,thirdcol:chararray,forthcol:chararray);
C = FOREACH B GENERATE REGEX_EXTRACT(firstcol, '^\\"(.*)', 1) AS (modfirstcol),REGEX_EXTRACT(secondcol, '^(.*)\\"', 1) AS (modsecondcol),thirdcol,forthcol;
D = FOREACH C GENERATE CONCAT(modfirstcol,',',modsecondcol),thirdcol,forthcol;
DUMP D;
Output :-
(column,1A,column2A,column3A)
(column,1B,column2B,column3B)
(column,1C,column2C,column3C)
(column,1D,column2D,column3D)
Please let me know if there is any better way
I am new to Pig Latin, I am trying below example with Pig BUILT IN functions.
A = LOAD 'student.txt' AS (name:chararray, term:chararray, gpa:float);
B = GROUP A BY name;
DUMP B;
(John,{(John,sm,3.8),(John,sp,4.0),(John,wt,3.7),(John,fl ,3.9)})
(Mary,{(Mary,sm,4.0),(Mary,sp,4.0),(Mary,wt,3.9),(Mary,fl,3.8)})
I need to retrieve 1st element => (John,sm,3.8) and last element => (John,fl ,3.9) from the bag.
Need help to resolve with out using UDF.
Ok.. You can use this solution.. But it is little lengthy.
names = LOAD '/user/user/inputfiles/names.txt' USING PigStorage(',') AS(name:chararray,term:chararray,gpa:float);
names_rank = RANK names;
names_each = FOREACH names_rank GENERATE $0 as row_id,name,term,gpa;
names_grp = GROUP names_each BY name;
names_first_each = FOREACH names_grp
{
order_asc = ORDER names_each BY row_id ASC;
first_rec = LIMIT order_asc 1;
GENERATE flatten(first_rec) as(row_id,name,term,gpa);
};
names_last_each = FOREACH names_grp
{
order_desc = ORDER names_each BY row_id DESC;
last_rec = LIMIT order_desc 1;
GENERATE flatten(last_rec) as(row_id,name,term,gpa);
};
names_unioned = UNION names_first_each,names_last_each;
names_extract = FOREACH names_unioned GENERATE name,term,gpa;
names_ordered = ORDER names_extract BY name;
dump names_ordered;
Output :-
(John,fl,3.9)
(John,sm,3.8)
(Mary,fl,3.8)
(Mary,sm,4.0)
I need to extract post code district from the input data below
AB55 4
DD7 6LL
DD5 2HI
My Code
A = load 'data' as postcode:chararray;
B = foreach A {
code_district = REGEX_EXTRACT(postcode,'<SOME EXP>',1);
generate code_district;
};
dump B;
Output should look like
AB55
DD7
DD5
what should be the regular expression to extract the first part of the string?
Can you try the below Regex?
Option1:
A = LOAD 'input' as postcode:chararray;
code_district = FOREACH A GENERATE REGEX_EXTRACT(postcode,'(\\w+).*',1);
DUMP code_district;
Option2:
A = LOAD 'input' as postcode:chararray;
code_district = FOREACH A GENERATE REGEX_EXTRACT(postcode,'([a-zA-Z0-9]+).*',1);
DUMP code_district;
Output:
(AB55)
(DD7)
(DD5)
This is my input file
a1,hello.VDF
a2,rim.VIM
a3.dr.VDD
I need output as below
a1,VDF
a2,VIM
a3,VDD
My script is the following:
myinput = LOAD 'file' USING PigStorage(',')
AS(t1:chararray,t2:chararray); foreached= FOREACH myinput GENERATE
t1,SUBSTRING(t2,INDEXOF(t2,'.',1),SIZE(t2));
It's throwing some error. Please help
Try this:
output = foreach myinput generate ((t1 matches '(.*)\\.(.*)'?SUBSTRING(t1, 0, 2):t1), (t1 matches '(.*)\\.(.*)'?SUBSTRING(t1, INDEXOF(t1,'.',0)+1, (int)SIZE(t1)):t2));
SIZE returns long, but SUBSTRING takes integers, so you need to do conversion:
foreached =
FOREACH myinput GENERATE t1,SUBSTRING(t2,INDEXOF(t2,'.',1)+1,(int)SIZE(t2));
I have been stuck on this for several hours and I cannot figure out what I am doing wrong.
I have a relation "grouped" with the schema of
grouped: {seedword: chararray,baggy: {outertup: (groupy: (seedword: chararray,coword: chararray))}}
A sample of what the relation looks like is:
(auto,{((auto,car)),((auto,truck))})
I need to generate just the seedword and a tuple of cowords. In my example I would want
(auto, (car, truck)).
I have tried:
FOREACH grouped GENERATE baggy::outertup.groupy.coword;
FOREACH grouped GENERATE baggy.outertup.groupy.coword;
FOREACH grouped GENERATE baggy.groupy.coword;
and none of these work, and give me error messages saying there is no such field. Please help! !!
HEre's some more of my code:
keywords = LOAD 'merged' USING as ( seedword:chararray, doc:chararray);
---COUNT HOW MANY DOCUMENTS EACH WORD IS IN
group_by_seedword = GROUP keywords BY $0;
invert_index = FOREACH group_by_seedword GENERATE $0 as seedword:chararray, keywords.$1;
word_doc_count= FOREACH invert_index GENERATE seedword, COUNT($1);
-- map words to document
words_in_doc= GROUP keywords BY doc;
word_docs = FOREACH words_in_doc GENERATE group AS doc, keywords.seedword;
--(document:(keyword, keyword, keyword...))
--map words to their cowords in doc
temp_join = JOIN keywords BY doc,word_docs BY doc;
--DUMP temp_join;
cowords_by_doc = FOREACH temp_join GENERATE $0 as seedword:chararray, $3 as cowords;
cowords_interm= FOREACH cowords_by_doc GENERATE seedword, FLATTEN(cowords);
cowords = FILTER cowords_interm BY (seedword!=$1);---GETS RID OF SINGLE DOC WORD;
temp_join_count1 = JOIN cowords BY $0, word_doc_count BY seedword;
-- GETS WORDS THAT OCCURE BY THEMSELVES IN A SINGLE DOCUMENT
G = JOIN cowords_interm BY $0 LEFT OUTER, cowords by $0;
orph_word = FILTER G BY $2 is null;
orph_word_count = FOREACH orph_word GENERATE $0,null, 0;
temp_join_count= UNION temp_join_count1, orph_word_count;
inter_frac = FOREACH temp_join_count GENERATE $0 as seedword:chararray, $1 as coword:chararray, 1.0/$3 as frac:double;
inter_frac_combine = GROUP inter_frac BY (seedword, coword);
inter_frac_sum = FOREACH inter_frac_combine GENERATE $0 , SUM(inter_frac.frac) as frac:double;
filtered = FILTER inter_frac_sum BY ($1 >=$relatedness_ratio);
grouped= GROUP filtered by $0.seedword;
g = FOREACH grouped GENERATE group as seedword:chararray, filtered.$0;
named = FOREACH g GENERATE $0 as seedword:chararray, $1 as baggy:bag{(outertup:tuple(groupy:tuple(seedword:chararray, coword:chararray)))};
the input file you can try should be like this:
car doc1.txt
auto doc1.txt
bunny doc2.txt
ball doc2.txt
toy car doc2.txt
random doc3.txt
plane doc3.txt
I'd had a similar issue where I couldn't reference inner tuples.
My solution was to flatten the data and then some more filtering and grouping.
Cheers
V