Operators in PIG Latin - operators

I have two values in a PIG Latin script, what should I do to use them mathematically, like if I had two variables
A=(5)
B=(4)
How can I do A+B or something like that ?


You need to use Foreach stmt to do any mathematical operation, please see the below sample example.
input.txt
2,1
5,3
7,5
Pigscript:
A = LOAD 'input.txt' USING PigStorage(',') AS (val1:int , val1:int);
B = FOREACH A GENERATE (val1+val2) AS sum, (val1-val2) AS diff;
DUMP B;
Output:
(3,1)
(8,2)
(12,2)


Yes you can able to do any arithmetic operation in it but the thing is that any variable declaration is kind of different in pig latin -
what you have to do is -
% declare A 5 % declare B 4
and in any of the foreach statement you can use
dummy = foreach column-1,...., column-n, A+B as summation;
this would work.
Let me know if I fulfilled you need.

Related

How to loop through tuples in a Bag, Pig

I am new to pig scripting.
I have an input, (A,B,{(XYZ,123,CDE)})
I am looking to loop through the bag inside and print the following records.
(A,B,XYZ)
(A,B,123)
(A,B,CDE)
Can someone please help me out!

Lets say X is your relation and it has (A,B,{(XYZ,123,CDE)}).ToBag converts the expression into bags and FLATTEN unnests the tuples,bag.
Y = FOREACH X GENERATE $0,$1,ToBag(FLATTEN($2));

Solved!!
Let us load below file (Tab separated)
A B {(XYZ,123,CDE)}
input_plus_bag = load '' USING PigStorage() AS (entry1:chararray, entry2:chararray, bag1:bag{(te1:chararray, te2:int, te3:chararray)});
intermed_output = foreach input_plus_bag generate entry1, entry2, FLATTEN(bag1);
Dump intermed_output;
This will give
(A,B,XYZ,123,CDE)
DESCRIBE intermed_output;
intermed_output: {entry1: chararray,entry2: chararray,bag1::te1: chararray,bag1::te2: int,bag1::te3: chararray}
Now perform TOBAG operation
intermed2_output = foreach intermed_output generate entry1, entry2, TOBAG(bag1::te1,bag1::te2,bag1::te3);
DUMP intermed2_output;
This will result in below output:-
(A,B,{(XYZ),(123),(CDE)})
Now final step is FLATTEN the bag
final_output = foreach intermed2_output generate entry1, entry2, FLATTEN($2);
And we have our desired output:-
(A,B,XYZ)
(A,B,123)
(A,B,CDE)

Change separator and generate output through PiG

I want to generate the below output from the given input. What will be the best way to get that.
Input:-
"column,1A,extra-A1,extra-A2",column2A,column3A
"((column,1B,extra-B1))",column2B,column3B
"column,1C,extra-C1,extra-C2,extra-C3,extra-C4",column2C,column3C
"column,1D,extra-D1",column2D,column3D
Output:-
column,1A,extra-A1,extra-A2|column2A|column3A
((column,1B,extra-B1))|column2B|column3B
column,1C,extra-C1,extra-C2,extra-C3,extra-C4|column2C|column3C
column,1D,extra-D1|column2D|column3D

I am able to resolve it using below, let me know if you have any better option
Input:-
"column,1A,extra-A1,extra-A2",column2A,column3A
"((column,1B,extra-B1))",column2B,column3B
"column,1C,extra-C1,extra-C2,extra-C3,extra-C4",column2C,column3C
"column,1D,extra-D1",column2D,column3D
Pig Script:-
A = LOAD '/home/hduser/pig_ex1/sample1.txt' AS line;
B = FOREACH A GENERATE SUBSTRING(line,1,(LAST_INDEX_OF(line,'"'))) AS firstcol, SUBSTRING(line,(LAST_INDEX_OF(line,'"')+2),(INT) SIZE(line)) as lastcol;
C = FOREACH B GENERATE firstcol, FLATTEN(STRSPLIT(lastcol,'\\,',2)) AS (secondcol,thirdcol);
D = FOREACH C GENERATE CONCAT(firstcol,'|',secondcol,'|',thirdcol);
Output:-
(column,1A,extra-A1,extra-A2|column2A|column3A)
(((column,1B,extra-B1))|column2B|column3B)
(column,1C,extra-C1,extra-C2,extra-C3,extra-C4|column2C|column3C)
(column,1D,extra-D1|column2D|column3D)

I am using Regular Expression:
Let be try this code:
a = LOAD '/home/hduser/pig_ex1/sample1.txt' as line;
b = FOREACH a GENERATE FLATTEN(REGEX_EXTRACT_ALL(line,'["](.*)["][,](.*)[,](.*)')) AS (f1,f2,f3);
c = FOREACH b GENERATE CONCAT(f1,'|',f2,'|',f3);
dump c;

Try org.apache.pig.piggybank.storage.CSVExcelStorage(','); from piggybank jar.

Unable to convert char array to decimal in the pig

Hi I am new to pig programming. I have one csv file and tab as a delimiter. In my price column I have lacs and crores. It looks like this 40.85 Lacs, 36.73 Lacs, 2.01 cr. I want to convert into decimal like this
40.85 Lacs - 40,85,000
36.73 Lacs - 36,73,000
2.01 cr - 2,01,00000
I tried the following code:
a = LOAD '/user/user1/input/city/cityname.CSV' using PigStorage('|') as (SourceWebSite:chararray,PropertyID:chararray,ListedOn:chararray,ContactName:chararray,TotalViews:int,Price:chararray,PriceperArea:chararray,NoOfBedRooms:int,NoOfBathRooms:int,FloorNoOfProperty:chararray,TotalFloors:int,Possession:chararray,BuiltUpArea:chararray,Furnished:chararray,Ownership:chararray,NewResale:chararray,Facing:chararray,title:chararray,PropertyAddress:chararray,NearByFacilities:chararray,PropertyFeatures:chararray,Sellerinfo:chararray,Description:chararray);
DUMP a;
b = FOREACH a GENERATE Price;
dump b;
c = FILTER b BY (Price matches '.*Lacs.*');
d = FOREACH c GENERATE Price * 10000.0,SUBSTRING(Price,00000);
d = foreach c generate Price,TOKENIZE(REPLACE(Price,'.','')) AS e;
I am struggling for two days in this. Any help will be appreciated.

Use REGEX_EXTRACT to get the first part of the value, until the blank, cast it to BigDecimal and then multiply it. With this input data, for example:
(40.85 Lacs,1)
(36.73 Lacs,2)
(2.01 cr,3)
The following code would work:
A = load 'data' using PigStorage(';');
B = foreach A generate (bigdecimal)REGEX_EXTRACT($0, '(.*) (.*)', 1) * 1000000;
dump B;
And the output would be:
(40850000.00)
(36730000.00)
(2010000.00)

Merge two lines in Pig

I would like to write a pig script for below query.
Input is:
ABC,DEF,,
,,GHI,JKL
MNO,PQR,,
,,STU,VWX
Output should be:
ABC,DEF,GHI,JKL
MNO,PQR,STU,VWX
Could anyone please help me?

It will be difficult to solve this problem using native pig. One option could be download the datafu-1.2.0.jar library and try the below approach.
input.txt
ABC,DEF,,
,,GHI,JKL
MNO,PQR,,
,,STU,VWX
PigScript:
REGISTER /tmp/datafu-1.2.0.jar;
DEFINE BagSplit datafu.pig.bags.BagSplit();
A = LOAD 'input.txt' USING PigStorage(',') AS(f1,f2,f3,f4);
B = GROUP A ALL;
C = FOREACH B GENERATE FLATTEN(BagSplit(2,$1)) AS mybag;
D = FOREACH C GENERATE FLATTEN(STRSPLIT(REPLACE(BagToString(mybag),'_null_null_null_null',''),'_',4));
E = FOREACH D GENERATE $2,$3,$0,$1;
DUMP E;
Output:
(MNO,PQR,STU,VWX)
(ABC,DEF,GHI,JKL)
Note:
Based on the above input format, my assumption will be 1st row last two cols will be null, 2nd row first two cols will be null, similarly for 3rd and 4th row also

Pig Script - Min, Avg, Max

Let us say I have these in a file ...
1
2
3
Using a Pig Script, how can I get this (number, minimum, mean, maximum in each line) ?
1,1,2,3
2,1,2,3
3,1,2,3
Please let me know the Pig Script. I am able to get the MIN, AVG, MAX using Pig built in functions, but am not able to get them all in each line.
Thanks
Naga

Use the TOBAG built-in UDF to get your fields into a bag, and then you can use the MIN, AVG, and MAX functions on that bag. You should have no trouble using all three summary functions on a single record.

Here is my simple solution for the problem.
I had the following numbers as input,
temp2.txt
1
2
3
4
5
.
.
16
17
18
19
20
I followed these steps,
1]loaded the data from the file
2]Then grouped all the data
3]Found Average,Minimum,Maximum from the grouped data
4]Then foreach value in loaded data generated data and the minimum , maximum and average values.
The code is as follows,
grunt> data = load '/home/temp2.txt' as (val);
grunt> g = group data all;
grunt> avg = foreach g generate AVG(data.val) as a;
grunt> min = foreach g generate MIN(data.val) as m;
grunt> max = foreach g generate MAX(data.val) as x;
grunt> values = foreach data generate val,min.m,max.x,avg.a;
grunt> dump values;
The following is the output,
Output
(1,1.0,20.0,10.5)
(2,1.0,20.0,10.5)
(3,1.0,20.0,10.5)
(4,1.0,20.0,10.5)
(5,1.0,20.0,10.5)
(6,1.0,20.0,10.5)
(7,1.0,20.0,10.5)
(8,1.0,20.0,10.5)
(9,1.0,20.0,10.5)
(10,1.0,20.0,10.5)
(11,1.0,20.0,10.5)
(12,1.0,20.0,10.5)
(13,1.0,20.0,10.5)
(14,1.0,20.0,10.5)
(15,1.0,20.0,10.5)
(16,1.0,20.0,10.5)
(17,1.0,20.0,10.5)
(18,1.0,20.0,10.5)
(19,1.0,20.0,10.5)
(20,1.0,20.0,10.5)