I'm trying to load a date field from PIG into a hive table, the PIG script has the field as:
testdate:chararray
The Hive table description is:
col_name data_type
testdate date
Pig 'chararray' type in column 0 cannot map to HCat 'DATE'type.
Target filed must be of HCat type {STRING or CHAR or VARCHAR}
I looked up http://pig.apache.org/docs/r0.9.1/basic.html and discovered PIG does not have a format for date with HCatalog. What is the best solution for storing this value as a date in HIVE if I must use PIG? Thanks.
Ok I figured it out, since there is limited info on this topic:
https://cwiki.apache.org/confluence/display/Hive/HCatalog+LoadStore#HCatalogLoadStore-TypesinHive0.13.0andLater.1
I needed to use the DATETIME instead of chararray
Table of type conversions:
Related
We have a source as a flat file which consists of a date column in format 19/08/2013.
We have a target in oracle table which consists of a date column in format 11-AUG-13.
When we are trying to pass the source column value in target using expression TO_DATE AND
TO_CHAR like
**source column is A ---> Source column
v1=TO_CHAR(A)
O1(output column)=TO_DATE(V1,'DD-MON-YY') we are getting the below error.
Invalid date: [19/8/2013]. The row will be skipped.**
Can anyone please help where I'm going wrong.
Thank you
You need to convert the str to date properly and then infa will load them.
Change the mapping like this -
Change data type in source to a string. read data as string.
Then use your expressions but change the to_date function like below.
v1(var column, data type string)=A
O1(output column, data type date)=IIF(IS_DATE(V1,'DD/MM/YYYY'), TO_DATE(V1,DD/MM/YYYY'),null)
IS_DATE - will check if the data is a date or not, if yes then only it will try to convert to a proper date else put null.
3. Then connect O1 column to a date type in oracle target.
This will make sure all data are loaded and none skipped.
I am preparing an Informatica mapping
Source:- Oracle
Target :- Teradata
There are 2 columns in the mapping
Source column name : INX_DATE(string) and UPX_DATE (string)
Target column name : INX_DATE(date/time) and UPX_DATE (date/time)
Source column contains something like this
UPX_Date:-08-FEB-16 12.00.23.826315000 AM +00:00
INX_Date:- 08-FEB-16 12.00.23.826315000 AM EUROPE/BELGRADE
Now the issue is how will I get this to my target which is date/time
and should be MM/DD/YYYY HH:MM:SS
You can substring the first 31 characters and then convert it to date.
TO_DATE(SUBSTR(INX_DATE,1,31),'DD-MON-YY HH12:MI:SS.NS AM')
As far as I know, PowerCenter Date/Time cannot store the time zone information. If you need that, you may need to adjust the time separately.
in Hive table value for one column is like 01/12/17.But I need the value in the format as 12-2017(month-year).How to convert it?
Convert the string to a unix_timestamp and output the required format using from_unixtime.
select from_unixtime(unix_timestamp(col_name,'MM/dd/yy'),'MM-yyyy')
in ssis I'm loading a data file that has a date field formatted as 11171977 into a table with a data type of varchar(8).
i can load this into the table fine; however, I need to reformat the data as yyyymmdd so it is 19771117.
my first attempt was to load as is then query with the following convert:
convert(varchar(8),convert(date,substring (BirthDate,1,2)+'/'+substring (BirthDate,3,2)+'/'+substring (BirthDate,5,4),101), 112)
that gives me yyyymmdd = 19771117.
however, I tried to do a derived column with no luck as it didn't like my expression.
how do i do this in an expression in the derived column?
Try something like this:
right(BirthDate,4)+left(BirthDate,4)
For Ex: I am bringing the Hive table column (datetime data type) value in Pig and want to extract on;y the DATE portion. I have tried using ToDate function. the below is the Error Information. Please help me in this critical situation.
The Original Value in this column is "2014-07-29T06:01:33.705-04:00", I need out put as "2014-07-29"
ToDate(eff_end_ts,'YYYY-MM-DD') AS Delta_Column;
2016-07-28 07:07:25,298 [main] ERROR org.apache.pig.tools.grunt.Grunt
- ERROR 1045: Could not infer the matching function for org.apache.pig.builtin.ToDate as multiple or none of them
fit. Please use an explicit cast.
Assuming your column name is f1 which has the timestamp with values like 2014-07-29T06:01:33.705-04:00, you will have to use GetYear(),GetMonth,GetDay and CONCAT it to the required format.
B = FOREACH A GENERATE CONCAT(
CONCAT(
CONCAT((chararray)GetYear(f1),'-')),
(CONCAT((chararray)GetMonth(f1),'-')),
(chararray)GetDay(f1)) AS Day;
I did the Work around to figure out and Its working by this way:
ToDate(ToString(eff_end_ts,'YYYY-MM-DD'),'YYYY-MM-DD') AS (datetime: Delta_Column)