I have a pipeline where im getting data from sqlserver and load it into Hive table.I have a timestamp column in the source which is like 'YYYY-MM-DD HH:MM:SS'
Sql table(datetime) ---> Hive stage table(string)---->Hive final table(timestamp)
The source table is in US/Pacific time zone. In the middle stage table, the format is like 'YYYY-MM-DD HH:MM:SS.0'.
How do i convert into a timestamp field for the final table? I want the final table column to look like 'YYYY-MM-DD HH:MM:SS'
I see from_unixtime being used, but when i try like below,it returns null.
FROM_UNIXTIME(UNIX_TIMESTAMP('date column','yyyy-mm-dd HH.mm.ss')) as ts
Im pretty new to using Hive and need some suggestion on what should i do here, Thanks.
If the timestamp string is in format 'yyyy-MM-dd HH:mm:ss.S' then you can cast it to timestamp type using timestamp() function.
timestamp(col)
Also you can insert string directly into timestamp column.
It works because 'yyyy-MM-dd HH:mm:ss.S' - is a default timestamp format.
You need conversion using FROM_UNIXTIME(UNIX_TIMESTAMP(col, format)) if the format is not 'yyyy-MM-dd HH:mm:ss.S'. This format you should convert to, not from. Specify correct FROM format, it is case-sensitive: MM is not the same as mm, delimiters do matter: dot is not the same as semicolon or space, etc.
See format manual here: SimpleDateFormat
Also see this post about timestamp with nanoseconds
Related
I uploaded a table using fastload and want to convert the event_time column to a timestamp or just get the dates since I don't need the time component.
The text format is 'MM/DD/YYYY HH:MM:SS AM/PM', for example, '05/24/2013 08:12:00 AM'
I have tried
CAST(event_time AS TIMESTAMP(0))
and
CAST(SUBSTRING (event_time,1,10) AS date)
but they don't work for me. The error I get is invalid timestamp/date.
As a bonus question,
I don't think there are any outliers in the data (i.e. every row is in format as described above), but if there were, how do I account for those errors? or will Teradata automatically pass on those rows?
The CASTs don't work because this is not the default timestamp/date format.
The easiest way utilizes the TO_DATE function which is supported since TD14:
TO_DATE(event_time, 'MM/DD/YYYY HH:MM:SS AM')
Before TD14 it's:
CAST(CAST(event_time AS TIMESTAMP FORMAT 'MM/DD/YYYYbHH:MI:SSbT') AS DATE)
Regarding outliers, they will result in an "invalid date/timestamp" error. When you cast them during FastLoad to a Timestamp those rows will fail to load and will be inserted into the ET-error table. But you can't use the newer CAST-syntax in FL, must be the old Teradata-style cast:
:event_time (TIMESTAMP, FORMAT 'MM/DD/YYYYbHH:MI:SSbT')
I am trying to create a table in my Oracle Database with the following structure:
CREATE TABLE orderlines(collectionTime TIMESTAMP NOT NULL);
An example of a timestamp that will be going in a row is:
20-01-2015 11:33:48-04:00
What would be the best datatype to use to store this timestamp?
TIMESTAMP, or DATE?
Also, what would the format be?
When I tried to import my CSV file containing all of the timestamps, SQL Developer Import Wizard didnt like the formatting. I tried
DD-MM-YYYY HH24:SS:MM-TZ
What would be the best datatype to use to store this timestamp? TIMESTAMP, or DATE?
The TIMESTAMP datatype is an extension to the DATE datatype. In addition to the datetime elements, the TIMESTAMP datatype holds fractions of a second. It comes in two forms, TIMESTAMP WITH TIME ZONE and TIMESTAMP WITH LOCAL TIME ZONE. So, if you want the precision till fraction of seconds along with the timezone elements, go for TIMESTAMP. Else, a DATE data type will give you datetime elements.
Also, what would the format be?
Format is only for DISPLAY. So use the format model which you would like to display. You need to use TO_CHAR and a proper format model. For example,
SQL> select to_char(sysdate, 'mm/dd/yyyy hh24:mi:ss') from dual;
TO_CHAR(SYSDATE,'MM
-------------------
01/21/2015 11:02:27
SQL> select to_char(sysdate, 'mm/dd/yyyy hh:mi:ss am') from dual;
TO_CHAR(SYSDATE,'MM/DD
----------------------
01/21/2015 11:03:04 am
SQL>
When I tried to import my CSV file containing all of the timestamps,
SQL Developer Import Wizard didnt like the formatting.
That is the issue with your locale-specific NLS_DATE_FORMAT.
Try 'DD-MM-YYYY HH24:MI:SS-TZH:TZM'. This is the mask that would be recognized by to_timestamp_tz. Notice that MM is for months and MI is for minutes. Also, you had hours:seconds:minutes in your mask, but that would result in a wrong value not an error.
How to convert the oracle timestamp to postgres timestamp,we have a values in oracle data base as below,
15-JUN-2014 01.00.00.0000 PM
I have stored as varchar in postgres/Greenplum. How can I cast to timestamp in postgres,
Output required As below ,
15-06-2014 13.00.00.0000
As 24 hrs format.
Timestamps are timestamps. They represent a date (incl. time) using an internal representation. But you can convert timestamps to/from strings using various formats. For example, if you need to display them or in order to export your data to an other software requiring some specific representation.
In the following code for Oracle, I convert from a string to a timestamp (TO_TIMESTAMP) using a format corresponding to your first example. Then I "convert back" that timestamp to a string (TO_CHAR) using an other format corresponding to your second representation:
-- Convert a string to a timestamp
WITH sample_data AS
(SELECT TO_TIMESTAMP('15-JUN-2014 01.00.00.0000 PM',
'DD-MON-YYYY HH12.MI.SS.FF4 PM') ts FROM DUAL)
-- Convert a timestamp to a string using a format corresponding
-- to "15-06-2014 13.00.00.0000"
SELECT TO_CHAR(ts, 'DD-MM-YYYY HH24.MI.SS.FF4') as result FROM sample_data;
Producing (the string):
result
15-06-2014 13.00.00.0000
Please note that you can convert from string to date in PostgreSQL too using (almost?) exactly the same date functions and formats.
SELECT To_timestamp(event_timestamp,'DD-MON-YY HH12.MI.SS.SSSS AM.PM')::timestamp without time zone
FROM jiodba.ext_wfa_accounting_msg limit 1000;
Please find the answer
I want to map a datetime mm/dd/yyyy hh24:mi:ss attribute in flatfile to Teradata table date yyyy-mm-dd attribute using informatica.
When I added to_date(date_field, 'yyyy-mm-dd') I'm encountering oracle fatal error. When I tried with to_date(to_char(date_field, 'yyyy-mm-dd')) it is giving invalid string input to to_date().
Can anyone help?
In Informatica, the format that you specify under to_date() function should be same as your source data format and not your target format.
So in your case, to_date function should be like this:
to_date (date_field, 'mm/dd/yyyy hh24:mi:ss')
This is because your flat file date has a format of mm/dd/yyyy hh24:mi:ss (Ensure that all the records in this column in your flat file date are really in this format - else you will encounter error)
Do not worry about target date format as long as the target column is a date datatype. By nature, date datatype does not have a format, it's only the display that needs format.
I'm trying to create a column of type TIMESTAMP such that would store timestamps in the following format (i.e. including the "AM"/"PM" indicator):
20-04-2013 06:56:37 AM
I suppose I could use the to_char(..) function and store the timestamps as TEXT values, however isn't it possible to achieve the same effect with a vanilla TIMESTAMP column?
If you want to format a timestamp, use to_char when you SELECT it, or format the date in the client.
Timestamps don't have a format. They're stored as the number of fractional seconds since 1970-01-01 UTC inside the database and they're formatted into ISO timestamps for input and output. Inside the database they're just a number.
There is no way to override the default timestamp output format, since that would lead to a setting that could confuse applications that expect a single consistent format.
If the app wants a different date output format it has to ask for it with an appropriate to_char call.
Is this roughly what you are looking for?
select to_char(now(), 'DD-MM-YYYY HH12:MI:SS AM')