How to format dates during the process of creating Hive tables?
I've currently been dumping some data into a discovery environment at work and storing dates as string, because if I format them as a DATE or TIMESTAMP the values are null.
Here's what the raw data looks like:
12/07/2016 05:07:28 PM
My understanding is that Hive accepts dates in this format
yyyy-mm-dd hh:mm:ss
I can format these using a select statement:
select id, receipt_dt, from_unixtime(unix_timestamp(receipt_dt ,'MM/dd/yyyy'), 'yyyy-MM-dd') as app_dt from MySchema.MyTable where app_num='123456'
How can I add in the statement
from_unixtime(unix_timestamp(receipt_dt ,'MM/dd/yyyy'), 'yyyy-MM-dd')
How can I add this in to the generic CREATE EXTERNAL STATEMENT below so that I no longer have to store dates as a string, or use an ALTER TABLE statement to change the formatting?
CREATE EXTERNAL TABLE IF NOT EXISTS MySchema.My_New_Table
( Field1 Format,
Field2 Format,
Field 3 Format,
)
.......
Use MyTable as staging table with raw data and create final/target table my_new_table with transformations i.e, date format...it will be EDW kind of process...
example:
CREATE EXTERNAL TABLE IF NOT EXISTS MySchema.My_New_Table
( Field1 int,
Field2 string,
Field3 date
)
... more definitions....
AS
select id, receipt_dt,
cast(from_unixtime(unix_timestamp(receipt_dt ,'MM/dd/yyyy'), 'yyyy-MM-dd') as date) as app_dt
from MySchema.MyTable ;
NOTE: This is not tested statement. You may need to try and edit and try...but you got the idea...
Then inserting delta should be similar process...
INSERT INTO TABLE MySchema.My_New_Table
AS
select id, receipt_dt,
cast(from_unixtime(unix_timestamp(receipt_dt ,'MM/dd/yyyy'), 'yyyy-MM-dd') as date) as app_dt
from MySchema.MyTable where <<conditions>>;
Related
Can we compare the columns of same table and get a result by selecting all the rows where both columns data does not match. Compare CREATE_DATE and UPDATE_DATE
Table with data
Expected output:
SELECT *
FROM Table
WHERE TIME_CREATED <> UPDATE_DATE
There should be no issue comparing date to timestamp, please update your OP if you have tried this already
If there is an issue, read this post and try something like this:
Oracle comparing timestamp with date (also shows use of truncate to disregard time of day)
SELECT *
FROM Table
WHERE to_timestamp(TIME_CREATED,'D/MM/YYYY HH:MM:SS') <> UPDATE_DATE
OR
SELECT *
FROM Table
WHERE TIME_CREATED <> TO_DATE(UPDATE_DATE,'DD.MM.YYYY:HH24:MI:SS')
You might have to play with the timestamp format a bit, see https://www.akadia.com/services/ora_date_time.html
I have three table in the database two of them contain dates however the dates are in two format, first 20-02-2011 and second is 25/09/2018. Let say each table have 10000 records and mixed with these two types of dates format. This why I why I create the column like --- (Transaction_Date, Varchar(10) Not Null)
I tried convert (Varchar(10),Transaction_Date,105)
and also tried replace(convert(varchar(10),Transaction_Date,105),'/','-')
However date and year functions are still not working.
Please suggest a possible way.
How about this?
select replace(date, replace(Transaction_Date, '/', '-'), 105)
That is: (1) convert to a date and (2) replace the slash before converting.
You need to remember about your culture. Saved format vs server culture. But this is very possible
select Cast('2-22-2011' as datetime) f1,
Cast('2/22/2011' as datetime) f2
I other words just use Cast
select cast(Transaction_Date as datetime) . . .
But you should as soon as possible get rid of columns that saves date as string and create new date/time column, and insert your date values there
alter table tbl add column temp datetime
update tbl set temp = Cast(Transaction_Date as datetime)
alter table tbl drop column Transaction_Date
alter table tbl add column Transaction_Date datetime
update tbl set Transaction_Date = temp
alter table tbl drop column temp
I have a value of 12/31/18 and created table in snowflake:
create table my_date (a date);
insert into my_date values ('12/31/18');
select * from my_date;
Result: 0018-12-31
I want to get: 2018-12-31
I saw about 2 number format:
https://docs.snowflake.net/manuals/sql-reference/parameters.html#label-two-digit-century-start
but not sure if this is specification of a column type or data needs to be transformed before the insert?
The parameter two_digit_century_start seems not to be used when parameter date_input_format is set to AUTO. You can get your example working correctly by setting the date format with a parameter ("alter session..." statement on line 2 below). Your complete working example would look like this:
create table my_date (a date);
alter session set DATE_INPUT_FORMAT = 'MM/DD/YY';
insert into my_date values ('12/31/18');
select * from my_date;
This results in 2018-12-31.
Snowflake best-practices recommend to specify the format explicitly with to_date(value, 'format') or by setting the format in parameters. You can find the best practices for date/time functions from Snowflake documentation here: https://docs.snowflake.net/manuals/user-guide/date-time-input-output.html#date-time-function-format-best-practices
I have a table in Access named TEST_DATE1 with the column TEST_DATE that is a String datatype and the records look like 20080130. yyyymmdd
I have a table in Oracle named TEST with the column TEST_DATE that is a DATE datatype and I want the records to look like 2008/01/30 yyyy/mm/dd.
I have the two tables linked and when I usually update tables between Access and Oracle I usually do a
INSERT INTO TEST
SELECT *
FROM TEST_DATE1;
How would you convert the string to a DATE using the INSERT INTO SELECT
I have tried
INSERT INTO TEST
(SELECT TO_DATE(TEST_DATE, 'yyyy/mm/dd'))
FROM TEST_DATE1;
Thanks!
To move the records to Oracle, converting a string to a date:
INSERT INTO test (test_date)
SELECT TO_DATE(test_date, 'YYYYMMDD')
FROM test_date1
Once it's in Oracle stored as a date, you can retrieve it in any format you like:
SELECT TO_CHAR(test_date, 'YYYY/MM/DD') as test_date
FROM test;
Am a Oracle PL/SQL newbie. Basically, I have a table created as follows:
CREATE TABLE MYTABLE (
ID VARCHAR2(3 CHAR) NOT NULL PRIMARY KEY,
LAST_UPDATE DATE
);
Am trying to write a script which will run through SQL*Plus command:
insert into MYTABLE (
id,
last_update)
values (
sys_guid,
--- Date call - what is placed here?
);
What can I do to insure that the date inserted is the correct date (at time of insertion) with this format (what function to call):
27-Oct-11
Will need this script to be executable from within SQL*Plus (read that there's numerous amounts of incompatibilities between ANSI SQL & PL/SQL running from SQL Developer vs. SQL*Plus).
Thanks for taking the time to read this!
Date fields don't have a format. You apply a format when you select them by using the to_char function. So what you want to do is
insert into mytable
(id, last_update)
values
(sys_guid, SYSDATE);
And you select from it using
select id, to_char(last_update, 'DD-Mon-YY')
from mytable;