Snowflake - insert date - sql

I have a value of 12/31/18 and created table in snowflake:
create table my_date (a date);
insert into my_date values ('12/31/18');
select * from my_date;
Result: 0018-12-31
I want to get: 2018-12-31
I saw about 2 number format:
https://docs.snowflake.net/manuals/sql-reference/parameters.html#label-two-digit-century-start
but not sure if this is specification of a column type or data needs to be transformed before the insert?


The parameter two_digit_century_start seems not to be used when parameter date_input_format is set to AUTO. You can get your example working correctly by setting the date format with a parameter ("alter session..." statement on line 2 below). Your complete working example would look like this:
create table my_date (a date);
alter session set DATE_INPUT_FORMAT = 'MM/DD/YY';
insert into my_date values ('12/31/18');
select * from my_date;
This results in 2018-12-31.
Snowflake best-practices recommend to specify the format explicitly with to_date(value, 'format') or by setting the format in parameters. You can find the best practices for date/time functions from Snowflake documentation here: https://docs.snowflake.net/manuals/user-guide/date-time-input-output.html#date-time-function-format-best-practices

Related

How to convert two types of dates format into one date format in SQL?

I have three table in the database two of them contain dates however the dates are in two format, first 20-02-2011 and second is 25/09/2018. Let say each table have 10000 records and mixed with these two types of dates format. This why I why I create the column like --- (Transaction_Date, Varchar(10) Not Null)
I tried convert (Varchar(10),Transaction_Date,105)
and also tried replace(convert(varchar(10),Transaction_Date,105),'/','-')
However date and year functions are still not working.
Please suggest a possible way.

How about this?
select replace(date, replace(Transaction_Date, '/', '-'), 105)
That is: (1) convert to a date and (2) replace the slash before converting.

You need to remember about your culture. Saved format vs server culture. But this is very possible
select Cast('2-22-2011' as datetime) f1,
Cast('2/22/2011' as datetime) f2
I other words just use Cast
select cast(Transaction_Date as datetime) . . .
But you should as soon as possible get rid of columns that saves date as string and create new date/time column, and insert your date values there
alter table tbl add column temp datetime
update tbl set temp = Cast(Transaction_Date as datetime)
alter table tbl drop column Transaction_Date
alter table tbl add column Transaction_Date datetime
update tbl set Transaction_Date = temp
alter table tbl drop column temp

Hadoop - Formatting dates when creating tables

How to format dates during the process of creating Hive tables?
I've currently been dumping some data into a discovery environment at work and storing dates as string, because if I format them as a DATE or TIMESTAMP the values are null.
Here's what the raw data looks like:
12/07/2016 05:07:28 PM
My understanding is that Hive accepts dates in this format
yyyy-mm-dd hh:mm:ss
I can format these using a select statement:
select id, receipt_dt, from_unixtime(unix_timestamp(receipt_dt ,'MM/dd/yyyy'), 'yyyy-MM-dd') as app_dt from MySchema.MyTable where app_num='123456'
How can I add in the statement
from_unixtime(unix_timestamp(receipt_dt ,'MM/dd/yyyy'), 'yyyy-MM-dd')
How can I add this in to the generic CREATE EXTERNAL STATEMENT below so that I no longer have to store dates as a string, or use an ALTER TABLE statement to change the formatting?
CREATE EXTERNAL TABLE IF NOT EXISTS MySchema.My_New_Table
( Field1 Format,
Field2 Format,
Field 3 Format,
)
.......

Use MyTable as staging table with raw data and create final/target table my_new_table with transformations i.e, date format...it will be EDW kind of process...
example:
CREATE EXTERNAL TABLE IF NOT EXISTS MySchema.My_New_Table
( Field1 int,
Field2 string,
Field3 date
)
... more definitions....
AS
select id, receipt_dt,
cast(from_unixtime(unix_timestamp(receipt_dt ,'MM/dd/yyyy'), 'yyyy-MM-dd') as date) as app_dt
from MySchema.MyTable ;
NOTE: This is not tested statement. You may need to try and edit and try...but you got the idea...
Then inserting delta should be similar process...
INSERT INTO TABLE MySchema.My_New_Table
AS
select id, receipt_dt,
cast(from_unixtime(unix_timestamp(receipt_dt ,'MM/dd/yyyy'), 'yyyy-MM-dd') as date) as app_dt
from MySchema.MyTable where <<conditions>>;

Postgres Data type conversion

I have this dataset that's in a SQL format. However the DATE type needs to be converted into a different format because I get the following error
CREATE TABLE
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
ERROR: date/time field value out of range: "28-10-96"
LINE 58: ...040','2','10','','P13-00206','','','','','1-3-95','28-10-96'...
^
HINT: Perhaps you need a different "datestyle" setting.
I've definitely read the documentation on date format
http://www.postgresql.org/docs/current/static/datatype-datetime.html
But my question is how do I convert all of the dates in a proper format without going through all the 500 or so data rows and making sure each one is correct before inserting into a DB. Backend is handle by rails, but I figured going through SQL to cleaning it up will be best here.
I have a CREATE TABLE statement above this dataset, and mind you the data set was given to be via a DBF converter/external source
Here's part of my dataset
INSERT INTO winery_attributes
(ID,NAME,STATUS,BLDSZ_ORIG,BLDSZ_CURR,HAS_CAVE,CAVESIZE,PROD_ORIG,PROD_CURR,TOUR_TASTG,VISIT_DAY,VISIT_WEEK,VISIT_YR,VISIT_MKTG,VISIT_NMEV,VISIT_ALL,EMPLYEENUM,PARKINGNUM,WDO,LAST_UP,IN_CITYBDY,IN_AIASP,NOTES,SMLWNRYEXM,APPRV_DATE,ESTAB_DATE,TOTAL_SIZE,SUBJ_TO_75,GPY_AT_75,AVA,SUP_DIST)
VALUES
(1,'ACACIA WINERY','PROD','8000','34436','','0','50000','250000','APPT','75','525','27375','3612','63','30987','22','97','x','001_02169-MOD_AcaciaWinery','','','','','1-11-79','1-9-82','34436','x','125000','Los Carneros','1');
INSERT INTO winery_attributes
(ID,NAME,STATUS,BLDSZ_ORIG,BLDSZ_CURR,HAS_CAVE,CAVESIZE,PROD_ORIG,PROD_CURR,TOUR_TASTG,VISIT_DAY,VISIT_WEEK,VISIT_YR,VISIT_MKTG,VISIT_NMEV,VISIT_ALL,EMPLYEENUM,PARKINGNUM,WDO,LAST_UP,IN_CITYBDY,IN_AIASP,NOTES,SMLWNRYEXM,APPRV_DATE,ESTAB_DATE,TOTAL_SIZE,SUBJ_TO_75,GPY_AT_75,AVA,SUP_DIST)
VALUES
('2','AETNA SPRING CELLARS','PROD','2500','2500','','0','2000','20000','TST APPT','0','3','156','0','0','156','1','10','x','','','','','x','1-4-86','1-6-86','2500','','0','Napa Valley','3');
INSERT INTO winery_attributes
(ID,NAME,STATUS,BLDSZ_ORIG,BLDSZ_CURR,HAS_CAVE,CAVESIZE,PROD_ORIG,PROD_CURR,TOUR_TASTG,VISIT_DAY,VISIT_WEEK,VISIT_YR,VISIT_MKTG,VISIT_NMEV,VISIT_ALL,EMPLYEENUM,PARKINGNUM,WDO,LAST_UP,IN_CITYBDY,IN_AIASP,NOTES,SMLWNRYEXM,APPRV_DATE,ESTAB_DATE,TOTAL_SIZE,SUBJ_TO_75,GPY_AT_75,AVA,SUP_DIST)
VALUES
('3','ALTA VINEYARD CELLAR','PROD','480','480','','0','5000','5000','NO','0','4','208','0','0','208','4','6','x','003_U-387879','','','','','2-5-79','1-9-80','480','','0','Diamond Mountain District','3');
INSERT INTO winery_attributes
(ID,NAME,STATUS,BLDSZ_ORIG,BLDSZ_CURR,HAS_CAVE,CAVESIZE,PROD_ORIG,PROD_CURR,TOUR_TASTG,VISIT_DAY,VISIT_WEEK,VISIT_YR,VISIT_MKTG,VISIT_NMEV,VISIT_ALL,EMPLYEENUM,PARKINGNUM,WDO,LAST_UP,IN_CITYBDY,IN_AIASP,NOTES,SMLWNRYEXM,APPRV_DATE,ESTAB_DATE,TOTAL_SIZE,SUBJ_TO_75,GPY_AT_75,AVA,SUP_DIST)
VALUES
('4','BLACK STALLION','PROD','43600','43600','','0','100000','100000','PUB','50','350','18200','0','0','18200','2','45','x','P13-00391','','','','','1-5-80','1-9-85','43600','','0','Oak Knoll District of Napa Valley','3');
INSERT INTO winery_attributes
(ID,NAME,STATUS,BLDSZ_ORIG,BLDSZ_CURR,HAS_CAVE,CAVESIZE,PROD_ORIG,PROD_CURR,TOUR_TASTG,VISIT_DAY,VISIT_WEEK,VISIT_YR,VISIT_MKTG,VISIT_NMEV,VISIT_ALL,EMPLYEENUM,PARKINGNUM,WDO,LAST_UP,IN_CITYBDY,IN_AIASP,NOTES,SMLWNRYEXM,APPRV_DATE,ESTAB_DATE,TOTAL_SIZE,SUBJ_TO_75,GPY_AT_75,AVA,SUP_DIST)
VALUES
('5','ALTAMURA WINERY','PROD','11800','11800','x','3115','50000','50000','APPT','0','20','1040','0','0','1040','2','10','','P13-00206','','','','','1-3-95','28-10-96','14915','x','50000','Napa Valley','4');

The dates in your data set are in the form of a string. Since they are not in the default datestyle (which is YYYY-MM-DD) you should explicitly convert them to a date as follows:
to_date('1-5-80', 'DD-MM-YY')
If you store the data in a timestamp instead, use
to_timestamp('1-5-80', 'DD-MM-YY')
If you are given the data set in the form of the INSERT statements that you show, then first load all the data as simple strings into varchar columns, then add date columns and do an UPDATE (and similarly for integer and boolean columns):
UPDATE my_table
SET estab = to_date(ESTAB_DATE, 'DD-MM-YY'), -- column estab of type date
apprv = to_date(APPRV_DATE, 'DD-MM-YY'), -- etc
...
When the update is done you can ALTER TABLE to drop the text columns with dates (integers, booleans).

Hive - Partition Column Equal to Current Date

I am trying to insert into a Hive table from another table that does not have a column for todays date. The partition I am trying to create is at the date level. What I am trying to do is something like this:
INSERT OVERWRITE TABLE table_2_partition
PARTITION (p_date = from_unixtime(unix_timestamp() - (86400*2) , 'yyyy-MM-dd'))
SELECT * FROM table_1;
But when I run this I get the following error:
"cannot recognize input near 'from_unixtime' '(' 'unix_timestamp' in constant"
If I query a table and make one of the columns that it work just fine. Any idea how to set the partition date to current system date in HiveQL?
Thanks in advance,
Craig

What you want here is Hive dynamic partitioning. This allows the decision for which partition each record is inserted into be determined dynamically as the record is selected. In your case, that decision is based on the date when you run the query.
To use dynamic partitioning your partition clause has the partition field(s) but not the value. The value(s) that maps to the partition field(s) is the value(s) at the end of the SELECT, and in the same order.
When you use dynamic partitions for all partition fields you need to ensure that you are using nonstrict for your dynamic partition mode (hive.exec.dynamic.partition.mode).
In your case, your query would look something like:
SET hive.exec.dynamic.partition.mode=nonstrict;
INSERT OVERWRITE TABLE table_2_partition
PARTITION (p_date)
SELECT
*
, from_unixtime(unix_timestamp() - (86400*2) , 'yyyy-MM-dd')
FROM table_1;

Instead of using unix_timestamp() and from_unixtime() functions, current_date() can used to get current date in 'yyyy-MM-dd' format.
current_date() is added in hive 1.2.0. official documentation
revised query will be :
SET hive.exec.dynamic.partition.mode=nonstrict;
INSERT OVERWRITE TABLE table_2_partition
PARTITION (p_date)
SELECT
*
, current_date()
FROM table_1;

I hope you are running a shell script and then you can store the current date in a variable. Then you create an empty table in Hive using beeline with just partition column. Once done then while inserting the data into partition table you can add that variable as your partition column and data will be inserted.

INSERT INTO SELECT FROM ACCESS TO ORACLE

I have a table in Access named TEST_DATE1 with the column TEST_DATE that is a String datatype and the records look like 20080130. yyyymmdd
I have a table in Oracle named TEST with the column TEST_DATE that is a DATE datatype and I want the records to look like 2008/01/30 yyyy/mm/dd.
I have the two tables linked and when I usually update tables between Access and Oracle I usually do a
INSERT INTO TEST
SELECT *
FROM TEST_DATE1;
How would you convert the string to a DATE using the INSERT INTO SELECT
I have tried
INSERT INTO TEST
(SELECT TO_DATE(TEST_DATE, 'yyyy/mm/dd'))
FROM TEST_DATE1;
Thanks!

To move the records to Oracle, converting a string to a date:
INSERT INTO test (test_date)
SELECT TO_DATE(test_date, 'YYYYMMDD')
FROM test_date1
Once it's in Oracle stored as a date, you can retrieve it in any format you like:
SELECT TO_CHAR(test_date, 'YYYY/MM/DD') as test_date
FROM test;