I have a source table in databricks which has a field called date. I found that the date is like a timestamp. For example, it's data is date = 2021-07-09 00:07 , date = 2021-12-21 09:19.
The above date is not a full timestamp, so I casted it as a string and I got the following results
cast(date as string) = 2021-07-09 00:07:09.529, 2021-12-21 09:19:21.019
I would like to know why casting it to a string is giving the milli seconds and is there any to_timestamp like function to cast the date instead of casting as string.
Can the date casted as string be compared with other dates casted as string?
if the date field has timestamp, then what you see is just visual representation that may omit some non-essential details - internally it's stored in more compact representation, like, long representing number of milliseconds. But when you do cast to string, then this compact representation is used & you get all details.
To cast timestamp type to date type, just do cast(date as date). If you want to convert string into date or timestamp, just use to_date or to_timestamp functions that allow to specify a format string that will be used for parsing of string components. And if you want to convert date or timestamp field into string and control the format of the string, then you need to look onto date_format function.
P.S. You can always refer to the SQL Reference that is the part of official Databricks documentation
Related
I am trying to better understand the date_format function offered by Spark SQL.As per the official databricks documentation (I am using databricks), this function expects any date/ string in a valid datetime format. Below is the link for the same.
I am finding it difficult to understand what is the exact definition of "valid" here. I am trying to understand the functionality through two examples here.
Input string in YYYY-MM-DD format (2021-07-09), for which I get the expected results correctly:
Input string in DD-MM-YYYY format (20-07-2021), and I get null:
Why is this happening? How did this function understand that the parameter that I am passing is indeed in YYYY-MM-DD format? It could also have been YYYY-DD-MM.
My requirement is that I implement a logic that could handle all kinds of valid date formats (MM-DD-YYYY, YYYY-MM-DD, DD-MM-YYYY) and format the dates accordingly.
The following is valid input and output formats for ANSI date/time data types:
Example: ANSIDATE yyyy-mm-dd 2007-02-28 TIME WITH TIME ZONE hh:mm:ss.ffff... [+|-]th:tm
The valid range of time zone offset is from -14:00 to +14:00. date complies with the ANSI SQL standard definition for the Gregorian calendar: "NOTE 85 - Datetime data types will allow dates in the Gregorian format to be stored in the date range 0001-01-01 CE through 9999-12-31 CE
See Databricks SQL datetime patterns for details on valid formats. The function checks that the resulting dates are valid dates in the Proleptic Gregorian calendar, otherwise it returns NULL
When you use "20-07-2021" it does not conform to "yyyy-mm-dd" so results in NULL
Alternately, you can use make_date function which Creates a date from year, month, and day fields. Or better use to_date function
select date_format(to_date('9/15/2021', 'MM/dd/yyyy'), 'yyyy/MM/dd')
See Datetime Patterns for Formatting and Parsing in Spark.
One of the columns my_column in my table is of type varchar where i store value of date in string. The format is 'MM/DD/YY'. I want to use cast function of hibernate to convert this column value to date. But it appears that the cast function accepts string of 'yy-mm-dd' format to convert. Is it possible to use cast for string with specific date format for conversion. Something like this: cast(am.my_column as date format 'mm/dd/yy')
cast function in hibernate is used this way: cast(am.my_column as date).
My use case is basically converting the column value from string to date and making a date comparison. cast(am.my_column as date format 'mm/dd/yy') > '12/30/2017'
Instead of casting a column value inside the query, you may consider to give '12/30/2017' as parameter (with type of Java Date). So that you can directly write am.my_column > :parameterDate
My table in hive has a filed of date in the format of '2016/06/01'. but i find that it is not in harmory with the format of '2016-06-01'.
They can not compare for instance.
Both of them are string .
So I want to know how to make them in harmory and can compare them. Or on the other hand, how to change the '2016/06/01' to '2016-06-01' so that them can compare.
Many thanks.
To convert date string from one format to another you have to use two date function of hive
unix_timestamp(string date, string pattern) convert time string
with given pattern to unix time stamp (in seconds), return 0 if
fail.
from_unixtime(bigint unixtime[, string format]) converts the
number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a
string representing the timestamp of that moment in the current
system time zone.
Using above two function you can achieve your desired result.
The sample input and output can be seen from below image:
The final query is
select from_unixtime(unix_timestamp('2016/06/01','yyyy/MM/dd'),'yyyy-MM-dd') from table1;
where table1 is the table name present in my hive database.
I hope this help you!!!
Let's say you have a column 'birth_day' in your table which is in your format,
you should use the following query to convert birth_day into the required format.
date_Format(birth_day, 'yyyy-MM-dd')
You can use it in a query in the following way
select * from yourtable
where
date_Format(birth_day, 'yyyy-MM-dd') = '2019-04-16';
Use :
unix_timestamp(DATE_COLUMN, string pattern)
The above command would help convert the date to unix timestamp format which you may format as you want using the Simple Date Function.
Date Function
cast(to_date(from_unixtime(unix_timestamp(yourdate , 'MM-dd-yyyy'))) as date)
here is my solution (for string to real Date type):
select to_date(replace('2000/01/01', '/', '-')) as dt ;
ps:to_date() returns Date type, this feature needs Hive 2.1+; before 2.1, it returns String.
ps2: hive to_date() function or date_format() function , or even cast() function, cannot regonise the 'yyyy/MM/dd' or 'yyyymmdd' format, which I think is so sad, and make me a little crazy.
I have a column in my table for storing dates and it is in 12-06-2013 15:32:45. I want to convert it to MM/DD/YYYY format. How can I do it?
Coulmn type is varchar
First you need to CONVERT VARCHAR() to datetime type and and then to CONVERT it to string in desired format:
SELECT CONVERT (varchar (10), CONVERT (date,'12-06-2013 15:32:45' ,103),101)
First 103 is used to interpret current date format, and second - 101 - target format.
If you change target date format from varchar to date then your output in MSMS will be in default display date format of you SQL Server, not the desired format. This is because SQL Server stores dates as integers and converts them before dispalying the value. Therefore if you need to store in certain format, then store in VARCHAR type.
Check out CAST and CONVERT functions on MDSN
or you can do it in your c# code like that :
First you have to take your date from your table and put it in string var "DT" exemple :
string strDT =db.TableTitle.date;
string date = strDT .ToString("MM/DD/YYYY");
DateTime DT = Convert.ToDateTime(date);
then you can use the DT time variable :)
I'm sure this is quite simple, but I've been stuck on it for some time. How can I convert a varchar field (YYYYMM) to a date (MM/01/YY) in SQL?
Thanks.
Edit: I'm using Open Office Base (HSQL), not MySQL; sorry for the confusion.
Try the str_to_date and date_format functions. Something like:
select date_format( str_to_date( my_column, '%Y%c' ), '%c/01/%y' ) from my_table
try :
SELECT STR_TO_DATE(CONCAT(myDate,'01'),'%Y%m%d')
FROM myTable
Use STR_TO_DATE:
From mysql.com:
STR_TO_DATE(str,format)
This is the inverse of the DATE_FORMAT() function. It takes a string str and a format string format. STR_TO_DATE() returns a DATETIME value if the format string contains both date and time parts, or a DATE or TIME value if the string contains only date or time parts.
The date, time, or datetime values contained in str should be given in the format indicated by format. For the specifiers that can be used in format, see the DATE_FORMAT() function description. If str contains an illegal date, time, or datetime value, STR_TO_DATE() returns NULL. Starting from MySQL 5.0.3, an illegal value also produces a warning.
Range checking on the parts of date values is as described in Section 11.3.1, “The DATETIME, DATE, and TIMESTAMP Types”. This means, for example, that “zero” dates or dates with part values of 0 are allowed unless the SQL mode is set to disallow such values.
mysql> SELECT STR_TO_DATE('00/00/0000', '%m/%d/%Y');
-> '0000-00-00'
mysql> SELECT STR_TO_DATE('04/31/2004', '%m/%d/%Y');
-> '2004-04-31'
Get the year:
SUBSTRING(field FROM 2 FOR 2)
Get the month:
SUBSTRING(field FROM -2 FOR 2)
Compose the date:
CONCAT(SUBSTRING(field FROM -2 FOR 2), '/01/', SUBSTRING(field FROM 2 FOR 2))
This will convert from YYYYMM to MM/01/YY.
To be clear: if you're looking for method to convert some value of type Varchar/Text to value of type Date than solutions are:
using CAST function
CAST(LEFT('201205',4)||'-'||SUBSTRING('201205' FROM 5 FOR 6)||'-01' AS DATE)
starting from OpenOffice 3.4 (HSQLDB 2.x) new Oracle-like function TO_DATE supposed to be available
TO_DATE('201205','YYYYMM')
in addition to the written i can mention that you also can construct a string with ANSI/ISO 'YYYY-MM-DD' formatted representation of the date,- Base will acknowledge that and succesfully convert it to the Date type if necessary (e.g. INSERTing in Date typed column etc.)
Here is doc's on HyperSQL and highly recommended OO Base guide by Andrew Pitonyak