I have a date value in this format 'dd-mmm-yyyy'(Example31-Mar-2020) in a glue table. I need to transform this to 'yyyy-mm-dd' (output:2020-03-31) format using sparkSql.
I have tried. "date_format(reference_line_attribute3, 'yyyy-mm-dd')" but this just gives null as output.
Please help.
Thank you
This should do the trick
df.withColumn("newDate",
date_format(
to_date($"reference_line_attribute3", "dd-MMM-yyyy"),
"yyyy-MM-dd"))
Output
+-------------------------+----------+
|reference_line_attribute3| newDate|
+-------------------------+----------+
| 31-Mar-2020|2020-03-31|
+-------------------------+----------+
Related
Edited: Want to convert the format.
I am kinda new in BigQuery, recently I was working on a project. I want to convert above type of format into yyyy/mm/dd format. How will I do that?
You can combine PARSE_DATE with FORMAT_DATE to get the desired output:
SELECT FORMAT_DATE("%Y/%m/%d",PARSE_DATE("%B %d, %Y","June 10, 2014")) AS date_str
PARSE_DATE will parse the provided string into a DATE type value in BQ format (YYYY-MM-DD) and then FORMAT_DATE will format this DATE to a string with the desired output.
In case someone is wondering how to convert the whole column;
FORMAT_DATE("%Y/%m/%d",PARSE_DATE("%B %d, %Y",ColumnName)) as DesiredColumnName
I am trying to change the format of a timestamp in AWS Athena but I am not able to get it correct, would someone please help?
The value (Data format: string (Partitioned)) of the column I am trying to change is
20220826T073200Z
and I would like the output to be
2022-08-26 07:32:00
You need to parse date first, for example with date_parse:
select date_parse('20220826T073200Z', '%Y%m%dT%H%i%sZ');
Output:
_col0
2022-08-26 07:32:00.000
If this is not good enough you can format it with date_format:
select date_format(date_parse('20220826T073200Z', '%Y%m%dT%H%i%sZ'), '%Y-%m-%d %H:%i:%s');
_col0
2022-08-26 07:32:00
in which format should I convert timestamp to receive timestamp value like this 15.08.2017 22:17:41.860000
?
thx
You're close, you just need to Cast it to a string after adding the format:
Cast(Cast(tscol AS FORMAT 'dd.mm.yyyyBhh:mi:ss.s(6)') AS CHAR(26))
Or shorter using
To_Char(tscol,'dd.mm.yyyy hh:mi:ssff')
I have this date format in hive: 20180618151752
Make the following query to leave it in the following way:
select
concat(substr(xdr.time_stop,1,8),' ',
substr(xdr.time_stop,9,2),':',
substr(xdr.time_stop,10,2),':',
substr(xdr.time_stop,11,2)) as date
from padl.fraude_vozm_xdr;
20180618 18:17:52
Now, I need to convert that string field to date, how could I do it?.
Thank you
Use from_unixtime and unix_timestamp.
from_unixtime(unix_timestamp('20180618151752','yyyyMMddHHmmss'),'yyyyMMdd HH:mm:ss')
I'm new to Spark SQL and am trying to convert a string to a timestamp in a spark data frame. I have a string that looks like '2017-08-01T02:26:59.000Z' in a column called time_string
My code to convert this string to timestamp is
CAST (time_string AS Timestamp)
But this gives me a timestamp of 2017-07-31 19:26:59
Why is it changing the time? Is there a way to do this without changing the time?
Thanks for any help!
You could use unix_timestamp function to convert the utc formatted date to timestamp
val df2 = Seq(("a3fac", "2017-08-01T02:26:59.000Z")).toDF("id", "eventTime")
df2.withColumn("eventTime1", unix_timestamp($"eventTime", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'").cast(TimestampType))
Output:
+-------------+---------------------+
|userid |eventTime |
+-------------+---------------------+
|a3fac |2017-08-01 02:26:59.0|
+-------------+---------------------+
Hope this helps!
Solution on Java
There are some Spark SQL functions which let you to play with the date format.
Conversion example : 20181224091530 -> 2018-12-24 09:15:30
Solution (Spark SQL statement) :
SELECT
...
to_timestamp(cast(DECIMAL_DATE as string),'yyyyMMddHHmmss') as `TIME STAMP DATE`,
...
FROM some_table
You can use the SQL statements by using an instance of org.apache.spark.sql.SparkSession. For example if you want to execute an sql statement, Spark provide the following solution:
...
// You have to create an instance of SparkSession
sparkSession.sql(sqlStatement);
...
Notes:
You have to convert the decimal to string and after you can achieve the parsing to timestamp format
You can play with the format the get however format you want...
In spark sql you can use to_timestamp and then format it as your requirement.
select
date_format(to_timestamp(,'yyyy/MM/dd HH:mm:ss'),"yyyy-MM-dd HH:mm:ss") as
from
Here 'timestamp' with value is 2019/02/23 12:00:00 and it is StringType column in 'event' table.
To convert into TimestampType apply to_timestamp(timestamp, 'yyyy/MM/dd HH:mm:ss). It is need to make sure the format for timestamp is same as your column value. Then you apply date_format to convert it as per your requirement.
> select date_format(to_timestamp(timestamp,'yyyy/MM/dd HH:mm:ss'),"yyyy-MM-dd HH:mm:ss") as timeStamp from event