I am updating my dataframe field Create Date using CURRENT_TIMESTAMP() time function of Spark SQL. The value being returned are in the UTC timezone.
I want the values in Eastern Time (with daylight savings handled). I would like to do the conversion of timezone when I am passing my SQL query to sqlContext.sql. How can I achieve this?
Thanks.
you can use from_utc_timestamp fucntion to do this.
import org.apache.spark.sql.types.TimestampType
// This sample syntax. You can replace your value at literal location.
from_utc_timestamp(lit("2018-12-01 00:00:00").cast(TimestampType), "EST5EDT")
This must handle Daylight savings too.
Related
A table with terabytes of data in bigquery got multiple columns set as string format but actually they contain datetime strings like
2016-10-24 15:00:00
I tried answer from this link to convert (CAST) the fields into timestamp format as below
SELECT
CAST( MURDER_DATE AS TIMESTAMP) AS CONVERTED_MURDER_DATE, *
FROM `death_list`;
That works but it converts all strings into timestamps with UTC timezone as below
2007-03-23 15:00:00.000 UTC
I need the data in a different timezone. Any clue?
Try using
DATETIME(CAST( MURDER_DATE AS TIMESTAMP), "Australia/Sydney"))
To my view, it seems to be a current limitation of BigQuery:
Timestamp type is always stored in UTC format. And you have no way to add any "timezone" information to it.
Datetime type nor stores any information about the timezone. You could still have a internal convention in your team/company that says that all the Datetime columns are stored in your local timezone, but I personally find it very awkward.
What we've decided so far in our company is to store everything in Timestamp (thus UTC format), and we never use Datetime due to lack of precision regarding the time zone. Then, if a client wants to get the information in another timezone, it has to do the conversion itself when reading the data.
Is there any way to get only "-04" or the difference value based on whatever time zone is added in a query?
select convert(datetime,GETDATE()) at time zone 'Eastern Standard Time'
Result: 2017-09-12 17:49:18.377 -04:00
if you are using sql server you can do this one of two ways depending on the data you want
datepart(tz,SYSDATETIMEOFFSET()) or datename(tz,SYSDATETIMEOFFSET()).
if you take a look at these functions you will notice they require offsets. GETDATE() does not return the correct type.
you will have to use SYSDATETIMEOFFSET() instead.
for more information on this please read this
I am trying to convert the date in SQL based on the parameter value in my Java code. However when the below query is executed I am getting error . Request you to help me in fixing this query.
SELECT TO_DATE ('2015-08-26T05:46:30.488+0100',
'YYYY-MM-DD"T"hh24:mi:ss.sTZH:TZM')
FROM DUAL
*
Error at line 2
ORA-01821: date format not recognized
Date and Time format info:
http://www.w3.org/TR/NOTE-datetime
You have two issues: TO_DATE doesn't recognise any time zone components or fractional seconds, you'll have to convert it to timestamp with time zone, and .s isn't how you represent fractional seconds anyway, you need .ff. The valid format models are shown in the documentation.
Putting those together you can do:
SELECT TO_TIMESTAMP_TZ ('2015-08-26T05:46:30.488+0100',
'YYYY-MM-DD"T"hh24:mi:ss.ffTZHTZM')
FROM DUAL;
TO_TIMESTAMP_TZ('2015-08-26T05:46:30.488+0100','YYYY-MM-DD"T"HH24:MI:SS.FFTZHTZ
-------------------------------------------------------------------------------
26-AUG-15 05.46.30.488000000 +01:00
If you really want it as a date you'll need to decide what to do with the time zone information - either assume it's local time (essentially ignore it), or convert to UTC or some other time zone. You may really want to keep it as a timestamp with time zone though.
Well, the error message is pretty specific. Oracle does not recognize the given date format YYYY-MM-DD"T"hh24:mi:ss.sTZH:TZM.
You can refer to this page in order to build a proper date format: https://www.techonthenet.com/oracle/functions/to_date.php
I know that to insert a datetime this format yyyy-mm-dd hh:mm:ss should be used.
However my dataset has a timestamp field that looks like yyyy-mm-dd hh:mm:ss +/-0X:00, where X can take many values and is different from my computer's local timezone.
What is the best way to insert a datetime field with such timezone information to SQLite?
SQLite's built-in date and time functions understand the timezone specification in these strings, but with different timezones, any other operations on these strings (even searches and comparisons) will not work correctly.
If you want to handle time zones, you have to either
convert all timestamps to one specific time zone (UTC), so that you can use them for sorting and searching; or
leave the time zone information in there, but do all searching, sorting and other computations not in SQL but in your application.
I'm trying to convert the datetimeOffset stored in SQL Azure to UTC DateTime so that I can perform a proper calculation.
The DateTimeOffset is stored as local NewZealand Time as:
2011-05-31 21:40:00.0000000 +10:00
I need to convert the above values to UTC so that I can perform date time calculation. Is there any way to achieve this using TSQL and Azure SSRS?
Thanks heaps.
Here's another thread that may give you some ideas: TSQL: How to convert local time to UTC? (SQL Server 2008)
Off the top of my head, if you know the datetime you are working with is from New Zealand, you know what the UTC offset is. It should be based on one of two values (if they have daylight savings).
So do a dateadd with a function call that determines if you are in daylight savings or not. How do you determine if you are in daylight savings for a given date? Good question. Unless there is a .NET function you could turn into a CLR function, I would just create a look up table and a function that tests against that. The function either returns a boolean or the proper offset for New Zealand.
I would save the new derived value into a new column, but I'm not sure what your needs are.