I have a file that contains forecasted events for the next two weeks. There is a datetime column which has the date and each 30 minute interval, and a time zone column.
I am using pytz to convert the different time zones (around 30+ unique ones) to UTC before loading them into a database. However, for the forecast file I am receiving an error:
NonExistentTimeError: 2016-10-16 00:00:00
Is there a way to go about this?
date interval time_zone
10/26/2016 22:30 US/Central
10/26/2016 22:30 US/Eastern
10/26/2016 23:00 America/Bogota
10/26/2016 23:00 Asia/Calcutta
Current code:
for tz in df['time_zone'].unique():
df.loc[df['time_zone'] == tz, 'datetime_utc'] = df.loc[df['time_zone'] == tz, 'datetime'].dt.tz_localize(tz).dt.tz_convert('UTC')
df['datetime_utc'] = df['datetime_utc'].dt.tz_localize(None)
Due to changes in daylight saving happening on the 16th October, 2016-10-16 00:00:00 really is a local time that does not exist for Brazil (It should instead read 2016-10-16 01:00:00)
Related
Could someone help me understand why these two queries are returning different results in bigquery?
select FORMAT_TIMESTAMP('%F %H:%M:%E*S', "2018-10-01 00:00:00" , 'Europe/London')
returns 2018-10-01 01:00:00
select PARSE_TIMESTAMP('%F %H:%M:%E*S', "2018-10-0100:00:00", "Europe/London")
returns 2018-09-30 23:00:00 UTC
As 2018-10-01 is during british summer time (UTC +1), I would've expected both queries to return 2018-09-30 23:00:00 UTC
The first is given a timestamp which is in UTC. It then converts it to the corresponding time in Europe/London. The return value is a string representing the time in the local timezone.
The second takes a string representation and returns a UTC timestamp. The representation is assumed to be in Europe/London.
So, the two functions are going in different directions, one from UTC to the local time and the other from the local time to UTC.
i have a select statement where i am converting timezones
Select
from_tz(cast(DATE_TIME as timestamp), 'US/Eastern') at time zone 'UTC' DATE_TIME_UTC
From Table1
but for some rows i am getting error due to DST
ORA-01878: specified field not found in datetime or interval
i want to write a query like
select
if error then do something else do the time conversion from table1
As you're on 12c you can use the enhanced subquery factoring that provides to define a local function; that can attempt the conversion with US/Eastern, and fall back to -4:00 if that fails.
Using your sample data and a couple of extra rows that will convert anyway:
alter session set nls_date_format = 'YYYY-MM-DD HH24:MI:SS';
alter session set nls_timestamp_tz_format = 'YYYY-MM-DD HH24:MI:SS TZR TZD';
with
function get_tstz(p_date in date) return timestamp with time zone is
dst_exception exception;
pragma exception_init(dst_exception, -1878);
begin
return from_tz(cast(p_date as timestamp), 'US/Eastern');
exception
when dst_exception then
return from_tz(cast(p_date as timestamp), '-04:00');
end get_tstz;
select date_time,
get_tstz(date_time) as date_time_converted,
get_tstz(date_time) at time zone 'UTC' as date_time_utc
from table1
/
DATE_TIME DATE_TIME_CONVERTED DATE_TIME_UTC
------------------- ---------------------------------- ---------------------------
2018-03-11 01:59:00 2018-03-11 01:59:00 US/EASTERN EST 2018-03-11 06:59:00 UTC UTC
2018-03-11 02:06:00 2018-03-11 02:06:00 -04:00 -04:00 2018-03-11 06:06:00 UTC UTC
2018-03-11 02:08:00 2018-03-11 02:08:00 -04:00 -04:00 2018-03-11 06:08:00 UTC UTC
2018-03-11 02:21:00 2018-03-11 02:21:00 -04:00 -04:00 2018-03-11 06:21:00 UTC UTC
2018-03-11 02:48:00 2018-03-11 02:48:00 -04:00 -04:00 2018-03-11 06:48:00 UTC UTC
2018-03-11 02:06:00 2018-03-11 02:06:00 -04:00 -04:00 2018-03-11 06:06:00 UTC UTC
2018-03-11 02:33:00 2018-03-11 02:33:00 -04:00 -04:00 2018-03-11 06:33:00 UTC UTC
2018-03-11 03:00:00 2018-03-11 03:00:00 US/EASTERN EDT 2018-03-11 07:00:00 UTC UTC
I've adjusted my NLS settings so you can see the difference in the converted values, as either EST, EDT or a fixed -4:00.
As mentioend in comments, you're ignoring the underlying data issues, and it would be better to correct the data that you know is wrong - assuming you can be sure why it is wrong and therefore how it is safe to fix; or to confirm your assertion that the original data is all supposed to be US/Eastern.
Fundamentally, as some are clearly not really US/Eastern, it doesn't seem safe to trust any of the data. Without knowing how and why those specifc records have values you don't expect, you can't be sure that any other values are what you expect either. Whatever application, tool or process inserted those dates may have (and probably did) insert other times which look OK but are also not actually US/Eastern. The rest may all convert without error, but that doesn't mean the UTC times are necessarily representative.
You also have a secondary problem in that you don't know whether a date you have recorded as 2017-11-05 01:00:00 was originally 01:00 EST or 01:00 EDT, as that hour was repeated when summertime ended. Oracle will just choose for you though.
You can create a custom function and checks if its a valid timestamp with time zone and use that function in the where clause of your query as follows for example.
create table t(x varchar(100));
insert into t
select '21-FEB-2009 18:00:00'
from dual
union all
select '31-FEB-2009 18:00:00' /*Junk date here..*/
from dual;
create or replace function fn_test(dt in varchar2)
return int
as
l_timestamp timestamp with time zone;
begin
l_timestamp :=from_tz(to_timestamp(dt,'DD-MON-YYYY hh24:mi:ss'), 'US/Eastern') at time zone 'UTC';
return 1;
exception
when others then
return null;
end;
/
select from_tz(to_timestamp(x,'DD-MON-YYYY hh24:mi:ss'),'US/Eastern') at time zone 'UTC'
from t
where fn_test(x) is not null
I got this error message for one of these two reasons:
Most likely: you are trying to convert a local time that does not exist for reasons of DST switch, e.g. '28-MAR-21 02:34' does not exist in Germany (timezone Berlin/Europe), because clocks jump from 1:59 AM to 3:00 AM during night. Solution: add one hour, UTC will then reflect the correct time.
Less likely: typo in timezone string: "Europe/ Berlin" or "Berlin/Europe" - both is wrong, correct is "Europe/Berlin" (pattern is Continent/City)
The NYC bike and taxi datasets list the time when events happened in local time. Timestamps like 2018-01-07 10:30:00 means it was 10am in NY at the time.
When I ingest these timestamps into BigQuery, BigQuery assumes they are GMT - appending the incorrect timezone information.
How can I fix this?
2 choices:
Use DATETIME instead of TIMESTAMP - DATETIME has the same information than TIMESTAMP, except no timezone information is added.
Since this is NY, you can append the US/Eastern timezone when ingesting - it will correctly identify summer daylight saving changes and so on
For example:
SELECT TIMESTAMP('2018-3-10 10:00:00', 'US/Eastern')
, TIMESTAMP('2018-5-10 10:00:00', 'US/Eastern')
2018-03-10 15:00:00 UTC
2018-05-10 14:00:00 UTC
I use postgres for the rails app and I have a unix timestamp in postgresql db. I have a requirement to select and group by the dd-mm-yyyy and by month name.
Consider I have the following unix timestamp
1425148200
and I would need to change this to datetime and I used to_timestamp which returned
2015-02-28 18:30:00 UTC
and I tried to convert the datetime to local timezone using
::timestamp without time zone AT TIME ZONE 'IST'
but that did not give time in required timezone and instead it returned
2015-02-28 16:30:00 UTC
and I tried to get the date part using ::date which returned
Sat, 28 Feb 2015
So please help me get the dd-mm-yyyy in specified timezone and month name(March) from the unix timestamp.
Thanks in Advance!
select to_char(to_timestamp('1425148200')::timestamptz at time zone 'UTC-5:30','DD-MM-YYYY & of course Month')
01-03-2015 & of course March
It is postgres mistake I guess
according to http://www.postgresql.org/docs/7.2/static/timezones.html
I want historic date convert UTC 0 to UTC local in SQL. Like;
2012-11-23
2013-01-08
2014-02-23
But we have 2 different time zone. We use UTC +2 after last sunday in March and use UTC +3 after last sunday October. I need solution immediately guys. Please help me...
Try this:
SELECT CONVERT_TZ('your date ','+your time zone','+time zone you want');
SELECT CONVERT_TZ('2004-01-01 12:00:00','+02:00' ,'+03:00'); // in your case, from +2 to +3
See this link
If you need a dynamic thing, you'll need the timezone of each history (maybe you can store it in a separeted column), so I think you can do something like this:
SELECT CONVERT_TZ('your_date_column','local_timezone' ,'time_zone_column');