Oracle DST Time Conversion Error ORA-01878 - sql

i have a select statement where i am converting timezones
Select
from_tz(cast(DATE_TIME as timestamp), 'US/Eastern') at time zone 'UTC' DATE_TIME_UTC
From Table1
but for some rows i am getting error due to DST
ORA-01878: specified field not found in datetime or interval
i want to write a query like
select
if error then do something else do the time conversion from table1

As you're on 12c you can use the enhanced subquery factoring that provides to define a local function; that can attempt the conversion with US/Eastern, and fall back to -4:00 if that fails.
Using your sample data and a couple of extra rows that will convert anyway:
alter session set nls_date_format = 'YYYY-MM-DD HH24:MI:SS';
alter session set nls_timestamp_tz_format = 'YYYY-MM-DD HH24:MI:SS TZR TZD';
with
function get_tstz(p_date in date) return timestamp with time zone is
dst_exception exception;
pragma exception_init(dst_exception, -1878);
begin
return from_tz(cast(p_date as timestamp), 'US/Eastern');
exception
when dst_exception then
return from_tz(cast(p_date as timestamp), '-04:00');
end get_tstz;
select date_time,
get_tstz(date_time) as date_time_converted,
get_tstz(date_time) at time zone 'UTC' as date_time_utc
from table1
/
DATE_TIME DATE_TIME_CONVERTED DATE_TIME_UTC
------------------- ---------------------------------- ---------------------------
2018-03-11 01:59:00 2018-03-11 01:59:00 US/EASTERN EST 2018-03-11 06:59:00 UTC UTC
2018-03-11 02:06:00 2018-03-11 02:06:00 -04:00 -04:00 2018-03-11 06:06:00 UTC UTC
2018-03-11 02:08:00 2018-03-11 02:08:00 -04:00 -04:00 2018-03-11 06:08:00 UTC UTC
2018-03-11 02:21:00 2018-03-11 02:21:00 -04:00 -04:00 2018-03-11 06:21:00 UTC UTC
2018-03-11 02:48:00 2018-03-11 02:48:00 -04:00 -04:00 2018-03-11 06:48:00 UTC UTC
2018-03-11 02:06:00 2018-03-11 02:06:00 -04:00 -04:00 2018-03-11 06:06:00 UTC UTC
2018-03-11 02:33:00 2018-03-11 02:33:00 -04:00 -04:00 2018-03-11 06:33:00 UTC UTC
2018-03-11 03:00:00 2018-03-11 03:00:00 US/EASTERN EDT 2018-03-11 07:00:00 UTC UTC
I've adjusted my NLS settings so you can see the difference in the converted values, as either EST, EDT or a fixed -4:00.
As mentioend in comments, you're ignoring the underlying data issues, and it would be better to correct the data that you know is wrong - assuming you can be sure why it is wrong and therefore how it is safe to fix; or to confirm your assertion that the original data is all supposed to be US/Eastern.
Fundamentally, as some are clearly not really US/Eastern, it doesn't seem safe to trust any of the data. Without knowing how and why those specifc records have values you don't expect, you can't be sure that any other values are what you expect either. Whatever application, tool or process inserted those dates may have (and probably did) insert other times which look OK but are also not actually US/Eastern. The rest may all convert without error, but that doesn't mean the UTC times are necessarily representative.
You also have a secondary problem in that you don't know whether a date you have recorded as 2017-11-05 01:00:00 was originally 01:00 EST or 01:00 EDT, as that hour was repeated when summertime ended. Oracle will just choose for you though.

You can create a custom function and checks if its a valid timestamp with time zone and use that function in the where clause of your query as follows for example.
create table t(x varchar(100));
insert into t
select '21-FEB-2009 18:00:00'
from dual
union all
select '31-FEB-2009 18:00:00' /*Junk date here..*/
from dual;
create or replace function fn_test(dt in varchar2)
return int
as
l_timestamp timestamp with time zone;
begin
l_timestamp :=from_tz(to_timestamp(dt,'DD-MON-YYYY hh24:mi:ss'), 'US/Eastern') at time zone 'UTC';
return 1;
exception
when others then
return null;
end;
/
select from_tz(to_timestamp(x,'DD-MON-YYYY hh24:mi:ss'),'US/Eastern') at time zone 'UTC'
from t
where fn_test(x) is not null

I got this error message for one of these two reasons:
Most likely: you are trying to convert a local time that does not exist for reasons of DST switch, e.g. '28-MAR-21 02:34' does not exist in Germany (timezone Berlin/Europe), because clocks jump from 1:59 AM to 3:00 AM during night. Solution: add one hour, UTC will then reflect the correct time.
Less likely: typo in timezone string: "Europe/ Berlin" or "Berlin/Europe" - both is wrong, correct is "Europe/Berlin" (pattern is Continent/City)

Related

Repeating time (1 AM-1:59:59 AM) on first Sunday of November - Daylight Savings ending day - Oracle

Good afternoon.
I understand that there is "each second of each minute of one specific hour" is repeated (1 AM - 1:59:59 AM) on first Sunday of November (Closing day of Daylight Saving time). So, duration from 0:00 AM (midnight) to 3 AM is 4 hours on that day.
SELECT TO_TIMESTAMP_TZ('2021-11-07 03:00:00 US/Mountain',
'yyyy-mm-dd hh24:mi:ss TZR')
- TO_TIMESTAMP_TZ('2021-11-07 00:00:00 US/Mountain', 'yyyy-mm-dd hh24:mi:ss TZR') FROM DUAL;
The above query is returning 4 hours as expected.
Here is my question -
I want to basically differentiate/represent the two occurrences of 1 AM (or any time between 1 AM, and 1:59:59 AM). How can I do? (I am using Oracle 12.1)
BTW, this following query is resulting 1 hour 30 minutes, so '2021-11-07 01:30:00 MST' represents the second instance of 1:30 AM. In the same manner, I was expecting '2021-11-07 01:30:00 MDT' to be the first instance, however it is resulting ORA-01882: timezone region not found. BTW, I prefer to have US/Mountain (or something like that) for region, rather MST vs. MDT
SELECT TO_TIMESTAMP_TZ('2021-11-07 03:00:00 MST', 'yyyy-mm-dd
hh24:mi:ss TZR')
- TO_TIMESTAMP_TZ('2021-11-07 01:30:00 MST', 'yyyy-mm-dd hh24:mi:ss TZR') FROM DUAL;
Sorry if I confused you. Please let me know if any questions.
Any help?
Thank you
Viswa
Added later:
I think I found the answer: We need to use TZD flag, and use MST/MDT values.
I did not like that as I prefer to use the region (such as US/Mountain).
So any enhancements will be appreciated.
SELECT TO_TIMESTAMP_TZ('2021-11-07 03:00:00 US/Mountain',
'yyyy-mm-dd hh24:mi:ss TZR')
- TO_TIMESTAMP_TZ('2021-11-07 00:00:00 US/Mountain', 'yyyy-mm-dd hh24:mi:ss TZR') FROM DUAL;
-- 4 hours:00 minutes, as expected
SELECT TO_TIMESTAMP_TZ('2021-11-07 03:00:00 MST', 'yyyy-mm-dd
hh24:mi:ss TZD')
- TO_TIMESTAMP_TZ('2021-11-07 01:30:00 MDT', 'yyyy-mm-dd hh24:mi:ss TZD') FROM DUAL;
-- 2:30 minutes
-- So any values ranging from 1:00 to 1:59:59 with a time zone of MDT are the first instance values.
SELECT TO_TIMESTAMP_TZ('2021-11-07 03:00:00 MST', 'yyyy-mm-dd
hh24:mi:ss TZD')
- TO_TIMESTAMP_TZ('2021-11-07 01:30:00 MST', 'yyyy-mm-dd hh24:mi:ss TZD') FROM DUAL;
-- 1 hour:30 minutes
-- So any values ranging from 1:00 to 1:59:59 with a time zone of MST are the second instance values.
Value TIMESTAMP '2021-11-07 01:00:00 US/Mountain' is ambiguous, it could be
2021-11-07 01:00:00-06:00 or 2021-11-07 01:00:00-07:00
If you don't specify the daylight-saving-time status then Oracle defaults to the standard time, (MST in your case)
You need to provide both, the timezone region and the Daylight Saving Time information, i.e. TO_TIMESTAMP_TZ('2021-11-07 01:00:00 US/Mountain MST', 'yyyy-mm-dd hh24:mi:ss TZR TZD') or TO_TIMESTAMP_TZ('2021-11-07 01:00:00 US/Mountain MDT', 'yyyy-mm-dd hh24:mi:ss TZR TZD')
Note, if you alter your session with ALTER SESSION SET ERROR_ON_OVERLAP_TIME = TRUE; then for ambiguous times like TIMESTAMP '2021-11-07 01:00:00 US/Mountain' Oracle does not default to standard time but raise an error:
ORA-01883: overlap was disabled during a region transition
Don't mistake "Time zone region" (TRZ) with "Daylight saving information" (TZD), however MST can mean both:
SELECT *
FROM V$TIMEZONE_NAMES
WHERE TZABBREV in ('MST', 'MDT');
Calculations of TIMESTAMP WITH TIME ZONE values are internally always performed on UTC times.
See Support for Daylight Saving Time

parse_timestamp vs format_timestamp bigquery

Could someone help me understand why these two queries are returning different results in bigquery?
select FORMAT_TIMESTAMP('%F %H:%M:%E*S', "2018-10-01 00:00:00" , 'Europe/London')
returns 2018-10-01 01:00:00
select PARSE_TIMESTAMP('%F %H:%M:%E*S', "2018-10-0100:00:00", "Europe/London")
returns 2018-09-30 23:00:00 UTC
As 2018-10-01 is during british summer time (UTC +1), I would've expected both queries to return 2018-09-30 23:00:00 UTC
The first is given a timestamp which is in UTC. It then converts it to the corresponding time in Europe/London. The return value is a string representing the time in the local timezone.
The second takes a string representation and returns a UTC timestamp. The representation is assumed to be in Europe/London.
So, the two functions are going in different directions, one from UTC to the local time and the other from the local time to UTC.

Oracle Spool file via CMD command deliver more Data than expected

i have a Oracle Table "Sales" with columns ID,Sales,TIMESTAMP. Data looks like this:
ID Sales TimeStamp
1 30 2018-08-20 00:00:00.989900 +02:00
1 35 2018-08-21 05:00:00.989900 +02:00
...
1 35 2018-08-27 05:00:00.989900 +02:00
i created a Talend Job to execute a SQL Spool file in CMD mode to export a Query into csv. The Spoolfile look like this:
alter session set NLS_TIMESTAMP_TZ_FORMAT ='YYYY-MM-DD HH24:mi:ss.ff6 TZH:TZM';
alter session set NLS_TIMESTAMP_FORMAT ='YYYY-MM-DD HH24:mi:ss.ff6';
alter session set NLS_DATE_FORMAT ='YYYY-MM-DD';
alter session set NLS_NUMERIC_CHARACTERS ='.,';
spool C:/test.csv
SET ECHO OFF
SET ...
SELECT * FROM Sales where timestamp< to_timestamp('2018-08-25 00:00:00.0000000','YYYY-MM-DD HH24:mi:ss:ff66 TZH:TZM')
when the TalendJob runs the Query on CMD mode, it gives me more data than expected with the Data to '2018-08-25 01:00:00'.
when i execute the SQL Query on Oracle Server manually, it gives correct Data to '2018-08-25 00:00:00'
==> Query on CMD on Talend give 1 hours of Data more than expected.
i don't really understand why that Problem happens.
My assumption is the Problem Timestamp in the Query "'2018-08-25 00:00:00.0000000'". this Timestamp has no time zone. but i am not sure.
can you please help me with this Problem?
Thankyou.
The manual query and the Talend query seem to be running in sessions with different time zones.
You aren't specifying a time zone in your fixed value, despite having TZH:TZM in the format model; and in fact you can't with to_timestamp():
select to_timestamp('2018-08-25 00:00:00.0000000 +02:00','YYYY-MM-DD HH24:mi:ss:ff6 TZH:TZM')
from dual;
ORA-01821: date format not recognized
because that function gives you a plain timestamp:
alter session set NLS_TIMESTAMP_FORMAT ='YYYY-MM-DD HH24:mi:ss.ff6';
alter session set NLS_TIMESTAMP_TZ_FORMAT ='YYYY-MM-DD HH24:mi:ss.ff6 TZH:TZM';
select to_timestamp('2018-08-25 00:00:00.0000000','YYYY-MM-DD HH24:mi:ss:ff6 TZH:TZM')
as plain_timestamp
from dual;
PLAIN_TIMESTAMP
--------------------------
2018-08-25 00:00:00.000000
When you use that plain timestamp in a comparison with your table column, which is a timestamp with time zone, there is an implicit conversion into the session time zone. You can see the effect that has by manually setting it:
alter session set time_zone = 'Europe/London';
select cast(
to_timestamp('2018-08-25 00:00:00.0000000','YYYY-MM-DD HH24:mi:ss:ff6 TZH:TZM')
as timestamp with time zone
) as timestamp_with_session_zone
from dual;
TIMESTAMP_WITH_SESSION_ZONE
---------------------------------
2018-08-25 00:00:00.000000 +01:00
alter session set time_zone = 'America/New_York';
select cast(
to_timestamp('2018-08-25 00:00:00.0000000','YYYY-MM-DD HH24:mi:ss:ff6 TZH:TZM')
as timestamp with time zone
) as timestamp_with_session_zone
from dual;
TIMESTAMP_WITH_SESSION_ZONE
---------------------------------
2018-08-25 00:00:00.000000 -04:00
So, to be getting different data from your two sessions, that comparison is using a different value, therefore the session time zones must be different.
The simple fix is to specify the time zone explicitly in your fixed value, but you need a different function to avoid the error seen earlier; and preferably with a region instead of an offset to allow for daylight savings (assuming the values in your table are region-based too):
select to_timestamp_tz('2018-08-25 00:00:00.0000000 Europe/Berlin','YYYY-MM-DD HH24:mi:ss:ff6 TZR')
as timestamp_with_berlin_zone
from dual;
TIMESTAMP_WITH_BERLIN_ZONE
---------------------------------
2018-08-25 00:00:00.000000 +02:00
or you could use a timestamp literal:
select timestamp '2018-08-25 00:00:00.0 Europe/Berlin' as timestamp_with_berlin_zone
from dual;
which gets the same value.
i haved tried to format the time zone in the Query with to_timestamp_tz(substr('2018-08-25 00:00:00.0000000'),1,25), 'YYYY-MM-DD HH24:mi:ss.ff6 TZH:TZM' at time zone 'berlin/europe') as input_timestamp but it stills gives me more data than expected.
Ignoring the odd substr() which just strips the last two zeros off what is already a fixed string, if you do:
select to_timestamp_tz('2018-08-25 00:00:00.0000000', 'YYYY-MM-DD HH24:mi:ss.ff6 TZH:TZM')
at time zone 'Europe/Berlin' as timestamp_with_wrong_time
from dual;
you get (with my session still on New York time for greater effect)
TIMESTAMP_WITH_WRONG_TIME
---------------------------------
2018-08-25 06:00:00.000000 +02:00
The time zone is now what you expected, but the time is wrong. You have much the same problem as before. You're still converting the fixed value with no time zone supplied into a timestamp with time zone, so it's implicitly using the session time zone:
select to_timestamp_tz('2018-08-25 00:00:00.0000000', 'YYYY-MM-DD HH24:mi:ss.ff6 TZH:TZM')
as timestamp_with_wrong_time
from dual;
TIMESTAMP_WITH_WRONG_TIME
---------------------------------
2018-08-25 00:00:00.000000 -04:00
and then the at timezone 'Europe/Berlin' just gives that exact same point in universal time - midnight in New York, which is 04:00 UTC - but in Berlin local time, which is 06:00. It's the same point of time, just viewed from different places/time zone.
Again, you just need to specify the time zone for the fixed time you're using for the comparison - as timestamp '2018-08-25 00:00:00.0 Europe/Berlin'.

How to deal with timestamps without a timezone?

The NYC bike and taxi datasets list the time when events happened in local time. Timestamps like 2018-01-07 10:30:00 means it was 10am in NY at the time.
When I ingest these timestamps into BigQuery, BigQuery assumes they are GMT - appending the incorrect timezone information.
How can I fix this?
2 choices:
Use DATETIME instead of TIMESTAMP - DATETIME has the same information than TIMESTAMP, except no timezone information is added.
Since this is NY, you can append the US/Eastern timezone when ingesting - it will correctly identify summer daylight saving changes and so on
For example:
SELECT TIMESTAMP('2018-3-10 10:00:00', 'US/Eastern')
, TIMESTAMP('2018-5-10 10:00:00', 'US/Eastern')
2018-03-10 15:00:00 UTC
2018-05-10 14:00:00 UTC

pytz - convert a datetime in the future to UTC

I have a file that contains forecasted events for the next two weeks. There is a datetime column which has the date and each 30 minute interval, and a time zone column.
I am using pytz to convert the different time zones (around 30+ unique ones) to UTC before loading them into a database. However, for the forecast file I am receiving an error:
NonExistentTimeError: 2016-10-16 00:00:00
Is there a way to go about this?
date interval time_zone
10/26/2016 22:30 US/Central
10/26/2016 22:30 US/Eastern
10/26/2016 23:00 America/Bogota
10/26/2016 23:00 Asia/Calcutta
Current code:
for tz in df['time_zone'].unique():
df.loc[df['time_zone'] == tz, 'datetime_utc'] = df.loc[df['time_zone'] == tz, 'datetime'].dt.tz_localize(tz).dt.tz_convert('UTC')
df['datetime_utc'] = df['datetime_utc'].dt.tz_localize(None)
Due to changes in daylight saving happening on the 16th October, 2016-10-16 00:00:00 really is a local time that does not exist for Brazil (It should instead read 2016-10-16 01:00:00)