My company is supporting the following applications for an US client:
IBM Mainframe
, DB2
, Unix
, Teradata
, Oracle
, SQL Server 2005
, Lawson
Since Daylight Savings will start by next month, I would like to know your general thoughts about what all we should be careful about so that we can avoid any bad incidents?
Thanks,
Visakh
Daylight saving can be a bit tricky, some thoughts:
The day on daylight saving has either 23 or 25 hours, not 24 as usual
There is either a gap between 02:00h and 03:00h or these times appear twice
You have to always use timezones when you get data from outside, if not, you will get problems. E.g. you can not know if the sender has already changed it's daylight saving setting or not because you can't reliably know if everything works correctly on the device and in addition, most users do not change the timezone(or daylight saving flag), they change the time itself and that will run you in serious trouble. Mobile phone do sometimes not change the daylight setting themselves, the users change simply change the time from 02:00h to 03:00h both CET! Thats obviously wrong since the time itself does not change, it's the timezone! It is still 02:00h CET but 03:00h CEST.
You should probably check the products/libraries you use if they work correctly with timezone-changes.
I tested our python (with mx.DateTime) environment, if it correctly knows the timezone to be used:
test program:
from mx.DateTime import *
def f(dt):
return dt.Format("%Y-%m-%d %H:%M %Z")
base = ISO.ParseDateTime('2009-03-29 00:00')
for i in range(10):
test = base + i*(30*oneMinute)
diff = test.ticks()-base.ticks()
print "Seconds between %s and %s: %d(%d) diff=%d" % (f(base),
f(test), diff,
i*1800, i*1800-diff)
output for me (living in Germany, March 29th is daylight saving day):
$ python test.py
Seconds between 2009-03-29 00:00 CET and 2009-03-29 00:00 CET: 0(0) diff=0
Seconds between 2009-03-29 00:00 CET and 2009-03-29 00:30 CET: 1800(1800) diff=0
Seconds between 2009-03-29 00:00 CET and 2009-03-29 01:00 CET: 3600(3600) diff=0
Seconds between 2009-03-29 00:00 CET and 2009-03-29 01:30 CET: 5400(5400) diff=0
Seconds between 2009-03-29 00:00 CET and 2009-03-29 02:00 CEST: 7200(7200) diff=0
Seconds between 2009-03-29 00:00 CET and 2009-03-29 02:30 CEST: 9000(9000) diff=0
Seconds between 2009-03-29 00:00 CET and 2009-03-29 03:00 CEST: 7200(10800) diff=3600
Seconds between 2009-03-29 00:00 CET and 2009-03-29 03:30 CEST: 9000(12600) diff=3600
Seconds between 2009-03-29 00:00 CET and 2009-03-29 04:00 CEST: 10800(14400) diff=3600
Seconds between 2009-03-29 00:00 CET and 2009-03-29 04:30 CEST: 12600(16200) diff=3600
As you can see with mx.DateTime that works correctly, but there are a lot of libraries and products where that fails.
Related
i have a select statement where i am converting timezones
Select
from_tz(cast(DATE_TIME as timestamp), 'US/Eastern') at time zone 'UTC' DATE_TIME_UTC
From Table1
but for some rows i am getting error due to DST
ORA-01878: specified field not found in datetime or interval
i want to write a query like
select
if error then do something else do the time conversion from table1
As you're on 12c you can use the enhanced subquery factoring that provides to define a local function; that can attempt the conversion with US/Eastern, and fall back to -4:00 if that fails.
Using your sample data and a couple of extra rows that will convert anyway:
alter session set nls_date_format = 'YYYY-MM-DD HH24:MI:SS';
alter session set nls_timestamp_tz_format = 'YYYY-MM-DD HH24:MI:SS TZR TZD';
with
function get_tstz(p_date in date) return timestamp with time zone is
dst_exception exception;
pragma exception_init(dst_exception, -1878);
begin
return from_tz(cast(p_date as timestamp), 'US/Eastern');
exception
when dst_exception then
return from_tz(cast(p_date as timestamp), '-04:00');
end get_tstz;
select date_time,
get_tstz(date_time) as date_time_converted,
get_tstz(date_time) at time zone 'UTC' as date_time_utc
from table1
/
DATE_TIME DATE_TIME_CONVERTED DATE_TIME_UTC
------------------- ---------------------------------- ---------------------------
2018-03-11 01:59:00 2018-03-11 01:59:00 US/EASTERN EST 2018-03-11 06:59:00 UTC UTC
2018-03-11 02:06:00 2018-03-11 02:06:00 -04:00 -04:00 2018-03-11 06:06:00 UTC UTC
2018-03-11 02:08:00 2018-03-11 02:08:00 -04:00 -04:00 2018-03-11 06:08:00 UTC UTC
2018-03-11 02:21:00 2018-03-11 02:21:00 -04:00 -04:00 2018-03-11 06:21:00 UTC UTC
2018-03-11 02:48:00 2018-03-11 02:48:00 -04:00 -04:00 2018-03-11 06:48:00 UTC UTC
2018-03-11 02:06:00 2018-03-11 02:06:00 -04:00 -04:00 2018-03-11 06:06:00 UTC UTC
2018-03-11 02:33:00 2018-03-11 02:33:00 -04:00 -04:00 2018-03-11 06:33:00 UTC UTC
2018-03-11 03:00:00 2018-03-11 03:00:00 US/EASTERN EDT 2018-03-11 07:00:00 UTC UTC
I've adjusted my NLS settings so you can see the difference in the converted values, as either EST, EDT or a fixed -4:00.
As mentioend in comments, you're ignoring the underlying data issues, and it would be better to correct the data that you know is wrong - assuming you can be sure why it is wrong and therefore how it is safe to fix; or to confirm your assertion that the original data is all supposed to be US/Eastern.
Fundamentally, as some are clearly not really US/Eastern, it doesn't seem safe to trust any of the data. Without knowing how and why those specifc records have values you don't expect, you can't be sure that any other values are what you expect either. Whatever application, tool or process inserted those dates may have (and probably did) insert other times which look OK but are also not actually US/Eastern. The rest may all convert without error, but that doesn't mean the UTC times are necessarily representative.
You also have a secondary problem in that you don't know whether a date you have recorded as 2017-11-05 01:00:00 was originally 01:00 EST or 01:00 EDT, as that hour was repeated when summertime ended. Oracle will just choose for you though.
You can create a custom function and checks if its a valid timestamp with time zone and use that function in the where clause of your query as follows for example.
create table t(x varchar(100));
insert into t
select '21-FEB-2009 18:00:00'
from dual
union all
select '31-FEB-2009 18:00:00' /*Junk date here..*/
from dual;
create or replace function fn_test(dt in varchar2)
return int
as
l_timestamp timestamp with time zone;
begin
l_timestamp :=from_tz(to_timestamp(dt,'DD-MON-YYYY hh24:mi:ss'), 'US/Eastern') at time zone 'UTC';
return 1;
exception
when others then
return null;
end;
/
select from_tz(to_timestamp(x,'DD-MON-YYYY hh24:mi:ss'),'US/Eastern') at time zone 'UTC'
from t
where fn_test(x) is not null
I got this error message for one of these two reasons:
Most likely: you are trying to convert a local time that does not exist for reasons of DST switch, e.g. '28-MAR-21 02:34' does not exist in Germany (timezone Berlin/Europe), because clocks jump from 1:59 AM to 3:00 AM during night. Solution: add one hour, UTC will then reflect the correct time.
Less likely: typo in timezone string: "Europe/ Berlin" or "Berlin/Europe" - both is wrong, correct is "Europe/Berlin" (pattern is Continent/City)
The NYC bike and taxi datasets list the time when events happened in local time. Timestamps like 2018-01-07 10:30:00 means it was 10am in NY at the time.
When I ingest these timestamps into BigQuery, BigQuery assumes they are GMT - appending the incorrect timezone information.
How can I fix this?
2 choices:
Use DATETIME instead of TIMESTAMP - DATETIME has the same information than TIMESTAMP, except no timezone information is added.
Since this is NY, you can append the US/Eastern timezone when ingesting - it will correctly identify summer daylight saving changes and so on
For example:
SELECT TIMESTAMP('2018-3-10 10:00:00', 'US/Eastern')
, TIMESTAMP('2018-5-10 10:00:00', 'US/Eastern')
2018-03-10 15:00:00 UTC
2018-05-10 14:00:00 UTC
I have an issue trying to get the exact GPS time. Looks like the issue is related to command mktime used in the code.
To get the correct GPS time I have added -2 hrs in UTC.
But the purpose is to have a code to calculate the GPS time automatically and don't use manually changes at UTC-2 as an example
Input file GMT time ( > 3 hrs in local time )
18/05/21 08:18:29
18/05/21 08:20:25
18/05/21 08:21:02
When I run the code below, got 1 hour less :
gawk -F'[/: ]' -v ts=$(date -d'01/06/1980' +%s) \
-v lap=18 '{$1="" d=sprintf(20$0);
print mktime(d)+lap-ts }' file
result output GPS TIME ( < 1 hr )
1210922327
1210922443
1210922480
When I run the code below, got exactly the correct GPS time.
gawk -F'[/: ]' -v ts=$(TZ=UTC-2 date -d'1/6/1980 0:00' +%s) \
-v lap=18 '{$1="" d=sprintf(20$0);
print mktime(d)+lap-ts }' file
Result output GPS TIME ( exactly time: OUTPUT DESIRED)
1210925927
1210926043
1210926080
Thanks in advance
What you are encountering is mostly related to the time-zone settings of your system. Due to Daylight Saving Time, your time-zone shifted one hour which causes the discrepancy. A very nice post of detecting DST, can be found here. Copying its examples, we can show that for the time-zone TZ=Europe/Stockholm, the time-zone changes to summer time depending on the date :
$ TZ=Europe/Stockholm date +%Z # CET or CEST depending of when its run
$ TZ=Europe/Stockholm date --date=20170101 +%Z # CET
$ TZ=Europe/Stockholm date --date=20170601 +%Z # CEST
$ TZ=CET date --date=20170101 +%Z # CET
$ TZ=CET date --date=20170601 +%Z # CEST, note that its auto adjusted to CEST
and therefore it will for sure have an effect on the epoch time which is given from 1970-01-01T00:00:00 UTC. With zdump we see when DST comes into affect :
$ zdump -v /usr/share/zoneinfo/Europe/Stockholm | grep 2018
/usr/share/zoneinfo/Europe/Stockholm Sun Mar 25 00:59:59 2018 UTC = Sun Mar 25 01:59:59 2018 CET isdst=0 gmtoff=3600
/usr/share/zoneinfo/Europe/Stockholm Sun Mar 25 01:00:00 2018 UTC = Sun Mar 25 03:00:00 2018 CEST isdst=1 gmtoff=7200
/usr/share/zoneinfo/Europe/Stockholm Sun Oct 28 00:59:59 2018 UTC = Sun Oct 28 02:59:59 2018 CEST isdst=1 gmtoff=7200
/usr/share/zoneinfo/Europe/Stockholm Sun Oct 28 01:00:00 2018 UTC = Sun Oct 28 02:00:00 2018 CET isdst=0 gmtoff=3600
and this is seen in the epoch time as :
$ TZ=Europe/Stockholm date -d "2018-03-25 01:59:59" +%s
1521939599
$ TZ=Europe/Stockholm date -d "2018-03-25 03:00:00" +%s
1521939600
$ TZ=Europe/Stockholm date -d "2018-03-25 02:00:00" +%s
date: invalid date ‘2018-03-25 02:00:00’
As you see, for TZ=Europe/Stockholm, the time 2018-03-25T02:00:00 does not exist and the other two are only 1 sec apart.
Summary, what does this all mean: it essentially means that your system automatically compensates for DST, unless your TZ is UTC. And this play a role on all date related command such as systime(), date or even Awk's mktime().
Can we avoid DST compensation with awk:
Since the OP wants the GPS-time, i.e. the total seconds since 1980-01-06T00:00:00, you essentially subtract two times. So if both are computed in the same TZ without DST correction, you always get the correct result. There are 2 ways to do this :
execute your command in a specific timezone:
By forcing the system to work in a single TZ (such as UTC, UTC+2, ...), there will be no DST correction. For the OP's question, the TZ of interest is UTC.
$ TZ="UTC" awk 'BEGIN { ts = mktime("1980 01 06 00 00 00") }
{ datespec="20"$0; gsub(/[/:]/," ",datespec);
print mktime(datespec) + lap - ts
}' lap=18 file
or from awk 4.20 onwards you can tell mktime() to assume that the date is in UTC by using the UTC flag. (mktime(datespec [, utc-flag ]))
$ awk 'BEGIN { ts = mktime("1980 01 06 00 00 00",1) }
{ datespec="20"$0; gsub(/[/:]/," ",datespec);
print mktime(datespec,1) + lap - ts
}' lap=18 file
both result in the following output.
1210925927
1210926043
1210926080
In both cases, you don't need to worry about your system-timezone and all the mumbo-jumbo related to daylight savings time.
disable DST correction with mktime: When adding a DST entry to the datespec part of mktime, you can tell the system to alwyas work in DST or not or let the system figure it out by himself. The latter is what you do not want. datespec is a string of the form YYYY MM DD HH MM SS [DST]. And this brings it down too:
$ awk 'BEGIN { ts = mktime("1980 01 06 00 00 00 0") }
{ datespec="20"$0; gsub(/[/:]/," ",datespec);
print mktime(datespec" 0") + lap - ts
}' lap=18 file
Documentation for mktime awk 4.2.0 onwards:
mktime(datespec [, utc-flag ])
Turn datespec into a timestamp in the same form as is returned by systime(). It is similar to the function of the same name in ISO C. The argument, datespec, is a string of the form "YYYY MM DD HH MM SS [DST]". The string consists of six or seven numbers representing, respectively, the full year including century, the month from 1 to 12, the day of the month from 1 to 31, the hour of the day from 0 to 23, the minute from 0 to 59, the second from 0 to 60,55 and an optional daylight-savings flag.
The values of these numbers need not be within the ranges specified; for example, an hour of -1 means 1 hour before midnight. The origin-zero Gregorian calendar is assumed, with year 0 preceding year 1 and year -1 preceding year 0. If utc-flag is present and is either non-zero or non-null, the time is assumed to be in the UTC time zone; otherwise, the time is assumed to be in the local time zone. If the DST daylight-savings flag is positive, the time is assumed to be daylight savings time; if zero, the time is assumed to be standard time; and if negative (the default), mktime() attempts to determine whether daylight savings time is in effect for the specified time.
If datespec does not contain enough elements or if the resulting time is out of range, mktime() returns -1.
I have a file that contains forecasted events for the next two weeks. There is a datetime column which has the date and each 30 minute interval, and a time zone column.
I am using pytz to convert the different time zones (around 30+ unique ones) to UTC before loading them into a database. However, for the forecast file I am receiving an error:
NonExistentTimeError: 2016-10-16 00:00:00
Is there a way to go about this?
date interval time_zone
10/26/2016 22:30 US/Central
10/26/2016 22:30 US/Eastern
10/26/2016 23:00 America/Bogota
10/26/2016 23:00 Asia/Calcutta
Current code:
for tz in df['time_zone'].unique():
df.loc[df['time_zone'] == tz, 'datetime_utc'] = df.loc[df['time_zone'] == tz, 'datetime'].dt.tz_localize(tz).dt.tz_convert('UTC')
df['datetime_utc'] = df['datetime_utc'].dt.tz_localize(None)
Due to changes in daylight saving happening on the 16th October, 2016-10-16 00:00:00 really is a local time that does not exist for Brazil (It should instead read 2016-10-16 01:00:00)
I've been working on parsing strings and I have a test case that has been causing problems for me. When parsing a date/time string with strptime, Daylight Savings Time is NOT accounted for. This is a bug as far as I can tell. I can't find any docs on this bug. Here is a test case in the Rails console. This is ruby 1.9.3-p215 and Rails 3.2.2.
1.9.3-p125 :049 > dt = DateTime.strptime("2012-04-15 10:00 Central Time (US & Canada)", "%Y-%m-%d %H:%M %Z")
=> Sun, 15 Apr 2012 10:00:00 -0600
1.9.3-p125 :050 > dt = DateTime.strptime("2012-04-15 10:00 Central Time (US & Canada)", "%Y-%m-%d %H:%M %Z").utc
=> Sun, 15 Apr 2012 16:00:00 +0000
1.9.3-p125 :051 > dt = DateTime.strptime("2012-04-15 10:00 Central Time (US & Canada)", "%Y-%m-%d %H:%M %Z").utc.in_time_zone("Central Time (US & Canada)")
=> Sun, 15 Apr 2012 11:00:00 CDT -05:00
As you can see, I have to convert to utc and then back to the timezone to get DST to be properly interpreted, but then the time is shifted one hour as well, so it's not what I parsed out of the string. Does someone have a workaround to this bug or a more robust way of parsing a date + time + timezone reliably into a DateTime object where daylight savings time is properly represented? Thank you.
Edit:
Ok, I found a workaround, although I'm not sure how robust it is.
Here is an example:
ActiveSupport::TimeZone["Central Time (US & Canada)"].parse "2012-04-15 10:00"
This parses the date/time string into the correct timezone. I'm not sure how robust the parse method is for handling this so I'd like to see if there is a better workaround, but this is my method so far.
This is a frustrating problem. The Rails method you're looking for is Time.zone.parse. First use DateTime.strptime to parse the string, then run it through Time.zone.parse to set the zone. Check out the following console output:
> Time.zone
=> (GMT-06:00) Central Time (US & Canada)
> input_string = "10/12/12 00:00"
> input_format = "%m/%d/%y %H:%M"
> date_with_wrong_zone = DateTime.strptime(input_string, input_format)
=> Fri, 12 Oct 2012 00:00:00 +0000
> correct_date = Time.zone.parse(date_with_wrong_zone.strftime('%Y-%m-%d %H:%M:%S'))
=> Fri, 12 Oct 2012 00:00:00 CDT -05:00
Notice that even though Time.zone's offset is -6 (CST), the end result's offset is -5 (CDT).
Ok, here is the best way I've found to handle this so far. I created a utility method in a lib file.
# Returns a DateTime object in the specified timezone
def self.parse_to_date(date_string, num_hours, timezone)
if timezone.is_a? String
timezone = ActiveSupport::TimeZone[timezone]
end
result = nil
#Chronic.time_class = timezone # Trying out chronic time zone support - so far it doesn't work
the_date = Chronic.parse date_string
if the_date
# Format the date into something that TimeZone can definitely parse
date_string = the_date.strftime("%Y-%m-%d")
result = timezone.parse(date_string) + num_hours.to_f.hours
end
result
end
Note that I add hours onto the time manually because Chronic.parse wasn't as robust as I liked in parsing times - it failed when no trailing zeros were added to a time, for example, as in 8:0 instead of 8:00.
I hope this is useful to someone. Parsing date/time/timzone strings into a valid date seems to be a very common thing, but I was unable to find any parsing code that incorporated all three together.