In Impala for some of the timestamp I am getting incorrect data while converting from UTC to PST.When i run same query in hive i am getting correct data but in impala PST conversion is not doing properly for some of the dates.
Impala:
select from_utc_timestamp('2006-08-30 08:00:00','PST')
2006-08-30 01:00:00
Hive:
select from_utc_timestamp('2006-08-30 08:00:00','PST')
2006-08-30 00:00:00
To overcome is there anyother way to convert utc to PST
Use below timezone conversions in Impala.
from_utc_timestamp(timestamp, string timezone)
timezone :
for Eastern Time: America/New_York or EST5EDT
for Central Time: America/Chicago or CST6CDT
for Mountain Time: America/Denver or MST7MDT
Phoenix won't observe daylight saving time: America/Phoenix
for Pacific Time: America/Los_Angeles or PST8PDT
for Alaska Time: America/Anchorage or America/Juneau
for Hawaii Time: America/Adak or EST5EDT
Honolulu won't observe day light saving time: Pacific/Honolulu
select from_utc_timestamp('2017-03-11 13:41:22.084','America/Los_Angeles')
2017-03-11 05:41:22.084000000
select from_utc_timestamp('2017-03-12 13:41:22.084','America/Los_Angeles')
2017-03-12 06:41:22.084000000
select from_utc_timestamp('2017-11-03 13:41:22.084','America/Los_Angeles')
2017-11-03 06:41:22.084000000
select from_utc_timestamp('2017-11-04 13:41:22.084','America/Los_Angeles')
2017-11-04 06:41:22.084000000
select from_utc_timestamp('2017-11-05 13:41:22.084','America/Los_Angeles')
2017-11-05 05:41:22.084000000
**Daylight saving time(2017) ends at 2:00 AM on Sunday, November 5**
select from_utc_timestamp('2017-11-05 08:58:00.000','America/Los_Angeles')
2017-11-05 01:58:00 (UTC -7hrs)
select from_utc_timestamp('2017-11-05 08:59:00.000','America/Los_Angeles')
2017-11-05 01:59:00 (UTC -7hrs)
select from_utc_timestamp('2017-11-05 09:00:00.000','America/Los_Angeles')
2017-11-05 01:00:00 (UTC -8hrs)
select from_utc_timestamp('2017-11-05 09:01:00.000','America/Los_Angeles')
2017-11-05 01:01:00 (UTC -8hrs)
Related
I'm trying to create a timeseries in google bigquery SQL. My data is a series of time ranges covering the period of activity for that record. Here is an example:
Start End
2020-11-01 21:04:00 UTC 2020-11-02 07:15:00 UTC
2020-11-01 21:45:00 UTC 2020-11-02 04:00:00 UTC
2020-11-01 22:00:00 UTC 2020-11-02 09:48:00 UTC
2020-11-01 22:00:00 UTC 2020-11-02 06:00:00 UTC
I wish to create a new table to total the number of active records within a 15 minute block. "21:00:00" would for example be 21:00 to 21:14.59. My desired output for the above would be:
Period Active_Records
2020-11-01 21:00:00 1
2020-11-01 21:15:00 1
2020-11-01 21:30:00 1
2020-11-01 21:45:00 2
2020-11-01 22:00:00 4
2020-11-01 22:15:00 4
etc until the end of the last active range.
I would also like to be able to generate this on the fly by querying a date range and having it return every 15 minute block in the range and how many active records there was in that period.
Any assistance would be greatly appreciated.
Below is for BigQuery Standard SQL
#standardSQL
select ts as period, count(1) as Active_Records
from unnest((
select generate_timestamp_array(timestamp_trunc(min(start), hour), max(`end`), interval 15 minute)
from `project.dataset.table`
)) ts
join `project.dataset.table`
on not (`end` < ts or start > timestamp_add(ts, interval 15 * 60 - 1 second))
group by ts
if to apply to sample data from your question - output is
I have a table and one of the columns is a timestamp. What I would like to do is a SQL query (BigQuery compatible) that rounds the timestamp of each line to the quarter of the hour previous to that time.
Examples:
2019-07-05 21:11:28 UTC -> 2019-07-05 21:00:00 UTC
2019-07-05 21:17:05 UTC -> 2019-07-05 21:15:00 UTC
2019-07-05 20:29:56 UTC -> 2019-07-05 20:15:00 UTC
2019-07-05 21:55:39 UTC -> 2019-07-05 21:45:00 UTC
I found TIMESTAMP_TRUNC that can round to minutes, but this will round to the timestamp's minute, not the quarter.
Do you guys have any idea of how could I do this?
Thanks in advance
Below is for BigQuery Standard SQL
#standardSQL
SELECT ts,
TIMESTAMP_SECONDS(UNIX_SECONDS(ts) - MOD(UNIX_SECONDS(ts), 15 * 60)) ts_rounded_to_quarter_of_hour
FROM `project.dataset.table`
Another, slightly refactored version is
#standardSQL
SELECT ts,
TIMESTAMP_SECONDS(ts_seconds_since_epoch - MOD(ts_seconds_since_epoch, 15 * 60)) ts_rounded_to_quarter_of_hour
FROM `project.dataset.table`, UNNEST([UNIX_SECONDS(ts)]) ts_seconds_since_epoch
And finally, my favorite version would be
#standardSQL
CREATE TEMP FUNCTION TIMESTAMP_TRUNC_TO_QUATER_OF_HOUR(ts TIMESTAMP) AS ((
SELECT TIMESTAMP_SECONDS(ts_seconds_since_epoch - MOD(ts_seconds_since_epoch, 15 * 60))
FROM UNNEST([UNIX_SECONDS(ts)]) ts_seconds_since_epoch
));
SELECT ts,
TIMESTAMP_TRUNC_TO_QUATER_OF_HOUR(ts) AS ts_rounded_to_quarter_of_hour
FROM `project.dataset.table`
You can test, play with above using sample data from your question as in below example
#standardSQL
CREATE TEMP FUNCTION TIMESTAMP_TRUNC_TO_QUATER_OF_HOUR(ts TIMESTAMP) AS ((
SELECT TIMESTAMP_SECONDS(ts_seconds_since_epoch - MOD(ts_seconds_since_epoch, 15 * 60))
FROM UNNEST([UNIX_SECONDS(ts)]) ts_seconds_since_epoch
));
WITH `project.dataset.table` AS (
SELECT TIMESTAMP '2019-07-05 21:11:28 UTC' ts UNION ALL --> 2019-07-05 21:00:00 UTC
SELECT '2019-07-05 21:17:05 UTC' UNION ALL --> 2019-07-05 21:15:00 UTC
SELECT '2019-07-05 20:29:56 UTC' UNION ALL --> 2019-07-05 20:15:00 UTC
SELECT '2019-07-05 21:55:39 UTC' --> 2019-07-05 21:45:00 UTC
)
SELECT ts,
TIMESTAMP_TRUNC_TO_QUATER_OF_HOUR(ts) AS ts_rounded_to_quarter_of_hour
FROM `project.dataset.table`
Obviously, all three above versions return below [same] result
Row ts ts_rounded_to_quarter_of_hour
1 2019-07-05 21:11:28 UTC 2019-07-05 21:00:00 UTC
2 2019-07-05 21:17:05 UTC 2019-07-05 21:15:00 UTC
3 2019-07-05 20:29:56 UTC 2019-07-05 20:15:00 UTC
4 2019-07-05 21:55:39 UTC 2019-07-05 21:45:00 UTC
If you are happy using stored procedures you can do something like this:
declare #d datetime='2019-07-05 21:11:28'
select DATEADD(mi, DATEDIFF(mi, 0, #d)/15*15, 0)
This works because the 0 integer in the example above is actually the start of the epoch (1900-01-01).
You can use timestamp_trunc() and some date arithmetic:
select timestamp_add(timestamp_trunc(current_timestamp, hour),
interval cast(extract(minute from current_timestamp) / 15 as int64)*15 minute
)
I'm trying to convert from UTC time to BST time in Oracle - however, it seems the data change time in oracle is 1 am instead of 2 am. Or what am i missing here? I used following code to illustrate and test the problem:
WITH time1 AS (select cast('2020-03-29 01:00:00 UTC' ASTIMESTAMP WITH TIME ZONE) AS UTC_time FROM dual)
SELECT UTC_time, (UTC_time AT TIME ZONE 'Europe/London') AS bst FROM time1
And at 1:00:00 UTC, the BST time is 2:00:00 - and it should be 1:00:00
Oracle is right, your expectation seems to be wrong. You can see the Europe/London DST changes here.
The IANA time zone database has the following for Europe/London:
# See EU for rules starting in 1996.
...
# Zone NAME GMTOFF RULES FORMAT [UNTIL]
Zone Europe/London -0:01:15 - LMT 1847 Dec 1 0:00s
0:00 GB-Eire %s 1968 Oct 27
1:00 - BST 1971 Oct 31 2:00u
0:00 GB-Eire %s 1996
0:00 EU GMT/BST
so since 1996 the UK has been following the EU rules, which are:
# Europe
# The following rules are for the European Union and for its
# predecessor organization, the European Communities.
# For brevity they are called "EU rules" elsewhere in this file.
# Rule NAME FROM TO TYPE IN ON AT SAVE LETTER/S
Rule EU 1977 1980 - Apr Sun>=1 1:00u 1:00 S
Rule EU 1977 only - Sep lastSun 1:00u 0 -
Rule EU 1978 only - Oct 1 1:00u 0 -
Rule EU 1979 1995 - Sep lastSun 1:00u 0 -
Rule EU 1981 max - Mar lastSun 1:00u 1:00 S
Rule EU 1996 max - Oct lastSun 1:00u 0 -
So... in the UK (and the rest of the EU) DST is applied from the last Sunday in March at 01:00 UTC, which next year is 2020-03-29 01:00:00 UTC. And it stays on DST until the last Sunday in October at 01:00 UTC, which next year is 2020-10-25 01:00:00 UTC.
Oracle is following those rules:
with time1 (utc_time) as (
select timestamp '2020-03-29 00:00:00.000 UTC' FROM dual
union all select timestamp '2020-03-29 00:59:59.999 UTC' FROM dual
union all select timestamp '2020-03-29 01:00:00.000 UTC' FROM dual
union all select timestamp '2020-03-29 01:59:59.999 UTC' FROM dual
union all select timestamp '2020-03-29 02:00:00.000 UTC' FROM dual
--
union all select timestamp '2020-10-25 00:00:00.000 UTC' FROM dual
union all select timestamp '2020-10-25 00:59:59.999 UTC' FROM dual
union all select timestamp '2020-10-25 01:00:00.000 UTC' FROM dual
union all select timestamp '2020-10-25 01:59:59.999 UTC' FROM dual
union all select timestamp '2020-10-25 02:00:00.000 UTC' FROM dual
)
select utc_time,
utc_time at time zone 'Europe/London' as london_time,
to_char(utc_time at time zone 'Europe/London', 'TZD') as "DST?"
from time1
order by utc_time;
UTC_TIME LONDON_TIME DST?
--------------------------------- ------------------------------------------- ------
2020-03-29 00:00:00.000000000 UTC 2020-03-29 00:00:00.000000000 EUROPE/LONDON GMT
2020-03-29 00:59:59.999000000 UTC 2020-03-29 00:59:59.999000000 EUROPE/LONDON GMT
2020-03-29 01:00:00.000000000 UTC 2020-03-29 02:00:00.000000000 EUROPE/LONDON BST
2020-03-29 01:59:59.999000000 UTC 2020-03-29 02:59:59.999000000 EUROPE/LONDON BST
2020-03-29 02:00:00.000000000 UTC 2020-03-29 03:00:00.000000000 EUROPE/LONDON BST
2020-10-25 00:00:00.000000000 UTC 2020-10-25 01:00:00.000000000 EUROPE/LONDON BST
2020-10-25 00:59:59.999000000 UTC 2020-10-25 01:59:59.999000000 EUROPE/LONDON BST
2020-10-25 01:00:00.000000000 UTC 2020-10-25 01:00:00.000000000 EUROPE/LONDON GMT
2020-10-25 01:59:59.999000000 UTC 2020-10-25 01:59:59.999000000 EUROPE/LONDON GMT
2020-10-25 02:00:00.000000000 UTC 2020-10-25 02:00:00.000000000 EUROPE/LONDON GMT
In central Europe DST applies from the same UTC time, but of course the local time is different:
with time1 (utc_time) as (
...
)
select utc_time,
utc_time at time zone 'Europe/Paris' as paris_time,
to_char(utc_time at time zone 'Europe/Paris', 'TZD') as "DST?"
from time1
order by utc_time;
UTC_TIME PARIS_TIME DST?
--------------------------------- ------------------------------------------ ------
2020-03-29 00:00:00.000000000 UTC 2020-03-29 01:00:00.000000000 EUROPE/PARIS CET
2020-03-29 00:59:59.999000000 UTC 2020-03-29 01:59:59.999000000 EUROPE/PARIS CET
2020-03-29 01:00:00.000000000 UTC 2020-03-29 03:00:00.000000000 EUROPE/PARIS CEST
2020-03-29 01:59:59.999000000 UTC 2020-03-29 03:59:59.999000000 EUROPE/PARIS CEST
2020-03-29 02:00:00.000000000 UTC 2020-03-29 04:00:00.000000000 EUROPE/PARIS CEST
2020-10-25 00:00:00.000000000 UTC 2020-10-25 02:00:00.000000000 EUROPE/PARIS CEST
2020-10-25 00:59:59.999000000 UTC 2020-10-25 02:59:59.999000000 EUROPE/PARIS CEST
2020-10-25 01:00:00.000000000 UTC 2020-10-25 02:00:00.000000000 EUROPE/PARIS CET
2020-10-25 01:59:59.999000000 UTC 2020-10-25 02:59:59.999000000 EUROPE/PARIS CET
2020-10-25 02:00:00.000000000 UTC 2020-10-25 03:00:00.000000000 EUROPE/PARIS CET
Possibly that is why you were expecting to not see the time change until 02:00, but if so you're confusing UTC and local time, and/or UK and central Europe.
Not directly relevant to your question, but in my CTE I've switched from casting a string to a timsetamp, to using a timestamp literal. As well as being slightly less typing, the format is unambiguous. When you cast you're relying on the session's NLS settings matching the string format you've supplied, so although the cast works for you, it might not work for someone else running your code. If you don't want to (or can't) use literals then it's safer to use to_timestamp_tz() with an explicit format mask.
I want to convert 10-OCT-17 07.57.14.253290000 AM date format into 2017-10-10 07:57:14
and
10-OCT-17 12.57.14.253290000 PM date format into 2017-10-10 12:57:14
and
10-OCT-17 07.57.14.253290000 PM date format into 2017-10-10 19:57:14
I want AM and PM should come into consideration .
cast(to_timestamp(pub_ts ,'dd-mon-yy HH.MI.SS') as timestamp without time zone)
Where pub_ts is a text. I have used this 15 days back through this would work.
It converts
10-OCT-17 12.57.14.253290000 PM date format into 2017-10-10 00:57:14
I assume you just need a right mask? EG:
postgres=# with e(v) as (values('10-OCT-17 12.57.14.253290000 PM'),('10-OCT-17 12.57.14.253290000 AM'))
select v,to_timestamp(v,'DD-MON-YY HH12.MI.SS.US000 AM')::timestamp(0) from e;
v | to_timestamp
---------------------------------+---------------------
10-OCT-17 12.57.14.253290000 PM | 2017-10-10 12:57:14
10-OCT-17 12.57.14.253290000 AM | 2017-10-10 00:57:14
(2 rows)
I have a datetime field that stores times in UTC format. There's another nchar field that stores the time zone difference based on a location. I'm trying to combine the two for a report so that the time displayed matches the appropriate time zone.
time_stamp | time_zone
---------------------------------
2015-11-24 21:00:00 | -0500
2015-11-23 15:00:00 | -0600
Expected output:
2015-11-24 16:00:00
2015-11-23 09:00:00
I was able to get this to work by using:
extend(time_stamp, year to minute) + (CAST(LEFT(time_zone,3) as int)) units hour
While this technically works for the current situation, I really don't like using the CAST and LEFT functions on the time_zone field since it breaks if the value is not negative. Seems like there's a much better solution, possible something with TO_CHAR. In an informix database, what is the proper way to combine the dateime and nchar fields so that the output time is correct? Ideally I would like to output in non 24 hr format (4:00 PM, etc...) but at this point I'm mainly focused on getting the correct time.
Ideally, your time zone column would be an INTERVAL HOUR TO MINUTE type; you'd then simply add the two columns to get the desired result. Since it is a character type, substringing in some form will be necessary. Using LEFT is one option; SUBSTRING is another; using the Informix subscripting notation is another. The CAST isn't crucial; Informix is pretty good about coercing things.
Unless you actually want only hours and minutes in the result (which is a legitimate choice), your EXTEND operation is unnecessary and undesirable; it means your result won't include the seconds value from your data.
Note that some time zones include minutes values. Newfoundland is on UTC-04:30; India is on UTC+05:30; Nepal is on UTC+05:45. (See World Time Zone for more information.) Getting the minutes accurate is harder because the sign has to be carried through.
As to formatting in AM/PM notation, apart from the question 'why', the answer is to use the TO_CHAR() function and a ghastligram expressing the time format that you want.
TO_CHAR()
GL_DATETIME
GL_DATE
Demonstration:
create table zone_char(time_stamp datetime year to second, time_zone nchar(5));
insert into zone_char values('2015-11-24 21:00:00', '-0500');
insert into zone_char values('2015-11-23 15:00:00', '-0600');
insert into zone_char values('2015-11-22 17:19:21', '+0515');
insert into zone_char values('2015-11-21 02:56:31', '-0430');
Various ways to select the data:
select extend(time_stamp, year to minute) + LEFT(time_zone,3) units hour,
time_stamp + LEFT(time_zone,3) units hour,
time_stamp + time_zone[1,3] units hour,
time_stamp + time_zone[1,3] units hour + (time_zone[1] || time_zone[4,5]) units minute,
TO_CHAR(time_stamp + time_zone[1,3] units hour + (time_zone[1] || time_zone[4,5]) units minute,
'%A %e %B %Y %I.%M.%S %p')
from zone_char;
Sample output:
2015-11-24 16:00 2015-11-24 16:00:00 2015-11-24 16:00:00 2015-11-24 16:00:00 Tuesday 24 November 2015 04.00.00 PM
2015-11-23 09:00 2015-11-23 09:00:00 2015-11-23 09:00:00 2015-11-23 09:00:00 Monday 23 November 2015 09.00.00 AM
2015-11-22 22:19 2015-11-22 22:19:21 2015-11-22 22:19:21 2015-11-22 22:34:21 Sunday 22 November 2015 10.34.21 PM
2015-11-20 22:56 2015-11-20 22:56:31 2015-11-20 22:56:31 2015-11-20 22:26:31 Friday 20 November 2015 10.26.31 PM
And note how much easier it is when the time zone is represented as an INTERVAL HOUR TO MINUTE:
alter table zone_char add hhmm interval hour to minute;
update zone_char set hhmm = time_zone[1,3] || ':' || time_zone[4,5];
SELECT:
select time_stamp, hhmm, extend(time_stamp + hhmm, year to minute),
time_stamp + hhmm,
TO_CHAR(time_stamp + hhmm, '%A %e %B %Y %I.%M.%S %p')
from zone_char;
Result:
2015-11-24 21:00:00 -5:00 2015-11-24 16:00 2015-11-24 16:00:00 Tuesday 24 November 2015 04.00.00 PM
2015-11-23 15:00:00 -6:00 2015-11-23 09:00 2015-11-23 09:00:00 Monday 23 November 2015 09.00.00 AM
2015-11-22 17:19:21 5:15 2015-11-22 22:34 2015-11-22 22:34:21 Sunday 22 November 2015 10.34.21 PM
2015-11-21 02:56:31 -4:30 2015-11-20 22:26 2015-11-20 22:26:31 Friday 20 November 2015 10.26.31 PM