I looked on the documentation for google big query data types, checking the differences between TimeStamp to Datetime data types.
As I understand the main difference is:
Unlike Timestamps, a DATETIME object does not refer to an absolute instance in time. Instead, it is the civil time, or the time that a user would see on a watch or calendar.
So when should I use Timestamp/Datetime?
Thanks
In most cases you will want to use the timestamp data type. It refers to an absolute point in time. BigQuery interprets any timezone information and represents the time internally as a UTC timestamp.
Very rarely would you use a datetime data type, which is a date and a time but no time zone. The example I like to give is that you'd use a datetime to represent pi day, 2017, since it occurs at 2017-03-14 15:09:26.535898 in each time zone separately.
Related
I want to update a date with a timezone (+2 hours) but it ends up as UTC (0 hours)
Date type is 'timestamp-with-timezone'
Query...
update table set date = '2022-05-25 13:28+02:00'
will end up as this in the database.
2022-05-25 11:28:00+00
What's wrong here?
tl;dr
Nothing wrong. Postgres stores values of TIMESTAMP WITH TIME ZONE in UTC, always an offset from UTC of zero. Any submitted offset or zone is used to adjust to UTC.
Details
Date type is 'timestamp-with-timezone'
No such type in standard SQL, nor in Postgres.
I’ll assume you meant TIMESTAMP WITH TIME ZONE.
it ends up as UTC (0 hours)
Read the fine manual. You are seeing documented behavior.
Postgres always stores values in a column of type TIMESTAMP WITH TIME ZONE in UTC, that is, with an offset of zero hours-minutes-seconds.
Any time zone or offset provided with an input is used to adjust into UTC. That provided zone or offset is then discarded.
So the name of the type TIMESTAMP WITH TIME ZONE is a misnomer. First, the authors of the SQL were thinking in terms of offset, not real time zones. Second, any submitted time zone is not stored. A submitted zone is used to adjust and then discarded.
If you need to track the original offset or zone, add an extra column. You’ll have to add code to store the offset amount or the time zone name.
update table set date = '2022-05-25 13:28+02:00' will end up as this in the database. 2022-05-25 11:28:00+00 What's wrong here?
Nothing is wrong. That is a feature, not a bug. Both of those strings represent the very same simultaneous moment.
FYI, database engines vary widely in their behavior handling date-time types and behaviors.
Some do as Postgres does regarding TIMESTAMP WITH TIME ZONE, adjusting to UTC and then discarding any provided time zone or offset. Some others may not.
The SQL standard barely touches on the topic of date-time handling. It declares a few types, and does that poorly with incomplete coverage of all cases. And the standard neglects to define behavior.
So, be very careful when it comes to date-time handling in your database work. Read very carefully the documentation for your particular database engine. Do not make assumptions. Run experiments to validate your understanding. And know that writing portable SQL code for date-time may not be feasible.
I'm struggling to deal with time zone and daylight saving when querying SAP HANA. The datetime stamp is in the form of NVARCHAR, eg 20210304132500 YYYYMMDDHHMISS in UTC, which means local time is 14:25:00(GMT +01:00) but my query returns 13:25:00 (UTC). How do I edit my results to match local time? Sample query below if that helps.
SELECT DATE_TIME,LOCATION,PART_NUMB
FROM "PUBLIC"."internal.sap.datamodel::ACTIVITY"
WHERE SUBSTRING(DATE_TIME,9,2) IN ('08','11')`
The desired result is local date_time in any format.
HANA comes with timezone conversion functions (UTCTOLOCAL) that can perform the necessary calculations.
These functions require that the data/time input is in either SQL date/time format or that it can be converted to that. They also require that the timezone data has been set up and maintained in the HANA DB. This is the actual information about which timezone has which offsets and daylight saving begin and end times.
For your example, it may make sense to expose the DATE_TIME as a type converted field DATE_TIME_UTC that is already in sql-date time:
to_seconddate (DATE_TIME, 'YYYYMMDDHHMISS') as DATE_TIME_UTC
With this conversion done, you can convert the timezone like this:
UTCTOLOCAL (DATE_TIME_UTC, 'Berlin', 'platform') as LOCAL_DATE_TIME
Note, that the target time zone name may be something like "GMT+1" but this is really just a name and not a calculation instruction. If "GMT+1" is not found in the list of timezone conversions, HANA won't just add an hour - it won't perform the calculation.
With this data type and timezone conversion done, you could have a WHERE clause like this:
WHERE
HOUR(LOCAL_DATE_TIME) IN (8, 11)
This order of transformations (data type -> time zone -> hour component) is of course rather expensive. It may be worthwhile to check whether the resulting query performance is satisfactory on realistic data volume.
Also important to note is that time zone conversion only works on complete date-time information, not just the time. That is to say, if the date is unknown, it cannot be determined which offset rule between two time zone applies. So, simply separating the hours and date components won't help in this case.
Finally, I've written quite a bit about handling date, time, and time zones in HANA, you may want to have a look at that:
The time is now, isn’t it?
Trouble with time?
You got the time?
I have a TIMESTAMP field in an hsqldb table that I want to set to "2015-02-11 16:02:01.488 America/Los_Angeles", but the insert fails even if I set the column to TIMESTAMP WITH TIMEZONE, the reason being hsqldb seems to support '2008-08-08 20:08:08-8:00' format but not spelled out like America/Los_Angeles. Is there way to make the insert accept America/Los_Angeles type zones ?
Sorry, but hsqldb doesn't support working with IANA/Olson time zones directly. You are correct that TIMESTAMP WITH TIMEZONE only supports a time zone offset. You can review the hsqldb docs for confirmation.
Many databases do not support named time zone. Oracle and Postgres support them, but most others do not.
Consider also that while a named time zone can usually determine the offset, there are still cases of ambiguity around the fall-back daylight saving time transition. In other words, if you had "2015-11-01 01:30:00 America/Los_Angeles", you could not deterministically tell whether it was Pacific Daylight Time (UTC-07:00) or Pacific Standard Time (UTC-08:00). This is why usually just the offset is stored.
The converse is also true though. If you only store "-08:00" then you can't deterministically know that it came from "America/Los_Angeles".
Here's a general guideline that will help:
If the local time is unimportant, then just store a TIMESTAMP based on UTC.
If the local time is important, but the value will never be modified, then store a TIMESTAMP WITH TIMEZONE, using the local time and it's associated time zone offset.
If the local time is important AND the value can be modified, then store a TIMESTAMP WITH TIMEZONE in one column, and the time zone name (ie. "America/Los_Angeles") in a second VARCHAR column, or elsewhere in your database. During an edit operation, use the time zone name to calculate the offset of the new value. It might be the same, or it may be different.
See also DateTime vs DateTimeOffset, which presents a similar argument for .Net and/or SQL Server.
Q1: I want to convert a unix timestamp (INT) to monetdb timestamp ('YYYY-MM-DD HH:MM:SS') format
but it is giving me the GMT time not my actual time.
When I do
select (epoch(cast(current_timestamp as timestamp))-epoch(timestamp '2013-04-25 11:49:00'))
where 2013-04-25 11:49:00 is my systems current time it gives the same difference
I tried using
set time zone interval '05:30' HOUR TO MINUTE;
but it did not change the result
How can I solve this problem??
Example Problem:
I wanted to convert unix timestamp 1366869289 which should be around "2013-04-25 11:25:00" but monetdb gives "2013-04-25 05:55:00"
Knowing nothing about MonetDB, but a lot about timezones, I decided to look in their documentation to see what kind of datatypes are supported and how conversions are handled.
I found this page on Temporal data types. Based on that, I can conclude that a timestamp in MonetDB is always intended to reference UTC/GMT time - which is consistent with other systems.
In order to get a value that is for a particular time zone, they offer the following example:
SET TIME ZONE INTERVAL '1' HOUR TO MINUTE
I assume this means to set the database to offset all times by 1 hour, effectively placing the values all in UTC+01:00, such as is the offset for British Summer Time.
The page also goes on to point out the problems that can arise with using just and offset to adjust time values (see TimeZone != Offset in the TimeZone tag wiki). It also offers a list of various named time zones. But it does not show how to set a time zone to one of the named values. Also, their list appears to be proprietary, and incomplete. While at first glance they appear to have similarities to the IANA/Olson time zone database - the identifiers they specify are not valid TZDB names.
There are some other functions listed on this page, without much explanation. One that looks promising for your needs is LOCALTIMESTAMP. Perhaps this will take the local time zone into account, which appears to be what you were looking for.
I could not find any additional details specific to MonetDB date/time/timezone handling. The documentation appears to be fairly incomplete. You might want to reach out to their mailing list.
I usually have an "interceptor" that right before reading/writing from/to the database does DateTime conversion (from UTC to local time, and from local time to UTC), so I can use DateTime.Now (derivations and comparisions) throughout the system without worrying about time zones.
Regarding serialization and moving data between computers, there is no need to bother, as the datetime is always UTC.
Should I continue storing my dates (SQL 2008 - datetime) in UTC format or should I instead store it using DateTimeOffset (SQL 2008 - datetimeoffset)?
UTC Dates in the database (datetime type) have been working and known for so long, why change it? What are the advantages?
I have already looked into articles like this one, but I'm not 100% convinced though. Any thoughts?
There is one huge difference, where you cannot use UTC alone.
If you have a scenario like this
One server and several clients (all geographically in different timezones)
Clients create some data with datetime information
Clients store it all on central server
Then:
datetimeoffset stores Local time of the client and ALSO offset to the UTC time
all clients know UTC time of all data and also a local time in the place where the information originated
But:
UTC datetime stores just UTC datetime, so you do not have information about local time in the client location where data originated
Other clients do not know the local time of the place, where datetime information came from
Other clients can only calculate their local time from the database (using UTC time) not the local time of the client, where the data originated
Simple example is flight ticket reservation system ... Flight ticket should contain 2 times:
- "take off" time (in timezone of "From" city)
- "landing" time (in timezone of "Destination" city)
You are absolutely correct to use UTC for all historical times (i.e. recording events happened). It is always possible to go from UTC to local time but not always the other way about.
When to use local time? Answer this question:
If the government suddenly decide to change daylight savings, would you like this
data to change with it?
Only store local time if the answer is "yes". Obviously that will only be for future dates, and usually only for dates that affect people in some way.
Why store a time zone/offset?
Firstly, if you want to record what the offset was for the user who carried out the action, you would probably be best just doing that, i.e. at login record the location and timezone for that user.
Secondly if you want to convert for display, you need to have a table of all local time offset transitions for that timezone, simply knowing the current offset is not enough, because if you are showing a date/time from six months ago the offset will be different.
A DATETIMEOFFSET gives you the ability to store local time and UTC time in one field.
This allows for very simple and efficient reporting in local or UTC time without the need to process the data for display in any way.
These are the two most common requirements - local time for local reports and UTC time for group reports.
The local time is stored in the DATETIME portion of the DATETIMEOFFSET and the OFFSET from UTC is stored in the OFFSET portion, thus conversion is simple and, since it requires no knowledge of the timezone the data came from, can all be done at database level.
If you don't require times down to milliseconds, e.g. just to minutes or seconds, you can use DATETIMEOFFSET(0). The DATETIMEOFFSET field will then only require 8 bytes of storage - the same as a DATETIME.
Using a DATETIMEOFFSET rather than a UTC DATETIME therefore gives more flexibility, efficiency and simplicity for reporting.