We are using a Vertica database with table columns of type timestamptz, all data is inserted according to the UTC timezone.
We are using spring-jdbc's NamedParameterJdbcTemplate
All queries are based on full calendar days, e.g. start date 2013/08/01 and end date 2013/08/31, which brings everything between '2013/08/01 00:00:00.0000' and '2013/08/31 23:59:59.9999'
We are trying to modify our queries to consider timezones, i.e. I can for my local timezone I can ask for '2013/08/01 00:00:00.0000 Asia/Jerusalem' till '2013/08/31 23:59:59.9999 Asia/Jerusalem', which is obviously different then '2013/08/01 00:00:00.0000 UTC' till '2013/08/31 23:59:59.9999 UTC'.
So far, I cannot find a way to do so, I tried setting the timezone in the session:
set timezone to 'Asia/Jerusalem';
This doesn't even work in my database client.
Calculating the difference in our Java code will not work for us as we also have queries returning date groupings (this will get completely messed up).
Any ideas or recommendations?
I am not familiar with Veritca, but some general advice:
It is usually best to use half-open intervals for date range queries. The start date should be inclusive, while the end date should be exclusive. In other words:
start <= date < end
or
start <= date && end > date
Your end date wouldn't be '2013/08/31 23:59:59.9999', it would instead be the start of the next day, or '2013/09/01 00:00:00.0000'. This avoids problems relating to precision of decimals.
That example is for finding a single date. Since you are querying a range of dates, then you have two inputs. So it would be:
startFieldInDatabase >= yourStartParameter
AND
endFieldInDatabase < yourEndParameter
Again, you would first increment the end parameter value to the start of the next day.
It sounds like perhaps Vertica is TZ aware, given that you talked about timestamptz types in your answer. Assuming they are similar to Oracle's TIMESTAMPTZ type, then it sounds like your solution will work just fine.
But usually, if you are storing times in UTC in your database, then you would simply convert the query input time(s) in advance. So rather than querying between '2013/08/01 00:00:00.0000' and '2013/09/01 00:00:00.0000', you would convert that ahead of time and query between '2013/07/31 21:00:00.0000' and '2013/08/31 21:00:00.0000'. There are numerous posts already on how to do that conversion in Java either natively or with Joda Time, so I won't repeat that here.
As a side note, you should make sure that whatever TZDB implementation you are using (Vertica's, Java's, or JodaTime's) has the latest 2013d update, since that includes the change for Israel's daylight saving time rule that goes into effect this year.
Okay, so apparently:
set time zone to 'Asia/Jerusalem';
worked and I just didn't realize it, but for the sake of helping others I'm going to add something else that works:
select fiels at time zone 'Asia/Jerusalem' from my_table;
will work for timestamptz fields
Related
I'm trying to calculate difference between two dates in postgres and found out that on several cases my tests fail, while debugging I found an interesting thing - when I subtract one date from another it seems to lack one hour, here's the script (table has only one timestamp field):
select now(), d1, now() - d1, extract(day from date_trunc('day', now() - d1))
from test;
And here's the result:
This seemed strange, so I decided to check it with some other service and got the result I expected (23 hrs instead of 22):
(see https://www.timeanddate.com/date/durationresult.html?d1=2&m1=3&y1=2019&d2=1&m2=4&y2=2019&h1=23&i1=55&s1=00&h2=23&i2=48&s2=30).
Can somebody explain these results? Am I doing something wrong or missing something obvious? I'm using Postgres 9.6 on macOS.
Many countries switch to daylight savings time between March 2nd and April 1st. Because the clocks move ahead there is one less hour between 2.March.2019 and 1.April.2019.
Beware that Postgres has its own time zone which may not match the user's time zone, especially for a web application. To deal with this, set the application to the user's time zone and the database to UTC. Translate all dates to UTC before passing them to the database.
My table has a category 'timestamp' where the timestamps are formatted 2015-06-22 18:59:59
However, using DBVisualizer Free 9.2.8 and Vertica, when I try to pull up rows by timestamp with a
SELECT * FROM table WHERE timestamp = '2015-06-22 18:59:59';
(directly copy-pasting the stamp), nothing comes up. Why is this happening and is there a way around it?
FYI, saying "the timestamps are formatted 2015-06-22 18:59:59" is incorrect if you are indeed using a TIMESTAMP type. Such types have their own internal representation of a date-time value, almost always a count since epoch. In your case with Vertica, 8 bytes are used for such storage. The formatting of the date-time value happens when a string representation is generated. Never confuse the string representation with the date-time value. Conflating the two may well be related to your problem/confusion.
A few different thoughts about possible problems…
String Literals
Are you sure Vertica takes strings as timestamp literals? That format you used is common SQL format. But given that Vertica seems to be a specialized database, I would double-check that.
If strings are not allowed, you may need to call some kind of function to transform the string into a date-time values.
Fractional Second
As the comment by Martin Smith points out, the doc for Timestamp-related data types in Vertica 7.1 says those types can have a fractional second to resolution of microseconds. That means up to 6 decimal places of a fraction.
So if you are searching for "2015-06-22 18:59:59" but the stored value is "2015-06-22 18:59:59.012345", no match on the query.
Half-Open
The fractional seconds issue described above is often the cause of problems people have when handling a span of time. If you naïvely try to pinpoint the ending time, you are likely to have problems. Seeing the "59:59" in your example string makes me think this applies to you.
The better approach to spans of time is "Half-Open" (or Half-Closed, whatever) where the beginning is inclusive while the ending is exclusive. Common notation for this is [). In comparison logic this means: value >= start AND value < stop. Notice the lack of EQUALS SIGN in the stop comparison. In English we would say "look for an hour's worth of invoices starting at 2:00 PM and going up to, but not including, 3:00 PM".
Half-Open for a week means Monday-Monday, for a month the first of one month to the first of the next month, and for a year the January 1 of one year to January 1 of the following year.
Half-Open means not using BETWEEN in SQL. SQL's BETWEEN has often be criticized. Instead do something like the following to look for an hour's worth of invoices. Notice the Z on the end of string literal which means "UTC time zone" ("Z" for "Zulu"). (But verify, as my SQL syntax may need fixing.)
SELECT *
FROM some_table_
WHERE invoice_received_ >= '2015-06-22 18:00:00Z'
AND invoice_received_ < '2015-06-22 19:00:00Z'
;
This query will catch any values such as '2015-06-22 18:59:59.654321" which seems to be eluding you.
Reserved Word
I hope you have not really named your table 'table' and your column 'timestamp'. Such use of keywords and reserved words can cause explicit errors or more subtle weird problems.
Tip: The easy way to avoid any of the over a thousand reserved words in various databases is to append a trailing underscore. The SQL standard explicitly promises to never using a trailing underscore in its reserved words. So use "timestamp_" rather than "timestamp". Another example: "invoice_" table and "received_" column. I recommend doing that as a habit on everything your name in SQL: columns, tables, constraints, indexes, and so on.
Time Zone
You are using the TIMESTAMP which is short for TIMESTAMP WITHOUT TIME ZONE. Or so I presume; the Vertica doc is vague but that is the common usage as seen in the Postgres doc, and may even be standard SQL.
Anyways, TIMESTAMP WITHOUT TIME ZONE is usually the wrong type for most business purposes. The WITH time zone is misnamed and often misunderstood as a consequence: It means "with respect for time zone" where data inputs that include an offset or other time zone information from UTC are adjusted to UTC during the INSERT/UPDATE operations. The WITHOUT type simply ignores any such offset or time zone information.
The WITHOUT type should only be used for the concept of a date-time generally without being tied to any one locality. For example, saying "Christmas this year starts at beginning of December 25, 2015". That means in any time zone rather than a specific time zone. Obviously Christmas starts earlier in Paris, for example, than in Montréal.
If you are timestamping legal documents such as invoices, or booking appointments with people across time zones, or scheduling shipments in various localities, you should be using WITH time zone type.
So back to your possible problem: Test how Vertica or your client app or your database driver is handling your input string. It may be adjusting time zones as part of the parsing of the string using your client machine’s current default time zone. When sent to the database, that value will not match the stored value if during storage no adjustment to UTC was made.
Tip: Generally best practice is to do all your storage and business logic in UTC, adjusting to local time zones only where expected by user.
I have this SQL query to a PostgreSQL database. Can it be shortened? I am thinking about the where part.
SELECT *
FROM reservations
WHERE (starts_at BETWEEN ? AND ?) OR (ends_at BETWEEN ? AND ?)
The values for the question-marks is:
The beginning of the current date in datetime format
The end of the the current date in datetime format
Same as point one
Same as point two
The code is meant to return all the reservations that begins or ends on a certain date. And works as it is supposed to. But I have to supply the same information multiple times into the query.
I so not actually use this exactly SQL, so there might be an obvious error somewhere, but please focus on the where part
I'm not a huge fan of BETWEEN in this context, because timestampor datetime can be fractional. In particular, specifying the last possible value on a given date is much more complicated than specifying the first possible value (midnight) because you have to specify the time as 23:59:59.999... out to whatever precision your RDBMS uses. PostgreSQL's timestamp is supposed to be accurate to the microsecond (1e-6 seconds), for example, so it's easy to specify a range that either includes times you don't want, or misses times that you do.
On the other hand, if you use BETWEEN with midnight of the following day so you don't have to know the precision of the time, you're including a time that doesn't exist in the date you're interested in. If your application is only precise to the second, or the minute, or to 5 minutes, then you may mis-categorize data or, worse, count it twice since it suddenly counts as being in two dates.
I would prefer:
WHERE (starts_at >= ? AND starts_at < ?)
OR (ends_at >= ? AND ends_at < ?)
Where the ? map to:
Midnight of the target date.
Midnight of the date after the target date.
Midnight of the target date.
Midnight of the date after the target date.
It's not as short, but it's decidedly safer unless you really want to specify your intervals that precisely.
However, you should not do the following, even though it's shorter:
WHERE DATE(starts_at) = ?
OR DATE(ends_at) = ?
You don't want to do that because it's not SARGEable.
This is also an example of why shortness or brevity is a poor measure of code quality. Generally, I'd order my preference like so:
Accuracy.
Performance.
Readability/maintainability.
Brevity.
No, you can't improve upon this.
WHERE (starts_at BETWEEN ? AND ?) OR (ends_at BETWEEN ? AND ?)
Q1: I want to convert a unix timestamp (INT) to monetdb timestamp ('YYYY-MM-DD HH:MM:SS') format
but it is giving me the GMT time not my actual time.
When I do
select (epoch(cast(current_timestamp as timestamp))-epoch(timestamp '2013-04-25 11:49:00'))
where 2013-04-25 11:49:00 is my systems current time it gives the same difference
I tried using
set time zone interval '05:30' HOUR TO MINUTE;
but it did not change the result
How can I solve this problem??
Example Problem:
I wanted to convert unix timestamp 1366869289 which should be around "2013-04-25 11:25:00" but monetdb gives "2013-04-25 05:55:00"
Knowing nothing about MonetDB, but a lot about timezones, I decided to look in their documentation to see what kind of datatypes are supported and how conversions are handled.
I found this page on Temporal data types. Based on that, I can conclude that a timestamp in MonetDB is always intended to reference UTC/GMT time - which is consistent with other systems.
In order to get a value that is for a particular time zone, they offer the following example:
SET TIME ZONE INTERVAL '1' HOUR TO MINUTE
I assume this means to set the database to offset all times by 1 hour, effectively placing the values all in UTC+01:00, such as is the offset for British Summer Time.
The page also goes on to point out the problems that can arise with using just and offset to adjust time values (see TimeZone != Offset in the TimeZone tag wiki). It also offers a list of various named time zones. But it does not show how to set a time zone to one of the named values. Also, their list appears to be proprietary, and incomplete. While at first glance they appear to have similarities to the IANA/Olson time zone database - the identifiers they specify are not valid TZDB names.
There are some other functions listed on this page, without much explanation. One that looks promising for your needs is LOCALTIMESTAMP. Perhaps this will take the local time zone into account, which appears to be what you were looking for.
I could not find any additional details specific to MonetDB date/time/timezone handling. The documentation appears to be fairly incomplete. You might want to reach out to their mailing list.
If I want to store date and time, is it better to store them in a separate date and time or use a single datetime?
When should we use date and time instead of a single datetime?
I want to filter my queries either using date or time.
When you are talking about a moment in time, whether a universal moment, or a specific date and time on someone's local calendar, you use a datetime. If you want to be sure that you are talking about an exact moment in time, regardless of the observer, then you use a datetimeoffset.
If you are storing just a date then you mean a date without a time component, meaning "any time on this date".
If you are storing just a time then you mean a time without a date component, meaning "this time on any date", or "this time on a date determined by some other means".
There is no practical purpouse to having both a date and a time that are about the same thing, sitting on the same row. Just use a datetime for that.
In SQL Server 2008 you have date and time data types so this becomes a non issue.
If it is good choice it really depends by your business and how you will query you data.
If for example you want to know all the orders places between 1 and 2 PM for any day using a separated Date and Time column will make it quicker
If you intentionally do not care about the time, it's more efficient to store this data as a date datatype. Think a customer birthday column, there's not too many cases I can think of that would use this time. If there happens to be a time attached to it (often a bug), this needs to be removed via a convert statement in order to do a compare. It also consumes additional space if you don't need these values (3 bytes compared to 8).
I think it's similar to having a status code table with the id as a bigint instead of a tinyint or the like (depending on how many status codes you would plan to have).
It's just a matter of what you're using the data for, if you think there's a good chance you'll ever need the that data, then use datetime, otherwise use date.
Nothing brilliant about separating date and time,
Better you save date and time in Same column,
Here they have discussed the same issue check it : are-there-any-good-reasons-for-keeping-date-and-time-in-separate-columns
you can also get date and time separately by query
SELECT
CONVERT(VARCHAR(10),GETDATE(),111) as DatePart,
convert(varchar(15), getdate(), 108) TimePart