I have a String in this format: 2018-11-01T00:00:00-07:00 and I would like to convert it to a TIMESTAMP and insert it into a TIMESTAMP column. However, when I insert it, it drops the -07:00 without first converting it to -00:00. How do I ensure that it is converted and stored in Redshift properly?
Here is an example:
select ORIGINAL_DATE, TO_TIMESTAMP(ORIGINAL_DATE,'YYYY-MM-DD HH24:MI:SS') FROM CDW_LANDING.X where id = XXXXXX;
=> 2018-11-01T00:00:00-07:00 2018-10-31 17:00:00
The TO_TIMESTAMP converts it to 2018-10-31 17:00:00 which is what I want. However, when I insert it, it becomes 2018-11-01 00:00:00 and simply drops the -07:00.
Here is the example:
insert into cdw_stage.X (ORIG_DT)
select TO_TIMESTAMP(ORIGINAL_DATE,'YYYY-MM-DD HH24:MI:SS')
from CDW_LANDING.INVOICE where id = XXXXXX;
But when I query it with select ORIG_DT from cdw_landing.X;, it displays 2018-11-01 00:00:00. What I would like to see is 2018-10-31 17:00:00 which is what the TO_TIMESTAMP function should do.
The ORIG_DT in Redshift is in TIMESTAMP format. The input date is in VARCHAR.
How do I get Redshift to save this correctly? I also added postgres tag because Redshift is based off of postgres. Thank you so much!!!
2018-11-01T00:00:00-07:00 is not a timestamp (timestamp without time zone) literal, strictly speaking. It is a timestamptz (timestamp with time zone) literal. This is the root of all pain in your question. The wrong cast to timestamp ignores the offset. The Postgres manual:
In a literal that has been determined to be timestamp without time zone, PostgreSQL will silently ignore any time zone indication. That
is, the resulting value is derived from the date/time fields in the
input value, and is not adjusted for time zone.
Bold emphasis mine.
The use of TO_TIMESTAMP() can't save you. The Redshift manual:
Formats that include a time zone (TZ, tz, or OF) are not supported as input.
(The same is true in Postgres.)
Solution
Cast to timestamptz (or use a column of that type to begin with), the rest should fall in place:
SELECT cast('2018-11-01T00:00:00-07:00' AS timestamptz);
Or:
SELECT '2018-11-01T00:00:00-07:00'::timestamptz;
The manual about casting in Redshift.
When an actual timestamptz is assigned to a timestamp column it is converted according to the current timezone setting of the session automatically. If you want a different target timezone, use the AT TIME ZONE construct. Details:
Ignoring time zones altogether in Rails and PostgreSQL
The related answer is for Postgres, but timestamp handling in Redshift (while differing in many other aspects!) is the same. The Redshift manual:
When converting DATE or TIMESTAMP to TIMESTAMPTZ, DATE or TIMESTAMP
are assumed to use the current session time zone. The session time
zone is UTC by default. For more information about setting the session
time zone, see timezone.
Related
I am having some trouble understanding how to deal in Postgres with time zone semantic:
Consider this table
TABLE MyTable (
MyDate TIMESTAMP NOT NULL,
// other columns
);
And this query
SELECT *
FROM MyTable
WHERE // conditions
AND tstzrange(#{start} ::timestamp with time zone, #{end} ::timestamp with time zone] #> MyDate::timestamp without time zone at time zone 'CET'
I understand that Postgres only stores the epoch value i.e. no time zone info is ever stored, so I understand the need to specify timezone for start and end as they are formatted strings that Postgres needs to calculate the epoch for.
What I don't really understand is:
MyDate::timestamp without time zone at time zone 'CET'
Postgres knows the epoch for MyDate since it's their values are stored, why the need to "convert" to a time zone ?
What are we actually saying here and can this be simplified ?
Postgres never stores timezone information in timestamp or timestamptz. In the timestamptz case the timestamp being stored is rotated to a UTC value using either the timezone information in the presented timestamp or the value of the setting TimeZone and then stored. In the timestamp case that is not done. On output a timestamptz is rotated back from UTC to whatever the TimeZone setting is or what via at time zone <some_tz>. In the timestamp case the retrieved value is assumed to be the setting of TimeZone unless you override with at time zone <some_tz>. Best practices is that you use timestamptz.
I have timestamps like the following '2018-04-18T18:11:16+01:00' in varchar type and want to transform them to a timestamp type. I am having trouble using the to_timestamp(). Is there a specific way to use the to_timestamp() function that will allow me to do the transformation i want? If not, is there any other way to achieve my goal?
Doing
to_timestamp('2018-04-18T18:11:16+01:00', 'DD-MM-YYYY HH24:MI:SS')
gives an error of
Error: date/time field value out of range:
"2018-04-18T18:11:16+01:00"
I would expect varchars like '2018-04-18T18:11:16+01:00' to be transformed to 2018-04-18 17:11:16 (where type is timestamp)
demo:db<>fiddle
If you simply want to get the string into a timestamp holding the time zone:
Simply cast it into timestamp with time zone (= timestamptz) type:
SELECT '2018-04-18T18:11:16+01:00'::timestamptz
If you just want to cut the time zone part and holding the time without any further calculations: Simply cast it into type timestamp without time zone (= timestamp)
SELECT '2018-04-18T18:11:16+01:00'::timestamp
If you want to convert it into a timestamp at UTC (calculating -1) you can do:
SELECT ('2018-04-18T18:11:16+01:00' AT TIME ZONE 'UTC')::timestamp
I would like to insert time data type in postgresql that includes the timezone and is aware of daylight savings time. This is what I have done:
CREATE TABLE mytable(
...
start_time time(0) with time zone,
end_time time(0) with time zone
)
INSERT INTO mytable(start_time, end_time)
VALUES(TIME '08:00:00 MST7MDT', TIME '18:00:00 MST7MDT')
I get the following error:
invalid input syntax for type time: "08:00:00 MST7MDT"
It works if I use 'MST' instead of 'MST7MDT', but I need it to be aware of DST. I also tried using 'America/Edmonton' as the timezone, but I got the same error.
What is the proper way to insert a time value (not timestamp) with timezone and DST?
EDIT:
I would actually like to use the 'America/Edmonton' syntax
The proper way is not to use time with time zone (note the space between time and zone) at all, since it is broken by design. It is in the SQL standard, so Postgres supports the type - but advises not to use it. More in this related answer:
Accounting for DST in Postgres, when selecting scheduled items
Since you are having problems with DST, timetz (short name) is a particularly bad choice. It is ill-equipped to deal with DST. It's impossible to tell whether 8:00:00 is in winter or summer time.
Use timestamp with time zone (timstamptz) instead. You can always discard the date part. Simply use start_time::time to get the local time from a timestamptz. Or use AT TIME ZONE to transpose to your time zone.
Generally, to take DST into account automatically, use a time zone name instead of a time zone abbreviation. More explanation in this related question & answer:
Time zone names with identical properties yield different result when applied to timestamp
In your particular case, you could probably use America/Los_Angeles (example with timestamptz):
INSERT INTO mytable(start_time, end_time)
VALUES
('1970-01-01 08:00:00 America/Los_Angeles'
, '1970-01-01 18:00:00 America/Los_Angeles')
I found this by checking:
SELECT * FROM pg_timezone_names
WHERE utc_offset = '-07:00'
AND is_dst;
Basics about time zone handling:
Ignoring time zones altogether in Rails and PostgreSQL
How about this?
INSERT INTO mytable(start_time, end_time)
VALUES('08:00:00'::time at time zone 'MST7MDT', '18:00:00'::time at time zone 'MST7MDT')
Do I need to convert TIMESTAMP to DATE before doing a comparison? EX:
AND SORD.CREATED >= TO_DATE(FROM_TZ(CAST(TO_DATE('','DD-MON-YYYY HH24:MI:SS') AS TIMESTAMP), 'Asia/Kuala_Lumpur') AT TIME ZONE 'UTC','DD-MON-YYYY HH24:MI:SS')
AND SORD.CREATED < TO_DATE(FROM_TZ(CAST(TO_DATE('','DD-MON-YYYY HH24:MI:SS') AS TIMESTAMP), 'Asia/Kuala_Lumpur') AT TIME ZONE 'UTC','DD-MON-YYYY HH24:MI:SS')
Just looking for an opinion, since I'm still new and learning about databases.
It's hard to tell exactly what you database you are using. From the functions in the code you provided, I am guessing Oracle? Please confirm.
In general, I can tell you the following:
You aren't passing any values. The first parameter to TO_DATE is empty, and that would contain the string to convert to a date.
You need to know what the values in your database are. Is SORD.CREATED a DATE or TIMESTAMP datatype? Does it keep UTC values? If so, then yes, you should convert local times to UTC before querying. If possible, you should do this outside of the query itself.
If the values in your database are of type TIMESTAMP WITH TIME ZONE, then you can use a DATE or TIMESTAMP with the AT TIME ZONE function, like you showed in your question.
Don't cast to/from a string unnecessarily. If your source data is already in a DATE or TIMESTAMP datatype, there's no need to convert to a string. Once a date is a date, you should keep it a date. Introducing string formatting can sometimes introduce errors.
I have a SQL Builder library that direcltly uses ADO.NET. I have a means of creating a select query with a greater-than-or-equal operator, like:
select *
from book
where book.date_created >= {some date}
My issue is that {some date} is going to always be in the UTC time zone, but it's being compared to the book.date_created column which is a TIMESTAMP(6) WITH TIME ZONE column, which will not be in the UTC timezone.
I can execute the query, but my results are off becuaes of timezone comparisons. My query is for all books where the date_created >= x, but some of the results returned are not greater than x because after subtracting 5 hours for the time zone, they are now less than x. The IDataRecord DateTime fields returned are converted to UTC using DateTime.SpecifyKind()
Can I form my query such that it interprets book.date_created in the UTC timezone?
Note: While I'd love to change my Oracle DB columns to not specify timezones, changing table structures is not something I can do.
Edit:
Currently, {some date} is a SQL Parameter. It's backing datatype is a DateTime with UTC as the timezone. As a parameter, it is a TimestampWithTZ. The Value of the parameter is a DateTime with the kind specified as UTC as well.
Update:
The issue seems to be related to my results set from the IDataRecord. When I pull DateTimes off, I use DateTime.SpecifyKind() to put them in UTC mode. The problem is, the date times come out as DateTimeKind.Unspecified. When converting from Unspecified to UTC, it just drops the timezone and declares it is UTC without changing the underlying value. I'm not sure how to have the IDataRecord pull in the TimeZone value.
You need to use the FROM_TZ function that transforms a TIMESTAMP into a TIMESTAMP WITH TIME ZONE. For example, if you know that your variable is in UTC time (+0:00):
SELECT *
FROM book
WHERE date_created >= from_tz(<timestamp>, '+0:00');
Here's a sample script that shows the behaviour you describe (your local time zone should be set to +1:00):
CREATE TABLE t (tz TIMESTAMP(6) WITH TIME ZONE);
INSERT INTO t VALUES
(to_timestamp_tz('20000101 00:00:00 +1:00','yyyymmdd hh24:mi:ss tzh:tzm'));
INSERT INTO t VALUES
(to_timestamp_tz('20000101 00:00:00 -1:00','yyyymmdd hh24:mi:ss tzh:tzm'));
-- This will return two values instead of one
SELECT *
FROM t
WHERE tz >= to_timestamp('20000101 00:00:00', 'yyyymmdd hh24:mi:ss');
-- This query will return only one row
SELECT *
FROM t
WHERE tz >= from_tz (to_timestamp('20000101 00:00:00',
'yyyymmdd hh24:mi:ss'), '+0:00');
below links will help you.
Datetime Datatypes and Time Zone Support
TIMESTAMP WITH TIME ZONE Data Type
Write Time Zone Aware Code in Oracle
ORACLE timezone summary
Oracle Date and Time data types