So I've been given a lovely little database. One of the tables in the database (several million rows large) has this column:
time_in character varying(255)
Stored in there is an epoch timestamp. What is the most sane way I can convert this to a proper epoch timestamp column without losing data?
First off there is no separate epoch timestamp datatype so the type you want to convert to is just regular timestamp. In the PostgreSQL Documentation - ALTER TABLE there's an example that fits to your case almost perfectly (I just added a cast to integer):
ALTER TABLE foo
ALTER COLUMN time_in SET DATA TYPE timestamp with time zone
USING
timestamp with time zone 'epoch' + time_in::integer * interval '1 second';
Note that the conversion might take some time and will produce an error if all of the rows are not valid epoch times.
Or, quoting the manual here:
A single-argument to_timestamp function is also available; it accepts
a double precision argument and converts from Unix epoch (seconds
since 1970-01-01 00:00:00+00) to timestamp with time zone. (Integer
Unix epochs are implicitly cast to double precision.)
ALTER TABLE foo ALTER COLUMN time_in
SET DATA TYPE timestamptz USING to_timestamp(time_in::float8);
But first, decide whether timestamp (timestamp without time zone) or timestamptz (timestamp with time zone) is the better choice for you:
Ignoring timezones altogether in Rails and PostgreSQL
Related
I have a table with a column defined as:
"timestamp" int8 NULL,
which stores values like
'1638462043745210034'
When I try to cast it to timestamp or timestamp with time-zone
SELECT '1638462043745210034'::timestamp at time zone 'UTC' ;
it returns an error:
date/time field value out of range: "1638462043745210034" Hint:
Perhaps you need a different "datestyle" setting.
What kind of datestyle can it be and how can it be converted to a normal timestamp?
Looks like nanoseconds since the epoch:
SELECT to_timestamp(1638462043745210034 / 1000000000.0);
to_timestamp
══════════════════════════════
2021-12-02 17:20:43.74521+01
(1 row)
I am having some trouble understanding how to deal in Postgres with time zone semantic:
Consider this table
TABLE MyTable (
MyDate TIMESTAMP NOT NULL,
// other columns
);
And this query
SELECT *
FROM MyTable
WHERE // conditions
AND tstzrange(#{start} ::timestamp with time zone, #{end} ::timestamp with time zone] #> MyDate::timestamp without time zone at time zone 'CET'
I understand that Postgres only stores the epoch value i.e. no time zone info is ever stored, so I understand the need to specify timezone for start and end as they are formatted strings that Postgres needs to calculate the epoch for.
What I don't really understand is:
MyDate::timestamp without time zone at time zone 'CET'
Postgres knows the epoch for MyDate since it's their values are stored, why the need to "convert" to a time zone ?
What are we actually saying here and can this be simplified ?
Postgres never stores timezone information in timestamp or timestamptz. In the timestamptz case the timestamp being stored is rotated to a UTC value using either the timezone information in the presented timestamp or the value of the setting TimeZone and then stored. In the timestamp case that is not done. On output a timestamptz is rotated back from UTC to whatever the TimeZone setting is or what via at time zone <some_tz>. In the timestamp case the retrieved value is assumed to be the setting of TimeZone unless you override with at time zone <some_tz>. Best practices is that you use timestamptz.
I have a String in this format: 2018-11-01T00:00:00-07:00 and I would like to convert it to a TIMESTAMP and insert it into a TIMESTAMP column. However, when I insert it, it drops the -07:00 without first converting it to -00:00. How do I ensure that it is converted and stored in Redshift properly?
Here is an example:
select ORIGINAL_DATE, TO_TIMESTAMP(ORIGINAL_DATE,'YYYY-MM-DD HH24:MI:SS') FROM CDW_LANDING.X where id = XXXXXX;
=> 2018-11-01T00:00:00-07:00 2018-10-31 17:00:00
The TO_TIMESTAMP converts it to 2018-10-31 17:00:00 which is what I want. However, when I insert it, it becomes 2018-11-01 00:00:00 and simply drops the -07:00.
Here is the example:
insert into cdw_stage.X (ORIG_DT)
select TO_TIMESTAMP(ORIGINAL_DATE,'YYYY-MM-DD HH24:MI:SS')
from CDW_LANDING.INVOICE where id = XXXXXX;
But when I query it with select ORIG_DT from cdw_landing.X;, it displays 2018-11-01 00:00:00. What I would like to see is 2018-10-31 17:00:00 which is what the TO_TIMESTAMP function should do.
The ORIG_DT in Redshift is in TIMESTAMP format. The input date is in VARCHAR.
How do I get Redshift to save this correctly? I also added postgres tag because Redshift is based off of postgres. Thank you so much!!!
2018-11-01T00:00:00-07:00 is not a timestamp (timestamp without time zone) literal, strictly speaking. It is a timestamptz (timestamp with time zone) literal. This is the root of all pain in your question. The wrong cast to timestamp ignores the offset. The Postgres manual:
In a literal that has been determined to be timestamp without time zone, PostgreSQL will silently ignore any time zone indication. That
is, the resulting value is derived from the date/time fields in the
input value, and is not adjusted for time zone.
Bold emphasis mine.
The use of TO_TIMESTAMP() can't save you. The Redshift manual:
Formats that include a time zone (TZ, tz, or OF) are not supported as input.
(The same is true in Postgres.)
Solution
Cast to timestamptz (or use a column of that type to begin with), the rest should fall in place:
SELECT cast('2018-11-01T00:00:00-07:00' AS timestamptz);
Or:
SELECT '2018-11-01T00:00:00-07:00'::timestamptz;
The manual about casting in Redshift.
When an actual timestamptz is assigned to a timestamp column it is converted according to the current timezone setting of the session automatically. If you want a different target timezone, use the AT TIME ZONE construct. Details:
Ignoring time zones altogether in Rails and PostgreSQL
The related answer is for Postgres, but timestamp handling in Redshift (while differing in many other aspects!) is the same. The Redshift manual:
When converting DATE or TIMESTAMP to TIMESTAMPTZ, DATE or TIMESTAMP
are assumed to use the current session time zone. The session time
zone is UTC by default. For more information about setting the session
time zone, see timezone.
I have a value in my csv file for timetamp as '1522865628160'. When I load the data in bigQuery where this field type is timestamp, it saves the timestamp as '1522865628160000'. so when I query like
select * from <tablename> limit 1
it gives me error
Cannot return an invalid timestamp value of 1522865628160000000 microseconds relative to the Unix epoch. The range of valid timestamp values is [0001-01-1 00:00:00, 9999-12-31 23:59:59.999999]; error in writing field timestamp"
please help
I think the issue here is that you tried to load your UNIX timestamp data into a timestamp column in BigQuery. A BigQuery timestamp column is not the same thing as a UNIX timestamp. The latter is just a numerical value representing the number of seconds since the start of the UNIX epoch in 1970.
So the fix here would be to load your data into an INT64 (or INTEGER if you are using legacy) column. From there, you may convert your UNIX timestamp to a bona fide date or timestamp.
There is a MSEC_TO_TIMESTAMP() function which can convert an integer number of milliseconds since the UNIX epoch to a bona fide timestamp, e.g.
SELECT MSEC_TO_TIMESTAMP(1522865628160)
2018-04-04 11:13:48 UTC
I am new to postgresql bot not to sql in general. I have a table that I need to read values from, on of the columns is a unix timestamp that I want to convert in to a more human readable format thus I found this:
SELECT lt,dw,up,to_char(uxts, 'YYYY-MM-DD HH24:MI:SS')
from products;
But that produces an error:
ERROR: multiple decimal points
I am lost here. I am sure someone can show me how to do it. The documentation isn't that clear to me. Postgresql 9.5 is the database.
to_char() converts a number, date or timestamp to a string, not the other way round.
You want to_timestamp()
Convert Unix epoch (seconds since 1970-01-01 00:00:00+00) to timestamp
So just apply that function on your column
SELECT lt,dw,up,to_timestamp(uxts) as uxts
from products;
This assumes that uxts is some kind of number data type (integer, bigint or double precision)