I'm playing with some tables in bigquery and I receive this error:
Cannot return an invalid timestamp value of -62169990264000000 microseconds relative to the Unix epoch.
The range of valid timestamp values is [0001-01-1 00:00:00, 9999-12-31 23:59:59.999999]
Doing the query in legacy sql and sorting ascending, it displays as 0001-11-29 22:15:36 UTC
How does it get transformed into microseconds?
This is the query:
#standardSQL
SELECT
birthdate
FROM
X
WHERE
birthdate IS NOT NULL
ORDER BY
birthdate ASC
**strong text**Confirming , that in BigQuery Legacy SQL
SELECT USEC_TO_TIMESTAMP(-62169990264000000)
produces 0001-11-29 22:15:36 UTC timestamp
whereas in BigQuery Standard SQL
SELECT TIMESTAMP_MICROS(-62169990264000000)
produces error:
TIMESTAMP value is out of allowed range: from 0001-01-01 00:00:00.000000+00 to 9999-12-31 23:59:59.999999+00.
How does it get transformed in microseconds?
TIMESTAMP
You can describe TIMESTAMP data types as either UNIX timestamps or calendar datetimes. BigQuery stores TIMESTAMP data internally as a UNIX timestamp with microsecond precision.
See more about TIMESTAMP type
Midnight of January 1 of the year 0001 (the minimum possible timestamp value in standard SQL) is -62135596800000000 in microseconds relative to the UNIX epoch, which is greater than -62169990264000000. I don't have a good explanation for legacy SQL's behavior with that timestamp value, but you can read about some suggestions for dealing with it in standard SQL in this item on the issue tracker. We plan to add some content to the migration guide about this timestamp behavior in the future as well.
Related
I have seperate date and time columns in my table. Date as mm/dd/yyyy, time as hh:mm but i can change the format. I want to list data between 2 date/time. How can I do that?
select * from testtable where date >= '01/10/2022' AND date <= '01/10/2023' AND time >= '13:45' AND time <= '15:50'
I wrote it but of course it doesn't work like what i expected.
The best fix and really the only one you want here would be to start storing your timestamps in a sortable ISO format yyyy-mm-dd hh:ii:ss. Then, use this query:
SELECT *
FROM testtable
WHERE date BETWEEN '2022-01-10 13:45:00' AND '2023-01-10 15:50:00';
The thing to realize here is that SQLite does not actually have a date column type. Rather, you always store your dates/timestamps as text, and therefore it is crucial to use an ISO sortable format as shown above.
If your target is sqlite, it lacks complex timestamp types. But you have another option here. You can store that as unix timestamp, it is an integer representing the number of seconds offset from the epoch which is 1970-01-01 00:00:00 UTC. The table format would then be:
CREATE TABLE testtable (
date INTEGER
);
You can the use the unixepoch function to translate a string representation to that unix timestamp format. To insert a new date, you would use:
INSERT INTO testtable (date) VALUES (unixepoch('2023-01-11T11:30:00+01:00'))
Finding a matching row is now as easy to compare integers together. You can convert the datetime representation to unix timestamp at the application level, most programming environments provide such time utility functions/library. Or can still use the unixepoch function from sqlite for your where clause.
SELECT * FROM testtable
WHERE date >= unixepoch('2022-10-01T13:45:00Z')
AND date <= unixepoch('2023-10-01T15:50:00Z')
The final Z indicates an UTC time zone but you can adjust that with another +HH:00 extenstion instead which reflect the offset from utc of your datetime representation.
I have a Oracle SQL statement where I have to get the current timestamp as one of the columns. But I dont require the Timezone which CURRENT_TIMESTAMP gives or the AM/PM given by LOCALTIMESTAMP.
I require the current timestamp in 24hr format without the timezone.
Is it possible to get that in Oracle SQL?
It seems you're mixing 2 concepts here: "datatype" and "date format mask".
data type: LOCALTIMESTAMP returns datatype TIMESTAMP and CURRENT_TIMESTAMP returns datatype TIMESTAMP WITH TIME ZONE. TIMESTAMP is similar to DATE but has a higher precision. As usual... checking the docs is worth it.
date format mask: determines how you display the date information. Americans can't read 24 hour format, the rest of the world is confused by AM/PM. Fortunately, you can decide how you want to display the date as explained in the oracle docs.
If you just want to return the current date in 24 hour format you could do something like:
SELECT
TO_CHAR(SYSDATE,'DD-MON-YYYY HH24:MI:SS') as mydate,
<other columns>
FROM
<table_name>
If you need the date to be more precise and you require fractional seconds then you can use SYSTIMESTAMP instead of DATE with a format mask 'DD-MON-YYYY HH24:MI:SS.FF9'
In my query I need to return only those records that are greater than another date, returned by the function. The field I am comparing is a timestamp and the date function returns is a of type DATE.
My query:
SELECT * FROM TABLE1
WHERE MY_TIMESTAMP > MyFunction1('Test Date');
when I am comparing two values where only time differs slightly - in munutes or seconds, the comparison does not work.
For example when
MY_TIMESTAMP=11/27/2018 12:15:42.000000 PM -05:00
and
MyFunction1('Test Date') returns 11/27/2018 12:22:00 PM
no record should be returned from TABLE1 for that ID but the record does get returned. What can I do to make the comparison more granular?
From Datetime and Interval Arithmetic
Oracle Database performs all timestamp arithmetic in UTC time. For
TIMESTAMP WITH LOCAL TIME ZONE data, Oracle Database converts the
datetime value from the database time zone to UTC and converts back to
the database time zone after performing the arithmetic. For TIMESTAMP WITH TIME ZONE data, the datetime value is always in UTC, so no
conversion is necessary.
If you provide a DATE or TIMESTAMP value (i.e. value without any time zone information) then Oracle attaches the SESSIONTIMEZONE to this value and then it does the comparison based on UTC.
Attaching SESSIONTIMEZONE to input value might be correct or not, it depends on your current session and content of MyFunction1
I have a String in this format: 2018-11-01T00:00:00-07:00 and I would like to convert it to a TIMESTAMP and insert it into a TIMESTAMP column. However, when I insert it, it drops the -07:00 without first converting it to -00:00. How do I ensure that it is converted and stored in Redshift properly?
Here is an example:
select ORIGINAL_DATE, TO_TIMESTAMP(ORIGINAL_DATE,'YYYY-MM-DD HH24:MI:SS') FROM CDW_LANDING.X where id = XXXXXX;
=> 2018-11-01T00:00:00-07:00 2018-10-31 17:00:00
The TO_TIMESTAMP converts it to 2018-10-31 17:00:00 which is what I want. However, when I insert it, it becomes 2018-11-01 00:00:00 and simply drops the -07:00.
Here is the example:
insert into cdw_stage.X (ORIG_DT)
select TO_TIMESTAMP(ORIGINAL_DATE,'YYYY-MM-DD HH24:MI:SS')
from CDW_LANDING.INVOICE where id = XXXXXX;
But when I query it with select ORIG_DT from cdw_landing.X;, it displays 2018-11-01 00:00:00. What I would like to see is 2018-10-31 17:00:00 which is what the TO_TIMESTAMP function should do.
The ORIG_DT in Redshift is in TIMESTAMP format. The input date is in VARCHAR.
How do I get Redshift to save this correctly? I also added postgres tag because Redshift is based off of postgres. Thank you so much!!!
2018-11-01T00:00:00-07:00 is not a timestamp (timestamp without time zone) literal, strictly speaking. It is a timestamptz (timestamp with time zone) literal. This is the root of all pain in your question. The wrong cast to timestamp ignores the offset. The Postgres manual:
In a literal that has been determined to be timestamp without time zone, PostgreSQL will silently ignore any time zone indication. That
is, the resulting value is derived from the date/time fields in the
input value, and is not adjusted for time zone.
Bold emphasis mine.
The use of TO_TIMESTAMP() can't save you. The Redshift manual:
Formats that include a time zone (TZ, tz, or OF) are not supported as input.
(The same is true in Postgres.)
Solution
Cast to timestamptz (or use a column of that type to begin with), the rest should fall in place:
SELECT cast('2018-11-01T00:00:00-07:00' AS timestamptz);
Or:
SELECT '2018-11-01T00:00:00-07:00'::timestamptz;
The manual about casting in Redshift.
When an actual timestamptz is assigned to a timestamp column it is converted according to the current timezone setting of the session automatically. If you want a different target timezone, use the AT TIME ZONE construct. Details:
Ignoring time zones altogether in Rails and PostgreSQL
The related answer is for Postgres, but timestamp handling in Redshift (while differing in many other aspects!) is the same. The Redshift manual:
When converting DATE or TIMESTAMP to TIMESTAMPTZ, DATE or TIMESTAMP
are assumed to use the current session time zone. The session time
zone is UTC by default. For more information about setting the session
time zone, see timezone.
I have a value in my csv file for timetamp as '1522865628160'. When I load the data in bigQuery where this field type is timestamp, it saves the timestamp as '1522865628160000'. so when I query like
select * from <tablename> limit 1
it gives me error
Cannot return an invalid timestamp value of 1522865628160000000 microseconds relative to the Unix epoch. The range of valid timestamp values is [0001-01-1 00:00:00, 9999-12-31 23:59:59.999999]; error in writing field timestamp"
please help
I think the issue here is that you tried to load your UNIX timestamp data into a timestamp column in BigQuery. A BigQuery timestamp column is not the same thing as a UNIX timestamp. The latter is just a numerical value representing the number of seconds since the start of the UNIX epoch in 1970.
So the fix here would be to load your data into an INT64 (or INTEGER if you are using legacy) column. From there, you may convert your UNIX timestamp to a bona fide date or timestamp.
There is a MSEC_TO_TIMESTAMP() function which can convert an integer number of milliseconds since the UNIX epoch to a bona fide timestamp, e.g.
SELECT MSEC_TO_TIMESTAMP(1522865628160)
2018-04-04 11:13:48 UTC