I have over 8 billion data in big query and it has a cloumn named EdgeEndTimestamp and its type is STRING. I dived into documentation, and I can not find a way to get data according to this field.
for example I want to get all data between 2021-06-01T20:45:00.104Z and 2021-06-01T21:15:00.104Z
Timestamp of EdgeEndTimestamp is like 2022-07-25T11:07:15Z
Since there is too much data in there what would be the most efficient way?
You can convert the string to a timestamp with TIMESTAMP() and use it in the WHERE clause.
Example:
WITH your_table AS
(
SELECT '2021-06-01T20:45:00.104Z' AS dt, 123 AS val
UNION ALL
SELECT '2021-06-01T21:15:00.104Z', 456
UNION ALL
SELECT '2021-06-01T21:45:00.104Z', 789
)
SELECT *
FROM your_table
WHERE TIMESTAMP(dt) BETWEEN TIMESTAMP('2021-06-01T20:45:00.104Z') AND TIMESTAMP('2021-06-01T21:15:00.104Z')
Related
I am trying to write an sql on snowflake that checks if a column data type (value) is timestamp
id
value
111
2022-01-14 15:03:43:000
select value from cat
where is_date(value)
if value is timestamp, to return 1 and 0 if its not a timestamp
You could use TRY_TO_DATE/TRY_TO_TIMESTAMP functions:
SELECT value
FROM tab
WHERE TRY_TO_DATE(value) IS NOT NULL;
If a specific date format is expected it could be provided i.e.: TRY_TO_DATE(value, 'dd-MM-yyyy')
You can use below format and try_to_timestamp returns NULL in case if it is invalida data and also you can apply format of Timestamp.
select try_to_timestamp('2022-01-14 15:03:43','YYYY-MM-DD HH24:MI:SS'), try_to_date('Invalid');
want to convert this number '20210412070422' to date format '2021-04-12' in hive
I am trying but this returns null value
from_unixtime(unix_timestamp(eap_as_of_dt, 'MM/dd/yyyy'))
The best methoid is to do without unix_timestamp/from_unixtime if possible and in your case it is possible. date() can be removed, string in yyyy-MM-dd format is compatible with date type:
select date(concat_ws('-',substr(ts,1,4),substr(ts,5,2),substr(ts,7,2)))
from
(
select '20210412070422' as ts
)s
Result:
2021-04-12
Another efficient method using regexp_replace:
select regexp_replace(ts,'^(\\d{4})(\\d{2})(\\d{2}).*','$1-$2-$3')
If you prefer using unix_timestamp/from_unixtime
select date(from_unixtime(unix_timestamp(ts, 'yyyyMMddHHmmss')))
from
(
select '20210412070422' as ts
)s
But it is more complex, slower (SimpleDateFormat class is involved) and error prone because will not work if data is not exactly in expected format, for example '202104120700'
Of course you can make it more reliable by taking substring of required length and using yyyyMMdd template:
select date(from_unixtime(unix_timestamp(substr(ts,1,8), 'yyyyMMdd')))
from
(
select '20210412070422' as ts
)s
It makes it even more complex.
Use unix_timestamp/from_unixtime only if simple substr or regexp_replace do not work for data format like '2021Apr12blabla'.
I have a timestamp from the source that has been loaded to BQ as a string. I'd like to write a query in BigQuery that will return timestamp in the following format 2020-01-06 11:09:14.000-0600. Here is the current format of the string field: 2020-01-06T11:09:14.000-0600, 2018-10-01T15:45:59.000-0500, etc.
I have tried the following:
SELECT parse_timestamp ("%Y-%m-%dT%H:%M:%S.%E3S", start_timestamp, "America/Chicago"), FROM bqtable
The goal is to perform arithmetic on the timestamp fields.
Any feedback is appreciated. Thank you.
I think the %S and %E3S% are conflicting, as they both are parsing the seconds part of the string.
Try this:
with data as (
select '2020-01-06T11:09:14.000-0600' as ts_string union all select '2018-10-01T15:45:59.000-0500'
)
select ts_string, parse_timestamp ("%Y-%m-%dT%H:%M:%E3S%z", ts_string, "America/Chicago") as ts
from data
I have a column called 'created_date' whose data type is string. It contains records that are of the pattern date and time. I want to create another column called 'modified_date' that will take just the date from the 'created_date' column so as to be able to do some mathematical computations on dates later. I want to do this using the SQL CAST operator.
Below is how I expect the output to be-
ID created_date modified_date
1 2017-11-01 16:30:40 2017-11-01
2 2017-11-23 15:30:40 2017-11-23
3 2017-11-16 14:30:40 2017-11-16
Any suggestions on how to do this?
I am going to assume that you are using BigQuery.
You can use:
select date(created_date)
You could also be more specific:
select date(substr(created_date, 1, 10))
Or convert to a datetime and then to a date:
select date(cast(created_date as datetime))
You could use a simple date_format() and str_to_date()
select date_format(str_to_date(created_date,'%Y-%m-%d %T'), '%Y-%m-%d') modified_date
Below is for BigQuery Standard SQL
#standardSQL
SELECT *, DATE(TIMESTAMP(created_date)) modified_date
FROM `project.dataset.table`
I want to do this using the SQL CAST operator.
Note: i do not recommend using generic CAST for DATE, DATETIME, TIMESTAMP data types. Instead you should use respective functions as in this answer. Or if string is not directly represent such datatypes - you can use respective PARSE_ function where you can set format in which date/datetime/timestamp is represented in string!
I have a field in a BigQuery table that is a STRING but is actually a timestamp, of the format "2019-06-14T11:31:07". So I am using CAST(sign_up_date AS TIMESTAMP) to convert to a usable TIMESTAMP.
This works perfectly in Legacy SQL, however, in Standard SQL, it brings up errors when the STRING is of the format "2019-06-14T09:09" (on the exact minute, missing ":00") or "2019-05-25T05:31:22.7263555" (as sometimes they come through with decimal seconds).
Any idea on how I can get it to work in Standard SQL? Obviously I could just use Legacy SQL, but I want to write in Standard as there are other functionns that work better in that one.
Thanks,
Benji
Below is example for BigQuery Standard SQL
#standardSQL
WITH `project.dataset.table` AS (
SELECT "2019-06-14T11:31:07" sign_up_date UNION ALL
SELECT "2019-05-25T05:31:22.7263555" UNION ALL
SELECT "2019-06-14T09:09"
)
SELECT sign_up_date,
COALESCE(
SAFE.PARSE_TIMESTAMP('%FT%R', sign_up_date),
SAFE.PARSE_TIMESTAMP('%FT%R:%E*S', sign_up_date)
) AS sign_up_date_as_timestamp
FROM `project.dataset.table`
with result
Row sign_up_date sign_up_date_as_timestamp
1 2019-06-14T11:31:07 2019-06-14 11:31:07 UTC
2 2019-05-25T05:31:22.7263555 2019-05-25 05:31:22.726355 UTC
3 2019-06-14T09:09 2019-06-14 09:09:00 UTC
As you can see this will cover all three patters you presented in your question.
If you will find more - you can add respective SAFE.PARSE_TIMESTAMP inside COALESCE