I have a timestamp from the source that has been loaded to BQ as a string. I'd like to write a query in BigQuery that will return timestamp in the following format 2020-01-06 11:09:14.000-0600. Here is the current format of the string field: 2020-01-06T11:09:14.000-0600, 2018-10-01T15:45:59.000-0500, etc.
I have tried the following:
SELECT parse_timestamp ("%Y-%m-%dT%H:%M:%S.%E3S", start_timestamp, "America/Chicago"), FROM bqtable
The goal is to perform arithmetic on the timestamp fields.
Any feedback is appreciated. Thank you.
I think the %S and %E3S% are conflicting, as they both are parsing the seconds part of the string.
Try this:
with data as (
select '2020-01-06T11:09:14.000-0600' as ts_string union all select '2018-10-01T15:45:59.000-0500'
)
select ts_string, parse_timestamp ("%Y-%m-%dT%H:%M:%E3S%z", ts_string, "America/Chicago") as ts
from data
Related
I am trying to insert the data into the final table in snowflake from the staging table. When the command is run it give the error:
Can't parse '20211101132344205550' as timestamp with format 'YYYYMMDD HH24:MI:SS.FF'
My table definition and insert statement is here.
I used the same kind of method last time it worked. Thank you so much in advance.
CREATE OR REPLACE TABLE dw.tb_fidctp_order(
sysdate DATE,
record_id NUMBER(18,0) ,
timestamp TIMESTAMP_NTZ(9),
<trim_excess>
);
INSERT INTO dw.tb_fidctp_order(
sysdate,
record_id,
timestamp,
<trim_excess>
)
SELECT
TO_DATE(LEFT(timestamp, 8), 'YYYYMMDD')
,CAST(record_id AS NUMBER(18,0))
,TO_TIMESTAMP(LEFT(timestamp,24),'YYYYMMDD HH24:MI:SS.FF')
<trim_excess>
FROM stg.tb_fidctp_order_input;
In Snowflake you need to define what your format is. So, if all of your timestamps are formatted as a straight string like that, then you are defining it incorrectly. Try something more like this:
SELECT to_timestamp(left(EXPIRY_DATETIME,24),'YYYYMMDDHH24MISSFF');
The to_timestamp() function is defining how the input string is formatted, not how you want the timestamp to be formatted as an output.
So the error message is the critical point, your formating string for timestamps have space and : time formatting, which needs to be removed.
Below I have used the tru_to_timestamp function because it returns NULL istead of erroring with is helpful to show the problem is your formatting:
SELECT
'20211101132344205550' AS a
,try_to_timestamp(a, 'YYYYMMDD HH24:MI:SS.FF') as b
,try_to_timestamp(a, 'YYYYMMDDHH24MISSFF') as c;
gives:
A
B
C
20211101132344205550
2021-11-01 13:23:44.205
which shows your ,TO_TIMESTAMP(LEFT(timestamp,24),'YYYYMMDD HH24:MI:SS.FF')
should be:
,TO_TIMESTAMP(LEFT(timestamp,24),'YYYYMMDDHH24MISSFF')
and then you would have no problem.
want to convert this number '20210412070422' to date format '2021-04-12' in hive
I am trying but this returns null value
from_unixtime(unix_timestamp(eap_as_of_dt, 'MM/dd/yyyy'))
The best methoid is to do without unix_timestamp/from_unixtime if possible and in your case it is possible. date() can be removed, string in yyyy-MM-dd format is compatible with date type:
select date(concat_ws('-',substr(ts,1,4),substr(ts,5,2),substr(ts,7,2)))
from
(
select '20210412070422' as ts
)s
Result:
2021-04-12
Another efficient method using regexp_replace:
select regexp_replace(ts,'^(\\d{4})(\\d{2})(\\d{2}).*','$1-$2-$3')
If you prefer using unix_timestamp/from_unixtime
select date(from_unixtime(unix_timestamp(ts, 'yyyyMMddHHmmss')))
from
(
select '20210412070422' as ts
)s
But it is more complex, slower (SimpleDateFormat class is involved) and error prone because will not work if data is not exactly in expected format, for example '202104120700'
Of course you can make it more reliable by taking substring of required length and using yyyyMMdd template:
select date(from_unixtime(unix_timestamp(substr(ts,1,8), 'yyyyMMdd')))
from
(
select '20210412070422' as ts
)s
It makes it even more complex.
Use unix_timestamp/from_unixtime only if simple substr or regexp_replace do not work for data format like '2021Apr12blabla'.
I have a date column as dd-mm-yyyy. I would like to convert it to yyyy/mm/dd in bigquery.I have written the following query:
SELECT cast(format(Date, 'yyyy/mm/dd') as string) as Date FROM t1.
The error is : Too many arguments to FORMAT for pattern "23/04/2020"; Expected 1; Got 2.
Can you please assist.
First you need to parse date from dd-mm-yyyy string and then format it as yyyy/mm/dd as in below
FORMAT_DATE('%Y/%m/%d', PARSE_DATE('%d-%m-%Y', day))
You can test, play with above using dummy data as in below example
#standardSQL
WITH `project.dataset.table` AS (
SELECT '15-01-2020' day UNION ALL
SELECT '05-10-2019'
)
SELECT day, FORMAT_DATE('%Y/%m/%d', PARSE_DATE('%d-%m-%Y', day)) AS formated_day
FROM `project.dataset.table`
with output
Row day formated_day
1 15-01-2020 2020/01/15
2 05-10-2019 2019/10/05
You want FORMAT_DATE, not FORMAT.
SELECT FORMAT_DATE("%Y/%m/%d", DATE "2008-12-25");
The reason you're having trouble with FORMAT is that you gave it a format string that doesn't take any parameters. Seeing this, the engine barfs-"I don't need any more parameters to render this string."
In presto, I have a date formatted as varchar that looks like below :
10:46:00
I need to cast this in timestamp. I have tried few but presto throwing errors as
Value cannot be cast to date:10:46:00 and Value cannot be cast to
timestamp:10:46:00
select cast('10:46:00' as DATE) from abc;
select cast('10:46:00' as TIMESTAMP) from abc;
Try with the below query it will solve your problem.
Input Query in Presto:
select (hour(date_parse(CheckStartTime,'%T')) + 1) as hr from TableName;
CheckStartTime:
Column name(varchar) of the table in the format of '12:32:20'.
Output:
13 (it will add one hour to the input time)
My table in hive has a filed of date in the format of '2016/06/01'. but i find that it is not in harmory with the format of '2016-06-01'.
They can not compare for instance.
Both of them are string .
So I want to know how to make them in harmory and can compare them. Or on the other hand, how to change the '2016/06/01' to '2016-06-01' so that them can compare.
Many thanks.
To convert date string from one format to another you have to use two date function of hive
unix_timestamp(string date, string pattern) convert time string
with given pattern to unix time stamp (in seconds), return 0 if
fail.
from_unixtime(bigint unixtime[, string format]) converts the
number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a
string representing the timestamp of that moment in the current
system time zone.
Using above two function you can achieve your desired result.
The sample input and output can be seen from below image:
The final query is
select from_unixtime(unix_timestamp('2016/06/01','yyyy/MM/dd'),'yyyy-MM-dd') from table1;
where table1 is the table name present in my hive database.
I hope this help you!!!
Let's say you have a column 'birth_day' in your table which is in your format,
you should use the following query to convert birth_day into the required format.
date_Format(birth_day, 'yyyy-MM-dd')
You can use it in a query in the following way
select * from yourtable
where
date_Format(birth_day, 'yyyy-MM-dd') = '2019-04-16';
Use :
unix_timestamp(DATE_COLUMN, string pattern)
The above command would help convert the date to unix timestamp format which you may format as you want using the Simple Date Function.
Date Function
cast(to_date(from_unixtime(unix_timestamp(yourdate , 'MM-dd-yyyy'))) as date)
here is my solution (for string to real Date type):
select to_date(replace('2000/01/01', '/', '-')) as dt ;
ps:to_date() returns Date type, this feature needs Hive 2.1+; before 2.1, it returns String.
ps2: hive to_date() function or date_format() function , or even cast() function, cannot regonise the 'yyyy/MM/dd' or 'yyyymmdd' format, which I think is so sad, and make me a little crazy.