SELECT ride_time, CAST (ride_time as INTERVAL) as ride_time_hr
ride_time -> str 00:07:35.000000
I am expecting 00:07:35
You can use the PARSE_TIME function:
SELECT ride_time, PARSE_TIME("%H:%M:%E*S", ride_time) as ride_time_hr
Output:
More info:
PARSE_TIME: https://cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#parse_time
Format elements: https://cloud.google.com/bigquery/docs/reference/standard-sql/format-elements#format_elements_date_time
Related
I am using an sql script to parse a json into a snowflake table using dbt.
One of the cols contain this datetime value: '2022-02-09T20:28:59+0000'.
What's the correct way to define ISO datetime's data type in Snowflake?
I tried date, timestamp and TIMESTAMP_NTZ like this in my dbt sql script:
JSON_DATA:",my_date"::TIMESTAMP_NTZ AS MY_DATE
but clearly, these aren't the correct one because later on when I test it in snowflake with select * , I get this error:
SQL Error [100040] [22007]: Date '2022-02-09T20:28:59+0000' is not recognized
or
SQL Error [100035] [22007]: Timestamp '2022-02-13T03:32:55+0100' is not recognized
so I need to know which Snowflake time/date data type suits the best for this one
EDIT:
This is what I am trying now.
SELECT
JSON_DATA:"date_transmission" AS DATE_TRANSMISSION
, TO_TIMESTAMP(DATE_TRANSMISSION:text, 'YYYY-MM-DDTHH24:MI:SS.FFTZH:TZM') AS DATE_TRANSMISSION_TS_UTC
, JSON_DATA:"authorizerClientId"::text AS AUTHORIZER_CLIENT_ID
, JSON_DATA:"apiPath"::text API_PATH
, MASTERCLIENT_ID
, META_FILENAME
, META_LOAD_TS_UTC
, META_FILE_TS_UTC
FROM {{ source('INGEST_DATA', 'TABLENAME') }}
I get this error:
000939 (22023): SQL compilation error: error line 6 at position 4
10:21:46 too many arguments for function [TO_TIMESTAMP(GET(DATE_TRANSMISSION, 'text'), 'YYYY-MM-DDTHH24:MI:SS.FFTZH:TZM')] expected 1, g
However, if I comment out the the first 2 lines(related to timpstamp types), the other two work perfectly fine. What's the correct syntax of parsing json with TO_TIMESTAMP?
Not that JSON_DATA:"apiPath"::text API_PATH gives the correct value for it in my snowflake tables.
Did some testing and it seems you have 2 options.
You can either get rid of the +0000 at the end: left(column_date, len(column_date)-5)
or try_to_timestamp with format
try_to_timestamp('2022-02-09T20:28:59+0000','YYYY-MM-DD"T"HH24:MI:SS+TZHTZM')
TZH and TZM are TimeZone Offset Hours and Minutes
So there are 2 main points here.
when getting data from JSON to pass to any of the timestamp functions that want a ::TEXT object, but the values to get from JSON are still ::VARIANT so they need to be cast. This is the cause of the error you quote
(22023): SQL compilation error: error line 6 at position 4
10:21:46 too many arguments for function [TO_TIMESTAMP(GET(DATE_TRANSMISSION, 'text'), 'YYYY-MM-DDTHH24:MI:SS.FFTZH:TZM')] expected 1, g
also your SQL is wrong there it should have been
TO_TIMESTAMP(DATE_TRANSMISSION::text,
How you handle the timezone format.As other have noted you (as I did in your last question) do you want to ignore the timezone values or read them. I forgot about the TZHTZM formatting. Given you have timezone data, you should use the TO_TIMESTAMP_TZ`TRY_TO_TIMESTAMP_TZto make sure the time zone data is keep, given you second example shows+0100`
putting those together (assuming you didn't want an extra date_transmission as a variant in you data) :
SELECT
TO_TIMESTAMP_TZ(JSON_DATA:"date_transmission"::text, 'YYYY-MM-DDTHH24:MI:SS+TZHTZM') AS DATE_TRANSMISSION_TS_UTC
, JSON_DATA:"authorizerClientId"::text AS AUTHORIZER_CLIENT_ID
, JSON_DATA:"apiPath"::text AS API_PATH
, MASTERCLIENT_ID
, META_FILENAME
, META_LOAD_TS_UTC
, META_FILE_TS_UTC
FROM {{ source('INGEST_DATA', 'TABLENAME') }}
You should use timestamp (not date which does not store the time information), but probably the format you are using is not autodetected. You can specify the input format as YYYY-MM-DD"T"HH24:MI:SSTZHTZM as shown here. The autodetected one has a : between the TZHTZM.
copy into DB.CANONICAL.ETAIL_TABLE(MAKE, SERIAL_NUMBER, DEVICE_KEY, CONFIG_TYPE, CSF, MESSAGE_TIMESTAMP)
from (select parse_json($1):header:make::Varchar(255),
parse_json($1):header:serialNumber::Varchar(255),
parse_json($1):header:deviceKey::Varchar(36),
parse_json($1):header:configType::String,
parse_json($1),
parse_json($1):header:messageTimestamp::TIMESTAMP_NTZ(2)
from #DB.CANONICAL.ETAIL_STAGE/firehose/)
ON_ERROR = CONTINUE;
Above is the copy statement that I have made. But I keep getting the following error
"Failed to cast variant value "2020-09-265T08:03:42.358Z" to TIMESTAMP_NTZ"
There's a problem with the input data: September 265 is not a date.
If you want to parse that date to null, you can use TRY_CAST instead.
If you want to attempt to fix the date (September 26? September 25?), you could try going through a regex first.
SELECT '2020-09-265T08:03:42.358Z'::TIMESTAMP_NTZ;
# Timestamp '2020-09-265T08:03:42.358Z' is not recognized
SELECT '2020-09-26T08:03:42.358Z'::TIMESTAMP_NTZ;
# 2020-09-26 08:03:42.358
SELECT TRY_CAST('2020-09-265T08:03:42.358Z' AS TIMESTAMP_NTZ);
# null
i am trying cast field with date function.
raw_data = LOAD '/user/cloudera/Chicago_Traffic_Tracker_- _Historical_Congestion_Estimates_by_Region.csv' USING PigStorage(',') AS ( TIME :chararray,REGION_ID:int,BUS_COUNT:int,NUMBER_OF_READS:int,SPEED:double);
raw_clean = FOREACH raw_data GENERATE ToDate(raw_data.TIME,'yyyy/MM/dd HH:mm:ss')as date_time:DateTime ;
I get the below error
Scalar has more than one row in the output. 1st :
(01/29/2015 01:40:35 PM,22,33,429,25.23), 2nd :(01/05/2015 01:10:46 PM,18,58,1058,21.14)
Input
01/29/2015 01:40:35 PM,22,33,429,25.23,a61e11c83f811b63e1dc64362f799dcac322fca8
01/05/2015 01:10:46 PM,18,58,1058,21.14,39c63427d0e1401a06f967fd43c30e291140c26e
Didn't try practicals: But Your Input date is in format 01/29/2015
01:40:35 i.e MM/dd/YYYY HH:mm:ss . Whereas you have specified it as
'yyyy/MM/dd HH:mm:ss'
Try something like :
raw_clean = FOREACH raw_data GENERATE ToDate(raw_data.TIME,'MM/dd/YYYY HH:mm:ss');
I used the ToDate(userinput, format) function to covert my chararray field. I used the ToDate(userinput, 'MM/dd/yyyy') to covert the field from chararray to date but looks like i am not seeing the output as i had expected.
Here is the code:
l_dat = load 'textfile' using PigStorage('|') as (first:chararray,last:chararray,dob:chararray);
c_dat = foreach l_dat generate ToDate(dob,'MM/dd/yyyy') as mydate;
describe c_dat;
dump c_dat;
data looks like this:
(firstname1,lastname1,02/02/1967)
(John,deloy,05/26/1967)
(frank,fun,05/18/1967)
Output looks like this:
c_dat: {mydate: datetime}
(1967-05-26T00:00:00.000-04:00)
(1967-05-18T00:00:00.000-04:00)
(1967-02-02T00:00:00.000-05:00)
The output i was expecting was dateObjects with data as shown below:
(05/26/1967)
(05/18/1967)
(02/02/1967)
Please advise if i am doing anything wrong?
Ref : http://pig.apache.org/docs/r0.12.0/func.html#to-date, the return type of ToDate function is DateTime object. You can observe that in the schema description shared in output
c_dat: {mydate: datetime}
If you are having the date in the required format, you need not do any conversion.
c_dat = foreach l_dat generate dob as mydate;
If you are interested in converting the chararray date to any other format then you have to use ToString() function after getting the DateTime object.
Step 1: Convert date chararray to Date Time Ojbect using ToDate(datesstring, inutformat)
Step 2 : Use ToString(DateTime object, required format) to get the string date in the required format.
This can be achieved in a single step as below.
ToString(ToDate(date,inputformat),requiredformat);
Ref : http://pig.apache.org/docs/r0.12.0/func.html#to-string for details.
I have one date chararray 2014-11-02 16:57:29. i want to convert this into long how can i do this?
I tried from the link
https://pig.apache.org/docs/r0.14.0/api/org/apache/pig/piggybank/evaluation/datetime/convert/ISOToUnix.html
DEFINE ISOToUnix org.apache.pig.piggybank.evaluation.datetime.convert.ISOToUnix();
DEFINE CustomFormatToISO org.apache.pig.piggybank.evaluation.datetime.convert.CustomFormatToISO();
ISOin = LOAD '/user/sample/test.csv' USING PigStorage() AS (dt:chararray, dt2:chararray);
toUnix = FOREACH ISOin GENERATE ISOToUnix(CustomFormatToISO(dt, 'YYYY-MM-DD HH:MM:SS')) AS event_time;
DUMP toUnix;
test.csv contains
2014-11-02 16:57:29 2014-11-02 13:13:05
I am getting
[POUserFunc (Name:
POUserFunc(org.apache.pig.piggybank.evaluation.datetime.convert.ISOToUnix)[long]
- scope-5 Operator Key: scope-5) children: null at []]: java.lang.NullPointerException.
Could somebody please help me out. Thanks
ISOToUnix(CONCAT(REPLACE(dt, ' ', 'T'), '.000Z')) looks a good an quick answer. here we convert the string to iso then using ISOToUnix using valid timezone