I am trying to bulk-load data with timestamps into DB2 UDB version 9 and it is not working out.
My target table has been created thusly
CREATE TABLE SHORT
(
ACCOUNT_NUMBER INTEGER
, DATA_UPDATE_DATE TIMESTAMP DEFAULT current timestamp
, SCHEDULE_DATE TIMESTAMP DEFAULT current timestamp
...
0)
This is the code for my bulk load
load from /tmp/short.csv of del modified by COLDEL, savecount 500 messages /users/chris/DATA/short.msg INSERT INTO SHORT NONRECOVERABLE DATA BUFFER 500
No rows are loaded. I get an error
SQL0180N The syntax of the string representation of a datetime value is
incorrect. SQLSTATE=22007
I tried forcing the timestamp format when I create my table thusly:
CREATE TABLE SHORT
(
ACCOUNT_NUMBER INTEGER
, DATA_UPDATE_DATE TIMESTAMP DEFAULT 'YYYY-MM-DD HH24:MI:SS'
, SCHEDULE_DATE TIMESTAMP DEFAULT 'YYYY-MM-DD HH24:MI:SS'
...
0)
but to no avail: SQL0574N DEFAULT value or IDENTITY attribute value is not valid for column. Timestamp format in source my data file looks like this
2002/06/18 17:11:02.000
How can I format my timestamp columns?
Use the TIMESTAMPFORMAT file type modifier for the LOAD utility to specify the format present in your data files.
load from /tmp/short.csv
of del
modified by COLDEL, TIMESTAMPFORMAT="YYYY/MM/DD HH:MM:SS.UUUUUU"
savecount 500
messages /users/chris/DATA/short.msg
INSERT INTO SHORT
NONRECOVERABLE
It almost looks like ISO to me.
If you are able to edit the data file and can safely change the slashes to hyphens, (maybe clip the decimal fraction) and DB2 accepts ISO, you're home.
Related
I am currently having a varchar tm column, that stores timestamps such as: '15.11.2021 11:07:27'
The datestyle is currently set to ISO,MDY
How can I transform that varchar using SELECT value in order to preserve its format?
If I use to_timestamp(tm, 'DD.MM.YYYY HH24:MI:SS')::timestamp without time zone I still get it in other format 2021-11-15 11:07:27
I also tried to do double to_timestamps, but then an error function to_timestamp(timestamp with time zone, unknown) does not exist appears.
So the question is: Is there any way to convert from varchar type "15.11.2021 11:07:27" to timestamp type 15.11.2021 11:07:27 using select statement?
DbFiddle
PS. Even though I used it in DBFiddle, I cant change datestyle on the target server
Postgres version 13.5
EDIT: also, if I use cast(tm as timestamp) I receive date/time field value out of range: "15.11.2021 11:07:27" error.
You could try using:
CONVERT(varchar, timestamp, 103) AS YOUR_VALUE
This way you would convert timestamp without changing its original form.
I am trying to insert the data into the final table in snowflake from the staging table. When the command is run it give the error:
Can't parse '20211101132344205550' as timestamp with format 'YYYYMMDD HH24:MI:SS.FF'
My table definition and insert statement is here.
I used the same kind of method last time it worked. Thank you so much in advance.
CREATE OR REPLACE TABLE dw.tb_fidctp_order(
sysdate DATE,
record_id NUMBER(18,0) ,
timestamp TIMESTAMP_NTZ(9),
<trim_excess>
);
INSERT INTO dw.tb_fidctp_order(
sysdate,
record_id,
timestamp,
<trim_excess>
)
SELECT
TO_DATE(LEFT(timestamp, 8), 'YYYYMMDD')
,CAST(record_id AS NUMBER(18,0))
,TO_TIMESTAMP(LEFT(timestamp,24),'YYYYMMDD HH24:MI:SS.FF')
<trim_excess>
FROM stg.tb_fidctp_order_input;
In Snowflake you need to define what your format is. So, if all of your timestamps are formatted as a straight string like that, then you are defining it incorrectly. Try something more like this:
SELECT to_timestamp(left(EXPIRY_DATETIME,24),'YYYYMMDDHH24MISSFF');
The to_timestamp() function is defining how the input string is formatted, not how you want the timestamp to be formatted as an output.
So the error message is the critical point, your formating string for timestamps have space and : time formatting, which needs to be removed.
Below I have used the tru_to_timestamp function because it returns NULL istead of erroring with is helpful to show the problem is your formatting:
SELECT
'20211101132344205550' AS a
,try_to_timestamp(a, 'YYYYMMDD HH24:MI:SS.FF') as b
,try_to_timestamp(a, 'YYYYMMDDHH24MISSFF') as c;
gives:
A
B
C
20211101132344205550
2021-11-01 13:23:44.205
which shows your ,TO_TIMESTAMP(LEFT(timestamp,24),'YYYYMMDD HH24:MI:SS.FF')
should be:
,TO_TIMESTAMP(LEFT(timestamp,24),'YYYYMMDDHH24MISSFF')
and then you would have no problem.
I have the target field type in hive as timestamp and from the source I get the json that has either proper timestamp field or "" or null sometimes. I am converting the source JsonToAvro before using PutHiveStreaming processor. The records with proper timestamp format gets into my hive target table successfully. But those that with ""/null (Empty String set) values show the error - Illegal format. Timestamp format should be" YYYY-MM-DD HH:MM:SS[.fffffffff] ". I know if I can default it to some date when it is null/empty, it works.But I do not want that. I want it to be as null in my target table when it is null. How can I achieve this?
I have a JSON dataset in HDFS that contains a timestamp and a count. The raw data looks like this:
{"timestamp": "2015-03-01T00:00:00+00:00", "metric": 23}
{"timestamp": "2015-03-01T00:00:01+00:00", "metric": 17}
...
The format of the timestamp almost matches the Hive-friendly 'yyyy-mm-dd hh:mm:ss' format but has a couple of differences: there's a 'T' between the date and time. There's also a timezone offset. For example, a timestamp might be 2015-03-01T00:00:00+00:00 instead of 2015-03-01 00:00:00.
I'm able to create a table, providing that I treat the timestamp column as a string:
add jar hdfs:///apps/hive/jars/hive-json-serde-0.2.jar;
CREATE EXTERNAL TABLE `log`(
`timestamp` string,
`metric` bigint)
ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde" WITH SERDEPROPERTIES ("timestamp"="$.timestamp", "metric"="$.metric")
LOCATION 'hdfs://path/to/my/data';
This isn't ideal since, by treating it like a string, we lose the ability to use timestamp functions (e.g. DATE_DIFF, DATE_ADD, etc...) without casting from within the query. A possible workaround would be to CTAS and CAST the timestamp using a regular expression, but this entails copying the data into its new format. This seems inefficient and not in the spirit of 'schema-on-read'.
Is there a way to create a schema for this data without processing the data twice (i.e. once to load, once to convert the timestamp to a true timestamp)?
You will need to decide whether to:
do CTAS as you described
push the conversion work/logic into the consumers/clients of the table
For the second option this means including the string to timestamp conversion in the sql statements executed against your external table.
Hi All i'm having an issue loading a file with timestamp values with sqlldr, can anyone please help with an idea?
This is what a line of my file to load with the timestamp looks like :
..... 2014-09-02-00.00.00.
And my ctrl file :
...
...
Field POSITION(1085) TIMESTAMP )
Every record is rejected with the error:
ORA-26041: DATETIME/INTERVAL datatype conversion error
SQL*Loader will try to interpret the string as a timestamp using your default NLS session parameters, which by default will be something like:
select value from nls_session_parameters where parameter = 'NLS_TIMESTAMP_FORMAT';
VALUE
--------------------------------------------------------------------------------
DD-MON-RR HH24.MI.SSXFF
... although yours appears to be something else as that setting gives ORA-01843: not a valid month' with the example string. The log file from the load will also show the format it tried to apply:
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
"<field>" 1085 * , DATETIME DD-MON-RR HH24.MI.SSXFF
Whatever your default format is, it doesn't seem to match the format in the file. You could change the format for the database or with a logon trigger but that's not going to be helpful if you have other sources or outputs that expect the existing format; and you shouldn't rely on NLS settings anyway as they could be different in another environment.
You can supply the timestamp format as part of the field definition:
... Field POSITION(1085) TIMESTAMP "YYYY-MM-DD-HH24.MI.SS")
If your column is timestamp with time zone - suggested by the tags but not the question - then it will be inserted in your session's time zone.
It's more common to specify the position range, though just providing the start position 1085 will work if this is the last field in the record.