My table has a few invalid timestamps that I think are too precise (maybe beyond microseconds) for BigQuery.
When I tried updating the table with the following query using Standard SQL:
UPDATE mytable
SET event_time = TIMESTAMP(DATETIME(TIMESTAMP_MILLIS(CAST(SUBSTR(SAFE_CAST(UNIX_MILLIS(event_time) AS string),1,13) AS int64))))
WHERE DATE(logtime) BETWEEN "2018-03-21" AND "2018-03-23"
AND event_time IS NOT NULL
I get the invalid timestamp error:
Cannot return an invalid timestamp value of 1521738691071000064 microseconds relative to the Unix epoch. The range of valid timestamp values is [0001-01-1 00:00:00, 9999-12-31 23:59:59.999999]; error in writing field event_time
I think the problem is the SET event_time = part, but I don't know how to get around setting the values in the event_time column without referring to it.
Anyone have any ideas on how to resolve this?
Necessity is the mother of invention. For anyone who has a similar issue, I've figured out a workaround.
Create a new table of the affected rows (include this in the WHERE clause: LENGTH(CAST(UNIX_MILLIS(event_time) as string)) > 13, while transforming the
invalid timestamp into a valid format using TIMESTAMP(DATETIME(TIMESTAMP_MILLIS(CAST(SUBSTR(SAFE_CAST(UNIX_MILLIS(event_time) AS string),1,13) AS int64))))
Delete the affected rows from the original table using the WHERE clause mentioned above.
Insert all rows from the new table back into the original table.
A lot more work, but it should do the trick.
Related
We are specifying all our datetime/timestamp datatypes as TIMESTAMP_LTZ while creating Snowflake tables.
for one of the values in Oracle source '0001-01-01 00:00:00.000001000' we are observing that when this value gets inserted in Snowflake tables this changes to '0000-12-31 23:52:58.000001000' .
There are no issues with other datetime/timestamp values except this.
There is no changes and alter to timezones when the query runs.
Is this some conversion issue due to to timezone or bug at Snowflake side?
This can happen based on the timezone set for your session/user. More details for Timezone parameter can be found here.
https://docs.snowflake.com/en/sql-reference/parameters.html#timezone
Please use below link to get the timezone set for your session/user/account.
https://docs.snowflake.com/en/sql-reference/parameters.html#viewing-the-parameters-and-their-values
show parameters like '%zone%';
alter session set TIMEZONE = 'America/New_York';
CREATE or replace TABLE time_test (timeval TIMESTAMP_LTZ);
INSERT INTO time_test values('0001-01-01 00:00:00.000001000');
select * from time_test;
Output of the select query is 0000-12-31 23:56:02.000.
I am trying to find the time difference between two time stamps and store it in a column. When I check the output in the table, I see the value to be a huge number and not the difference in day/hours. I am using amazon redshift as the database.
Data_type of time_duration column : varchar
Given below is the sample:
order_no,order_date,complain_date,time_duration
1001,2018-03-10 04:00:00,2018-03-11 07:00:00,97200000000
But I am expecting time_duration column to show 1 day,3 hours
This issue happens when I store the time_duration in a table and then query to view the output.
Could anyone assist.
Thanks.
Do it following way, it will give hours difference
select datediff(hour,order_date,complain_date) as diff_in_hour from your_table ;
If you want to do it day, do it following way
select datediff(day,order_date,complain_date) as diff_in_day from your_table;
You could use datediff function to update your table column time_duration.
Refer Redshift documentation for more details.
I get the following error when running this simple query in Bigquery with standard SQL:
SELECT MIN(created) as mm FROM `projectId.ds.User`
Column created has type cloud.helix.Timestamp, which differs from the expected type of INT64 Dismiss
COMPOSE QUERY
The created field has datatype Timestamp, is nullable, but contains no null values.
This query works however in Legacy SQL:
SELECT MIN(created) as mm FROM [projectId:ds.User]
Any advice?
A fix has been rolled out by Google addressing this error.
https://code.google.com/p/google-bigquery/issues/detail?id=841
Hi I created a table in which one column is of date type and also works as PK. I tried to insert value 2009-01-07 into this column and had this error. Isn't Date default format yyyy-mm-dd? I don't understand this.
Msg 241, Level 16, State 1, Line 3
Conversion failed when converting date and/or time from character string.
This is my query:
INSERT INTO Table_Name
Values ('2009-01-07', other column values)
Your value '2009-01-07' should be converted.
Date literals are always a deep source of troubles... Best was, to use either
Unseparated: 20090107
ODBC : {d'2009-01-07'}
ISO8601 : 2009-01-07T00:00:00
But your format is short ISO 8601 and should work...
Some possible reasons:
Other values in your VALUES list
a trigger
a constraint
As a_horse_with_no_name stated in comment: Without a column list after INSERT INTO Table(col1, col2, ...) There is a great risk to state your values in a different order thus pairing values with the wrong columns...
Invalid (but good looking) dates such as 2016-06-31
Or a - well known - issue with SQL-Server. Sometimes the order of execution is absolutely not the way one expected it. There are several issues with conversion errors...
What you can try
Use ODBC format (which is treated as DATETIME immediately)
DECLARE a variable with this value and put it in place
Thank you all for the prompt replies. I read and tried all of them and found out why.
'2009-01-07' can be inserted into a Column with "Date" as data type if no CONSTRAINT has issue with that;
my problem was caused by a CHECK constraint on that column.
Originally I set CONSTRAINT as
Column_Name = 'Wednesday'
After I modified it to
DATEName(dw,[Column_Name]) = 'Wednesday'
the inserting began to work.
Thanks again.
I'm creating an incremental load which would be pulling data from ORACLE to SQL Server. The incremental load will be based off a MODIFIED_DATE column.
I have created a result set variable that stores the MAX modified_date from the destination table. So the engine will only check the rows of the MODIFIED_DATES that are greater than the variable and perform a lookup to see if the row needs to be added, updated or deleted.
So I have my MAX MODIFIED DATE RESULT SET and I also have created another variable that will house the SOURCE QUERY which will be have a WHERE clause that see if the MODIFIED_DATE column is greater than the MAX MODIFIED_DATE variable.
Example:
Select column_name,column_name
From table
Where modified_date > '"+ #[User::LastModifiedDate]+ "'"
It is throwing me an error off:
The data types "DT_WSTR" and "DT_DATE" are incompatible for binary operator "+". The operand types could not be implicitly cast into compatible types for the operation. To perform this operation, one or both operands need to be explicitly cast with a cast operator.
Now, I have done a ton of searching but I cant seem to find a way to do this. The only solution that I found online is to ADD A (DT_WSTR, 25) in front of the variable which causes the variable expression to evaluate and this is the only way I can get the variable expression to evaluate.
Example:
(DT_WSTR, 25) #[User::LastModifiedDate]+ "'"
When I run it it is telling me it is NOT A VALID MONTH
The MODIFIED_DATE column in the DESTINATION table is in SQL Server and it has a DataTime as the date type which reads like this:
2008-06-10 22:22:25.000
YYYY-MM-DD
The MODIFIED_DATE column in the SOURCE table in oracle reads like:
6/10/2008 10:22:25 PM
MM/DD/YY HH:MM:SS
How would I be able to resolve this? Also what do you think is the best way to perform an incremental load based the MODIFIED_DATE column? Is my way one of the more efficient ways or is there another route I can take?
You need to make your SSIS component call the following statement:
Select column_name,column_name
From table
Where modified_date > to_date('whateverformat','"+ (DT_WSTR,25)#[User::LastModifiedDate]+ "')"
The problem is you are tangling the strings versus dates. Your lastmodifieddate must be a string for the expression builder to function, but i suspect Oracle is expecting modified_date to be a date, so just use the to_date function