Generating external tables in SnowFlake works very well, but when you use the dbt to do this, it generates errors in the validation of fields that come null for timestamp_ntz.
CREATE OR REPLACE EXTERNAL TABLE EX_USERS
( deleted_at timestamp_ntz as (NULLIF(value :deleted_at, '')::timestamp_ntz)
dbt
name: deleted_at data_type: timestamp_ntz description: "deleted_at"
Failed to cast variant value "" to TIMESTAMP_NTZ
Instead of using ::timestamp TRY_TO_TIMESTAMP_NTZ could be used:
deleted_at timestamp_ntz AS (TRY_TO_TIMESTAMP_NTZ(some_col) )
When you get a Failed to cast error back from Snowflake, the value returned in the error message is (unfortunately) trimmed of spaces. This means that it might not be '' that is your deleted_at but it could be ' ' or ' ' etc.
In other words, your NULLIF might be comparing one (or more) spaces against the empty string. In Snowflake, this comparison will return false. For example
SELECT NULLIF(value :deleted_at, '')::timestamp_ntz
FROM (SELECT {'deleted_at':' '} as value)
will return
SQL Error [100071] [22000]: Failed to cast variant value " " to TIMESTAMP_NTZ
Now, you could use TRY_TO_TIMESTAMP_NTZ, but that would mean silently ignoring any "real" errors (such as attempting to convert say 2023-02-29 to a timestamp. Therfore, it might be preferable to code this
NULLIF(TRIM(value:deleted_at), '')::timestamp_ntz
or, you could use the TRIM_SPACE option on the CREATE EXTERNAL TABLE if you are happy for all input columns to be trimmed
Related
I need to check whether the values received in a file upload column has exponential or long values.
For example, if value is 5.02E+13 instead of numeric value - 50100434157080 then need to restrict it with a message saying format is incorrect.
For this I pass the upload content from frontend to backend in a temporary table and then get it checked if passed value has exponential value or numeric value.
Tried using T-SQL function isnumeric() but it didn't give me expected result. Any other function available?
Since you mention using isnumeric I assume you are using SQL Server, in which case you can try try_cast, for example
select case when Try_Cast(Column as bigint) is null then 'not integer' else 'integer' end
from table
You could also use like
select case when Column like '%e%' then 'exponent' else 'number' end
from table
I have a simple view that takes a varchar and converts it to a float.
Select CAST(TRIM(Measurement) AS float) as Measurement
from MyTable
where Type = 'Some Value'
Now the view sees the column as a float, however when i try to query my view with a where clasue of Measurement = 10, then I get the error:
Error converting data type varchar to float.
I know the string value will always be a number (I have checked many times). I also got this through multiple rounds of testing before this error popped up in prod. My guess is that there are other measurements in the table that are not part of my results, but are causing this error (those measurements may not have existing during the testing).
Is there a way to clean up the query so the results are always treated as a float?
Thanks
You clearly have bad values in the column. You can find them using a regular expression (in most databases) or a try_ function in SQL Server.
For instance:
Select Measurement
from MyTable
where Type = 'Some Value' AND try_convert(float, measurement) is null;
Or:
where type = 'Some Value' and
not measurement ~ '^-?[0-9]*[.]?[0-9]+$'
Th ~ is Postgres for a regular expression match. Other databases have similar functionality with different syntax.
I'm trying to load some data from stage to relational environment and something is happening I can't figure out.
I'm trying to run the following query:
SELECT
CAST(SPLIT_PART(some_field,'_',2) AS BIGINT) cmt_par
FROM
public.some_table;
The some_field is a column that has data with two numbers joined by an underscore like this:
some_field -> 38972691802309_48937927428392
And I'm trying to get the second part.
That said, here is the error I'm getting:
[Amazon](500310) Invalid operation: Invalid digit, Value '1', Pos 0,
Type: Long
Details:
-----------------------------------------------
error: Invalid digit, Value '1', Pos 0, Type: Long
code: 1207
context:
query: 1097254
location: :0
process: query0_99 [pid=0]
-----------------------------------------------;
Execution time: 2.61s
Statement 1 of 1 finished
1 statement failed.
It's literally saying some numbers are not valid digits. I've already tried to get the exactly data which is throwing the error and it appears to be a normal field like I was expecting. It happens even if I throw out NULL fields.
I thought it would be an encoding error, but I've not found any references to solve that.
Anyone has any idea?
Thanks everybody.
I just ran into this problem and did some digging. Seems like the error Value '1' is the misleading part, and the problem is actually that these fields are just not valid as numeric.
In my case they were empty strings. I found the solution to my problem in this blogpost, which is essentially to find any fields that aren't numeric, and fill them with null before casting.
select cast(colname as integer) from
(select
case when colname ~ '^[0-9]+$' then colname
else null
end as colname
from tablename);
Bottom line: this Redshift error is completely confusing and really needs to be fixed.
When you are using a Glue job to upsert data from any data source to Redshift:
Glue will rearrange the data then copy which can cause this issue. This happened to me even after using apply-mapping.
In my case, the datatype was not an issue at all. In the source they were typecast to exactly match the fields in Redshift.
Glue was rearranging the columns by the alphabetical order of column names then copying the data into Redshift table (which will
obviously throw an error because my first column is an ID Key, not
like the other string column).
To fix the issue, I used a SQL query within Glue to run a select command with the correct order of the columns in the table..
It's weird why Glue did that even after using apply-mapping, but the work-around I used helped.
For example: source table has fields ID|EMAIL|NAME with values 1|abcd#gmail.com|abcd and target table has fields ID|EMAIL|NAME But when Glue is upserting the data, it is rearranging the data by their column names before writing. Glue is trying to write abcd#gmail.com|1|abcd in ID|EMAIL|NAME. This is throwing an error because ID is expecting a int value, EMAIL is expecting a string. I did a SQL query transform using the query "SELECT ID, EMAIL, NAME FROM data" to rearrange the columns before writing the data.
Hmmm. I would start by investigating the problem. Are there any non-digit characters?
SELECT some_field
FROM public.some_table
WHERE SPLIT_PART(some_field, '_', 2) ~ '[^0-9]';
Is the value too long for a bigint?
SELECT some_field
FROM public.some_table
WHERE LEN(SPLIT_PART(some_field, '_', 2)) > 27
If you need more than 27 digits of precision, consider a decimal rather than bigint.
If you get error message like “Invalid digit, Value ‘O’, Pos 0, Type: Integer” try executing your copy command by eliminating the header row. Use IGNOREHEADER parameter in your copy command to ignore the first line of the data file.
So the COPY command will look like below:
COPY orders FROM 's3://sourcedatainorig/order.txt' credentials 'aws_access_key_id=<your access key id>;aws_secret_access_key=<your secret key>' delimiter '\t' IGNOREHEADER 1;
For my Redshift SQL, I had to wrap my columns with Cast(col As Datatype) to make this error go away.
For example, setting my columns datatype to Char with a specific length worked:
Cast(COLUMN1 As Char(xx)) = Cast(COLUMN2 As Char(xxx))
I am using in insert statement to convert BDE table (source) to a Firebird table (destination) using IB Datapump. So the INSERT statement is fed by source table values via parameters. One of the source field parameters is alphanum (SOURCECHAR10 char(10), holds mostly integers and needs to be converted to integer in the (integer type) destination column NEWINTFLD. If SOURCECHAR10 is not numeric, I want to assign 0 to NEWINTFLD.
I use IIF and SIMILAR to to test whether the string is numeric, and assign 0 if not numeric as follows:
INSERT INTO "DEST_TABLE" (......, "NEWINTFLD",.....)
VALUES(..., IIF( :"SOURCECHAR10" SIMILAR TO '[[:DIGIT:]]*', :"SOURCECHAR10", 0),..)
For every non numeric string however, I still get conversion errors (DSQL error code = -303).
I tested with only constants in the IIF result fields like SOURCECHAR10" SIMILAR TO '[[:DIGIT:]]*', 1, 0) and that works fine so somehow the :SOURCECHAR10 in the true result field of the IIF generates the error.
Any ideas how to get around this?
When your query is executed, the parser will notice that second use of :"SOURCECHAR10" is used in a place where an integer is expected. Therefor it will always convert the contents of :SOURCECHAR10 into an integer for that position, even though it is not used if the string is non-integer.
In reality Firebird does not use :"SOURCECHAR10" as parameters, but your connection library will convert it to two separate parameter placeholders ? and the type of the second placeholder will be INTEGER. So the conversion happens before the actual query is executed.
The solution is probably (I didn't test it, might contain syntax errors) to use something like (NOTE: see second example for correct solution):
CASE
WHEN :"SOURCECHAR10" SIMILAR TO '[[:DIGIT:]]*'
THEN CAST(:"SOURCECHAR10" AS INTEGER)
ELSE 0
END
This doesn't work as this is interpreted as a cast of the parameter itself, see CAST() item 'Casting input fields'
If this does not work, you could also attempt to add an explicit cast to VARCHAR around :"SOURCECHAR10" to make sure the parameter is correctly identified as being VARCHAR:
CASE
WHEN :"SOURCECHAR10" SIMILAR TO '[[:DIGIT:]]*'
THEN CAST(CAST(:"SOURCECHAR10" AS VARCHAR(10) AS INTEGER)
ELSE 0
END
Here the inner cast is applied to the parameter itself, the outer cast is applied when the CASE expression is evaluated to true
I am using the LIST function to create a ';' delimited list of values. The type is numeric (19,2). For some reason the precision appears to be ignored when using the list function. When performing a simple select on this column the values look good, ie "12.00". However, if I use a LIST() my results are of format "12.000000"
This is my LIST usage:
LIST(case when tblWOService.PricePerVehicle is null then ' ' else CONVERT(decimal(19,2),tblWOService.PricePerVehicle end,';')
The CONVERT does not change the result. Any ideas?
Thanks!
Have you tried explicitly converting your empty string?
LIST(
case when tblWOService.PricePerVehicle is null then CONVERT(decimal(19,2),' ')
else CONVERT(decimal(19,2),tblWOService.PricePerVehicle) end,';'
)
I've run into a similar datatype issue with CASE statements in T-SQL.