I am trying to UNION ALL around 20 or so tables to consolidate into a single view. I keep getting an error that states:
'The numeric value XXXXX is not recognized'.
This error is contained in 1 column in each of the tables, but the data type for that column is VARCHAR(256) in each of the tables. No matter what I cast the column to I still get the same error.
The UNION ALL works perfectly if I comment that column out.
I've tried casting all columns to the same datatype, no luck.
I've tried commenting out the column in question, which works, but I need that column.
I've tried only UNION-ing a few of the tables, which sometimes works and sometimes doesn't, depending on the document type.
SELECT
CAST(QUICKBOOKS_MEXICO.BILL_LINE.DESCRIPTION AS VARCHAR(256)) AS DESCRIPTION
FROM QUICKBOOKS_MEXICO.BILL_LINE
UNION ALL
SELECT
CAST(QUICKBOOKS_EUROPE_BV.PURCHASE_LINE.DESCRIPTION AS VARCHAR(256)) AS DESCRIPTION
FROM QUICKBOOKS_EUROPE_BV.PURCHASE_LINE
The columns should seamlessly UNION.
Here is the error message:
Numeric value 'Exchange Gain Or Loss' is not recognized
It's worth mentioning that if I remove all the other fields BESIDES the column that is throwing the error from the query, it performs just fine. Truly baffling!
It seems to me that the error message that you get is not related to this column, but with another column where you might be casting the data as NUMERIC, but instead getting this value 'Exchange Gain Or Loss'.
One way to ignore this conversion error, is to use TRY_CAST instead of CAST, so when a value can not be converted to the intended data type, it will simply return NULL
Related
I'm working with a rather large Insert Into . . Select . . From . .
I have over 500+ lines of SQL in this script and I'm getting this error:
INSERT INTO MtgeMaster ( [Col1]
,[Col2]
,[Col3]
, etc., etc. )
SELECT [Col1]
,[Col2]
,[Col3]
, etc., etc.
FROM MtgeMktg
When I run the code above I get this error:
Msg 8114, Level 16, State 5, Line 164
Error converting data type varchar to numeric.
It looks like the error comes from line 164, but line 164 is literally my [Col1] field, and this is VARCHAR. So, I'm going from VARCHAR to VARCHAR. There is no VARCHAR to NUMERIC.
Also, if I add a couple of blank lines and re-run the process, I get this:
Msg 8114, Level 16, State 5, Line 166
Error converting data type varchar to numeric.
All it's really doing is going to the line with the INSERT INTO clause.
The error must be coming from another line, but it's hard to tell what's throwing the error when I have 500+ lines of SQL to go through.
SQL Server does not makes this easy. I have found that a brute force approach is necessary. I like to start by loading the data into a staging table where all the columns are strings. This makes it easier to manipulate.
You can use one of two methods to find the error. The first is to use try_convert() on each column to determine where the error is.
The second is to do a binary search to find the offending row. Load the first half of the data to see if the error is there. Then divide that half in half. And so on.
It looks like the error before and after adding few blank lines is same. It is probably a datatype conversion issue. You could try using cast() function and convert the data type of the [Col1] in select field of MtgeMktg table to match with the datatype of [Col1] of the MtgeMaster table.
So, here's my attempt to answer this question, based on what is given.
I would review the table structure of MtgeMaster, and see what columns are supposed to be numeric. Let's just say, for this example, Col3 of MtgeMaster is numeric. You may have multiple numeric columns.
I would then query MtgeMktg and check whether the column you're trying to save from MtgeMktg to MtgeMaster is numeric or not. You may have to do this for each column that is numeric in MtgeMaster. If I were doing it (which I did in SQLFiddle), it would be something to the nature of the following:
select * from MtgeMktg where ISNUMERIC(Col3) = 0
Anything that returns from this query will tell you what rows have a non numeric column.
Simple fiddle listed here:
Obviously, you would have 1 of 2 decisions to make at this point. Fix the data, or filter out the rows that have the non-numeric data in it. I presume you'd need the rows though.
i have subquery, that returns me varchar column, in some cases this column contains only numeric values and in this cases i need to cast this column to bigint, i`ve trying to use CAST(case...) construction, but CASE is an expression that returns a single result and regardless of the path it always needs to result in the same data type (or implicitly convertible to the same data type). Is there any tricky way to change column datatype depending on condition in PostgreSQL or not? google cant help me((
SELECT
prefix,
module,
postfix,
id,
created_date
FROM
(SELECT
s."prefix",
coalesce(m."replica", to_char(CAST((m."id_type" * 10 ^ 12) AS bigint) + m."id", 'FM0000000000000000')) "module",
s."postfix",
s."id",
s."created_date"
FROM some_subquery
There is really no way to do what you want.
A SQL query returns a fixed set of columns, with the names and types being fixed. So, a priori what you want to do does not fit well within SQL.
You could work around this, by inventing your own type, that is either a big integer or a string. You could store the value as JSON. But those are work-arounds. The SQL query itself is really returning one "type" for each column; that is how SQL works.
When performing a select query from a data base the returned result will have columns of a certain type.
If you perform a simple query like
select name as FirstName
from database
then the type of the resulting FirstName column will be that of database.name.
If you perform a query like
select age*income
from database
then the resulting data type will be that of the return value from the age*income expression.
What happens you use something like
select try_convert(float, mycolumn)
from database
where database.mycolumn has type of nvarchar. I assume that the resulting column has type of float which is decided by the return type of the first call to try_convert.
But consider this example
select coalesce(try_convert(float, mycolumn), mycolumn)
from database
which should give a column with the values of mycolumn unchanged if try_convert fails, but mycolumn as a float when/if that is possible.
Is this determination made as the first row is handled?
Or will the type always be determined by the function called independently of the data in the rows?
Is it possible to conditionally perform a conversion?
I would like to convert to float in the case where this is possible for all rows and leave unchanged in case it fails for any row.
Update 1
It seems that the answer to the first part of the question is that the column type is determined by the expression at compile time which means that you cannot have a dynamic type of your column depending on the data.
I see two workaround for this
Option 1
For each column count the number of not null rows of try_convert(float, mycolumn) and if this number is 0 then do not perform conversion. This will of course read the rows many times and might be inefficient.
Option 2
Simple repeat all columns; once without conversion and once with conversion and then simply use the interesting one.
One could also perform another select statement where only columns with non-null values are included.
Background
I have a dynamically generated pivot table with many (~200 columns) of which some have string values and others have numbers.
I would like to cast all columns as float where this is possible and leave the other columns unchanged (or cast as nvarchar).
The data
The data is mostly NULL values with some columns having text string and other columns having numbers. There are no columns with "mixed" content.
The types are determined at compile time, not at execution. try_convert(float, ...) knows exactly the type at parse/compile time, because float here is a keyword, not a value. As for expressions like COALESCE(foo, bar) the type similarly determined at compile time, following the rules of data type precedence lad already linked.
When you build your dynamic pivot you'll have to know the result type, using the same inference rules the SQL parser/compiler uses. I understand some rules are counter intuitive, when in doubt, test it out.
For the detail oriented: some expressions types can be determined at parse time, eg. N'foo'. But most have to be resolved at compile time, when the names of tables and columns are bind to actual object in the database, because only then the type is discovered.
I tried to search for previous articles related to this, but I can't find one specific to my situation. And because I'm brand new to StackOverflow, I can't post pictures so I'll try to describe it.
I have two datasets. One is 34 rows, 1 column of all NULLs. The other 13 rows, 1 column of varchars.
When I try to UNION ALL these two together, i get the following error:
Conversion failed when converting the varchar value to data type int.
I don't understand why I'm getting this error. I've UNIONed many NULL columns and varchar columns before, among many other types and I don't get this conversion error.
Can anyone offer suggestions why this error occurs?
The error occurs because you have corresponding columns in the two of the subqueries where the type of one is an integer and the type of the other is a character. Then, the character value has -- in at least one row -- a value that cannot be automatically converted to an integer.
This is easy to replicate:
select t.*
from (select 'A' as col union all
select 1
) t;
Here is the corresponding SQL Fiddle.
SQL Server uses pretty sophisticated type precedence rules for determining the destination type in a union. In practice, though, it is best to avoid using implicit type conversions. Instead, explicitly cast the columns to the type you intend.
EDIT:
The situation with NULL values is complicated. By itself, the NULL value has no type. So, the following works fine:
select NULL as col
union all
select 'A';
If you type the NULL, then the query will fail:
select cast(NULL as int) as col
union all
select 'A';
Also, if you put SQL Server in a position where it has to assign a type, then SQL Server will make the NULL an integer. Every column in a table or result set needs a type, so this will also fail:
select (select NULL) as col
union all
select 'A';
Perhaps your queries are doing something like this.
I have also encountered this error when I accidentally had the fields out of sequence in the 2 SELECT queries that I was unioning. Adjusting the fields' sequence fixed the problem.
The question
Is it possible to ask SSIS to cast a value and return NULL in case the cast is not allowed instead of throwing an error ?
My environment
I'm using Visual Studio 2005 and Sql Server 2005 on Windows Server 2003.
The general context
Just in case you're curious, here is my use case. I have to store data coming from somewhere in a generic table (key/value structure with history) witch contains some sort of value that can be strings, numbers or dates. The structure is something like this :
table Values {
Id int,
Date datetime, -- for history
Key nvarchar(50) not null,
Value nvarchar(50),
DateValue datetime,
NumberValue numeric(19,9)
}
I want to put the raw value in the Value column and try to put the same value
in the DateValue column when i'm able to cast it to Datetime
in the NumberValue column when i'm able to cast it to a number
Those two typed columns would make all sort of aggregation and manipulation much easier and faster later.
That's it, now you know why i'm asking this strange question.
============
Thanks in advance for your help.
You could also try a Derived Column component and test the value of the potential date/number field or simply cast it and redirect any errors as being the NULL values for these two fields.
(1) If you just simply cast the field every time with a statement like this in the Derived Column component: (DT_DATE)[MYPOTENTIALDATE] - you can redirect the rows that fail this cast and manipulate the data from there.
OR
(2) You can do something like this in the Derived Column component: ISNULL([MYPOTENTIALDATE]) ? '2099-01-01' : (DT_DATE)[MYPOTENTIALDATE]. I generally send through '2099-01-01' when a date is NULL rather than messing with NULL (works better with Cubes, etc).
Of course (2) won't work if the [MYPOTENTIALDATE] field comes through as other things other than a DATETIME or NULL, i.e., sometimes it is a word like "hello".
Those are the options I would explore, good luck!
In dealing with this same sort of thing I found the error handling in SSIS was not specific enough. My approach has been to actually create an errors table, and query a source table where the data is stored as varchar, and log errors to the error table with something like the below. I have one of the below statements for each column, because it was important for me to know which column failed. Then after I log all errors, I do a INSERT where I select those records in SomeInfo that do not have an errors. In your case you could do more advanced things based on the ColumnName in the errors table to insert default values.
INSERT INTO SomeInfoErrors
([SomeInfoId]
,[ColumnName]
,[Message]
,FailedValue)
SELECT
SomeInfoId,
'PeriodStartDate',
'PeriodStartDate must be in the format MM/DD/YYYY',
PeriodStartDate
FROM
SomeInfo
WHERE
ISDATE(PeriodStartDate) = 0 AND [PeriodStartDate] IS NOT NULL;
Tru using a conditional split and have the records where the data is a date go along one path and the other go along a different path where they are updated to nullbefore being inserted.