How to prevent CAST errors on SSIS? - sql

The question
Is it possible to ask SSIS to cast a value and return NULL in case the cast is not allowed instead of throwing an error ?
My environment
I'm using Visual Studio 2005 and Sql Server 2005 on Windows Server 2003.
The general context
Just in case you're curious, here is my use case. I have to store data coming from somewhere in a generic table (key/value structure with history) witch contains some sort of value that can be strings, numbers or dates. The structure is something like this :
table Values {
Id int,
Date datetime, -- for history
Key nvarchar(50) not null,
Value nvarchar(50),
DateValue datetime,
NumberValue numeric(19,9)
}
I want to put the raw value in the Value column and try to put the same value
in the DateValue column when i'm able to cast it to Datetime
in the NumberValue column when i'm able to cast it to a number
Those two typed columns would make all sort of aggregation and manipulation much easier and faster later.
That's it, now you know why i'm asking this strange question.
============
Thanks in advance for your help.

You could also try a Derived Column component and test the value of the potential date/number field or simply cast it and redirect any errors as being the NULL values for these two fields.
(1) If you just simply cast the field every time with a statement like this in the Derived Column component: (DT_DATE)[MYPOTENTIALDATE] - you can redirect the rows that fail this cast and manipulate the data from there.
OR
(2) You can do something like this in the Derived Column component: ISNULL([MYPOTENTIALDATE]) ? '2099-01-01' : (DT_DATE)[MYPOTENTIALDATE]. I generally send through '2099-01-01' when a date is NULL rather than messing with NULL (works better with Cubes, etc).
Of course (2) won't work if the [MYPOTENTIALDATE] field comes through as other things other than a DATETIME or NULL, i.e., sometimes it is a word like "hello".
Those are the options I would explore, good luck!

In dealing with this same sort of thing I found the error handling in SSIS was not specific enough. My approach has been to actually create an errors table, and query a source table where the data is stored as varchar, and log errors to the error table with something like the below. I have one of the below statements for each column, because it was important for me to know which column failed. Then after I log all errors, I do a INSERT where I select those records in SomeInfo that do not have an errors. In your case you could do more advanced things based on the ColumnName in the errors table to insert default values.
INSERT INTO SomeInfoErrors
([SomeInfoId]
,[ColumnName]
,[Message]
,FailedValue)
SELECT
SomeInfoId,
'PeriodStartDate',
'PeriodStartDate must be in the format MM/DD/YYYY',
PeriodStartDate
FROM
SomeInfo
WHERE
ISDATE(PeriodStartDate) = 0 AND [PeriodStartDate] IS NOT NULL;

Tru using a conditional split and have the records where the data is a date go along one path and the other go along a different path where they are updated to nullbefore being inserted.

Related

Change Datatype of json_value during select into so I can sum column

I have a column in a table that is json. It contains several columns within it.
Example:
Row1: "sTCounts":[{"dpsTypeTest":"TESTTRIAL","cnt":3033244.0}
Row2: "sTCounts":[{"dpsTypeTest":"TESTTRIAL","cnt":3.3}
I need to sum the cnt value for all rows in table. For instance, the above would produce a result of 3033247.3
I'm not familiar with stored procs enough to master. I thought the easiest route would be to create a temp table and extract the value into a column, and then write a query to sum the column values.
The problem is that it creates a column with datatype nvarchar(4000). It won't let me sum that column. I thought of changing the datatype but not sure how. I am trying CAST without luck.
select CAST(json AS varchar) AS JSON_VALUE(jsontext,
'$.sTCounts.cnt') AS PerfCount, TitleNumber
INTO dbo_Testing_Count0
from PerformanceTest
select sum(PerfCount)
from dbo_Testing_Count
Group by PerfCount
The error message is:
Incorrect syntax near 'jsontext'.
Any ideas? I am open to another method to sum the column or changing the datatype whichever the experts can aid on. I appreciate it.
The JSON you provide in your question is not valid... This seems to be just a fragment of a larger JSON. As your data starts with a [ you have to think of it as an array, so the simple json path '$.serviceTierCounts.cnt' won't work probably...
Try this, I've added the opening { and the closing brackets at the end:
DECLARE #mockupTable TABLE(ID INT IDENTITY, YourJson NVARCHAR(MAX));
INSERT INTO #mockupTable VALUES
(N'{"serviceTierCounts":[{"dpsType":"TRIAL","cnt":3033244.0}]}')
,(N'{"serviceTierCounts":[{"dpsType":"TRIAL","cnt":3.3}]}');
--You can read one scalar value using JSON_VALUE directly with a cast. But in this case I need to add [0]. This will tell the engine to read the first (zero-based index!) object's cnt property.
SELECT CAST(JSON_VALUE(YourJson,'$.serviceTierCounts[0].cnt') AS DECIMAL(14,4))
FROM #mockupTable
--But I think, that it's this what you are looking for:
SELECT *
FROM #mockupTable
CROSS APPLY OPENJSON(YourJson,'$.serviceTierCounts')
WITH(dpsType varchar(100)
,cnt decimal(14,4));
The WITH clause will return the object in typed columns side-by-side.
For easy proceeding, you can wrap this as a CTE and continue with the set in the following SELECT.

Return either blank or numeric value

I have looked and looked and can not find an answer anywhere so I am hoping you guys can help me. I am pulling data from multiple tables using a left join. Some info is not in the right table so it pulls across as NULL. The end user is requesting that all NULL values show as blank. I can easily do this for columns with string datatypes but I have not been able to figure out a way to do this for columns with numeric, int, money, or float datatypes.
See screenshot for simplistic example:
Using SQL Sever 2012
Probably not the answer you want to hear but you might need to convert the numeric columns in your result set to a string type (nvarchar, varchar...etc) and then put a CASE WHEN structure in your SELECT around each one of those columns and check if the value of the field is NULL then print '' the blank string.
Try this;
select
NumberNull=case when ISNUMERIC(YourField)=1 then cast(YourField AS varchar(100)) else '' end

Alter column from varchar to decimal when nulls exist

How do I alter a sql varchar column to a decimal column when there are nulls in the data?
I thought:
ALTER TABLE table1
ALTER COLUMN data decimal(19,6)
But I just get an error, I assume because of the nulls:
Error converting data type varchar to numeric. The statement has been terminated.
So I thought to remove the nulls I could just set them to zero:
ALTER TABLE table1
ALTER COLUMN data decimal(19,6) NOT NULL DEFAULT 0
but I dont seem to have the correct syntax.
Whats the best way to convert this column?
edit
People have suggested it's not the nulls that are causing me the problem, but non-numeric data. Is there an easy way to find the non-numeric data and either disregard it, or highlight it so I can correct it.
If it were just the presence of NULLs, I would just opt for doing this before the alter column:
update table1 set data = '0' where data is null
That would ensure all nulls are gone and you could successfully convert.
However, I wouldn't be too certain of your assumption. It seems to me that your new column is perfectly capable of handling NULL values since you haven't specified not null for it.
What I'd be looking for is values that aren't NULL but also aren't something you could turn in to a real numeric value, such as what you get if you do:
insert into table1 (data) values ('paxdiablo is good-looking')
though some may argue that should be treated a 0, a false-y value :-)
The presence of non-NULL, non-numeric data seems far more likely to be causing your specific issue here.
As to how to solve that, you're going to need a where clause that can recognise whether a varchar column is a valid numeric value and, if not, change it to '0' or NULL, depending on your needs.
I'm not sure if SQL Server has regex support but, if so, that'd be the first avenue I'd investigate.
Alternatively, provided you understand the limitations (a), you could use isnumeric() with something like:
update table1 set data = NULL where isnumeric(data) = 0
This will force all non-numeric values to NULL before you try to convert the column type.
And, please, for the love of whatever deities you believe in, back up your data before attempting any of these operations.
If none of those above solutions work, it may be worth adding a brand new column and populating bit by bit. In other words set it to NULL to start with, and then find a series of updates that will copy data to this new column.
Once you're happy that all data has been copied, you should then have a series of updates you can run in a single transaction if you want to do the conversion in one fell swoop. Drop the new column and then do the whole lot in a single operation:
create new column;
perform all updates to copy data;
drop old column;
rename new column to old name.
(a) From the linked page:
ISNUMERIC returns 1 for some characters that are not numbers, such as plus (+), minus (-), and valid currency symbols such as the dollar sign ($).
Possible solution:
CREATE TABLE test
(
data VARCHAR(100)
)
GO
INSERT INTO test VALUES ('19.01');
INSERT INTO test VALUES ('23.41');
ALTER TABLE test ADD data_new decimal(19,6)
GO
UPDATE test SET data_new = CAST(data AS decimal(19,6));
ALTER TABLE test DROP COLUMN data
GO
EXEC sp_RENAME 'test.data_new' , 'data', 'COLUMN'
As people have said, that error doesn't come from nulls, it comes from varchar values that can't be converted to decimal. Most typical reason for this I've found (after checking that the column doesn't contain any logically false values, like non-digit characters or double comma values) is when your varchar values use comma for decimal pointer, as opposed to period.
For instance, if you run the following:
DECLARE #T VARCHAR(256)
SET #T = '5,6'
SELECT #T, CAST(#T AS DEC(32,2))
You will get an error.
Instead:
DECLARE #T VARCHAR(256)
SET #T = '5,6'
-- Let's change the comma to a period
SELECT #T = REPLACE(#T,',','.')
SELECT #T, CAST(#T AS DEC(32,2)) -- Now it works!
Should be easy enough to look if your column has these cases, and run the appropriate update before your ALTER COLUMN, if this is the cause.
You could also just use a similar idea and make a regex search on the column for all values that don't match digit / digit+'.'+digit criteria, but i suck with regex so someone else can help with that. :)
Also, the american system uses weird separators like the number '123100.5', which would appear as '123,100.5', so in those cases you might want to just replace the commas with empty strings and try then?

Get first or second values from a comma separated value in SQL

I have a column that stores data like (42,12). Now I want to fetch 42 or 12 (two different select queries). I have searched and found some similar but much more complex scenarios. Is there any easy way of doing it? I am using MSSQL Server 2005.
Given there will always be only two values and they will be integer
The reason you have this problem is because the database (which you may not have any control over), violates first normal form. Among other things, first normal form says that each column should hold a single value, not multiple values. This is bad design.
Now, having said this, the first solution that pops into my head is to write a UDF that parses the value in this column, based on the delimiter and returns either the first or second value.
You can try something like this
DECLARE #Table TABLE(
Val VARCHAR(50)
)
INSERT INTO #Table (Val) SELECT '42,12'
SELECT *,
CAST(LEFT(Val,CHARINDEX(',',Val)-1) AS INT) FirstValue,
CAST(RIGHT(Val,LEN(Val) - CHARINDEX(',',Val)) AS INT) SecondValue
FROM #Table
You can use something like this:
SELECT SUBSTRING_INDEX(field, ',', 1)
Note: Its not the efficient way of doing things in rdbms. Consider normalizing your Database.

Forcing a datatype in MS Access make table query

I have a query in MS Access which creates a table from two subqueries. For two of the columns being created, I'm dividing one column from the first subquery into a column from the second subquery.
The datatype of the first column is a double; the datatype of the second column is decimal, with scale of 2, but I want the second column to be a double as well.
Is there a way to force the datatype when creating a table through a standard make-table Access query?
One way to do it is to explicitly create the table before putting anything into it.
Your current statement is probably like this:
SELECT Persons.LastName,Orders.OrderNo
INTO Persons_Order_Backup
FROM Persons
INNER JOIN Orders
ON Persons.P_Id=Orders.P_Id
WHERE FirstName = 'Alistair'
But you can also do this:
----Create NewTable
CREATE TABLE NewTable(FirstName VARCHAR(100), LastName VARCHAR(100), Total DOUBLE)
----INSERT INTO NewTableusing SELECT
INSERT INTO NewTable(FirstName, LastName, Total)
SELECT FirstName, LastName,
FROM Person p
INNER JOIN Orders o
ON p.P_Id = o.P_Id
WHERE p.FirstName = 'Alistair'
This way you have total control over the column types. You can always drop the table later if you need to recreate it.
You can use the cast to FLOAT function CDBL() but, somewhat bizarrely, the Access Database Engine cannot handle the NULL value, so you must handle this yourself e.g.
SELECT first_column,
IIF(second_column IS NULL, NULL, CDBL(second_column))
AS second_column_as_float
INTO Table666
FROM MyTest;
...but you're going to need to ALTER TABLE to add your keys, constraints, etc. Better to simply CREATE TABLE first then use INSERT INTO..SELECT to populate it.
You can use CDbl around the columns.
An easy way to do this is to create an empty table with the correct field types and then to an Append-To query and Access will automatically convert the data to the destination field.
I had a similar situation, but I had a make-table query creating a field with NUMERIC datatype that I wanted to be short text.
What I did (and I got the idea from Stack) is to create the table with the field in question as Short Text, and at the same time build a delete query to scrub the records. I think it's funny that a DELETE query in access doesn't delete the table, just the records in it - I guess you have to use a DROP TABLE function for that, to purge a table...
Then, I converted my make-table query to an APPEND query, which I'd never done before... and I just added the running of the DELETE query to my process.
Thank you, Stack Overflow !
Steve
I add a '& ""' to the field I want to make sure are stored as text, and a ' *1 ' (as in multiplying the amount by 1) to the fields I want to store as numeric.
Seems to do the trick.
To get an Access query to create a table with three numeric output fields from input numeric fields, (it kept wanting to make the output fields text fields), had to combine several of the above suggestions. Pre-establish an empty output table with pre-defined output fields as integer, double and double. In the append query itself, multiply the numeric fields by one. It worked. Finally.