I have an SSIS package which transfers data from SAS to SQL Server. I'm creating a derived column but cannot get the REPLACENULL feature to work. I am receiving the error
"Invalid character value for cast specification"
which I am sure is because of NULL values in the source. Here is my current derived column expression:
REPLACENULL(DATEADD("d",(DT_I8)AuthEndDate,(DT_DATE)"1960-01-01"),0)
The REPLACENULL function isn't working here. Any way that this can be done? I am using SSIS 2008. Thanks.
Since you want blank/0 to flow downstream you need an if, not a replacenull()
Please note blank/0 in SQL Server writes as 1900-01-01, blank/0 are not valid date values, so it defaults to this. The only other option is NULL.
This should work if the destination is a datetime column:
ISNULL(AuthEndDate) ? (DT_DATE)0 : DATEADD("d",(DT_I8)AuthEndDate,(DT_DATE)"1960-01-01")
If you opt to write NULL instead of 1900-01-01, you can do this:
ISNULL(AuthEndDate) ? NULL(DT_DATE) : DATEADD("d",(DT_I8)AuthEndDate,(DT_DATE)"1960-01-01")
You must use one of the following expressions:
DATEADD("d",(DT_I8)REPLACENULL([AuthEndDate],0),(DT_DATE)"1960-01-01")
OR
ISNULL([AuthEndDate]) ? (DT_DATE)"1960-01-01" : DATEADD("d",(DT_I8)AuthEndDate,(DT_DATE)"1960-01-01")
Note that you cannot use a logic that can return multiple datatypes, both cases must return on datatype (DT_DATE)
If the source is returning NULLs, then try a REPLACENULL at AuthEndDate rather than after the calculations. The statement you have will try to do calculations with NULLs which is never good.
So...
REPLACENULL(DATEADD("d",REPLACENULL((DT_I8)AuthEndDate,0),(DT_DATE)"1960-01-01"),0)
Edit - replacenull and then apply conversion:
DATEADD("d",(DT_I8)REPLACENULL(AuthEndDate,0),(DT_DATE)"1960-01-01")
You won't need the first REPLACENULL anymore since we now handle it before the calculations.
Therefore:
DATEADD("d",REPLACENULL((DT_I8)AuthEndDate,0),(DT_DATE)"1960-01-01")
Edit: You can change the 0 to whatever you want the NULL to become, 0 was a placeholder.
Related
Folks
I am in the process of moving a decade old back-end from DB2 9.5 to Oracle 19c.
I frequently see in SQL queries and veiw definitions bizarre timestamp(nullif('','')) constructs used instead of a plain null.
What is the point of doing so? Why would anyone in their same mind would want to do so?
Disclaimer: my SQL skills are fairly mediocre. I might well miss something obvious.
It appears to create a NULL value with a TIMESTAMP data type.
The TIMESTAMP DB2 documentation states:
TIMESTAMP scalar function
The TIMESTAMP function returns a timestamp from a value or a pair of values.
TIMESTAMP(expression1, [expression2])
expression1 and expression2
The rules for the arguments depend on whether expression2 is specified and the data type of expression2.
If only one argument is specified it must be an expression that returns a value of one of the following built-in data types: a DATE, a TIMESTAMP, or a character string that is not a CLOB.
If you try to pass an untyped NULL to the TIMESTAMP function:
TIMESTAMP(NULL)
Then you get the error:
The invocation of routine "TIMESTAMP" is ambiguous. The argument in position "1" does not have a best fit.
To invoke the function, you need to pass one of the required DATE, TIMESTAMP or a non-CLOB string to the function which means that you need to coerce the NULL to have one of those types.
This could be:
TIMESTAMP(CAST(NULL AS VARCHAR(14)))
TIMESTAMP(NULLIF('',''))
Using NULLIF is more confusing but, if I have to try to make an excuse for using it, is slightly less to type than casting a NULL to a string.
The equivalent in Oracle would be:
CAST(NULL AS TIMESTAMP)
This also works in DB2 (and is even less to type).
It is not clear why - in any SQL dialect, no matter how old - one would use an argument like nullif('',''). Regardless of the result, that is a constant that can be calculated once and for all, and given as argument to timestamp(). Very likely, it should be null in any dialect and any version. So that should be the same as timestamp(null). The code you found suggests that whoever wrote it didn't know what they were doing.
One might need to write something like that - rather than a plain null - to get null of a specific data type. Even though "theoretical" SQL says null does not have a data type, you may need something like that, for example in a view, to define the data type of the column defined by an expression like that.
In Oracle you can use the cast() function, as MT0 demonstrated already - that is by far the most common and most elegant equivalent.
If you want something much closer in spirit to what you saw in that old code, to_timestamp(null) will have the same effect. No reason to write something more complicated for null given as argument, though - along the lines of that nullif() call.
I have a table that holds drafted data. Most of the values on the table are nullable. I want to return an empty string in the case the value is null and I'm grabbing and copying the draft data onto the frontend. What would be the best way to handle this? I understand I can use ISNULL(variable, '') but don't want it to do that for every single value. Or a ternary on the front end to return a empty string if null. Would there be a better way to handle this?
You can have a trigger on your table before insert or update and populate the fields you want with an empty value for all values that match your "treat null as empty" rule.
In my opinion changing data model to satisfy UI requirements is not a great idea. I would rather have a transformer in the middle (or a custom RowMappeer) that will do the transformation on the data already pulled from the database.
Also note that for some databases (such as Oracle) an empty string and a null are interchangeable in which case my suggestion to use a trigger won't work.
I know you said you did not want to use ISNULL but this is the main reason this function exists so I would rather start using it. If you have lots of such fields you can always use a clever editor to help you build your query using ISNULL rather than typing.
You can use the nvl sql function, which allows you to change null value for any value you want.
I am learning how to add the check constraint, but it did not really work as I expected.
So, I have a column limit, which actually require some constraints, as some of the user broke our system when they enter record back end.
Column 'limit':
Constraint expected : If not NULL or Empty, then record must be float/numeric.
Can anyone help to solve this?
Thanks in advance!
You can use triggers to check numeric using ISNUMERIC(), and throw error if not. But this is bad design because you are not using database constraint but trigger validation.
I still suggest you add a new checklist column where user select by themself if CK_limit is empty or not.
I think this is what you are looking for SQL_VARIANT_PROPERTY.
Or you can use client side validations or you can use stored procedures to track this.
Do not use isnumeric() for the constraint. For instance, isnumeric() accepts the following as valid numbers:
'$'
'.'
'-'
'3e4'
Instead, use try_convert(). You can do:
(try_convert(float, limit) is not null or
limit is null or
limit = ''
)
However, the right solution is to simply use the correct data type for the column. Don't store numeric values in strings.
I am trying to subset my data with PROC SQL, and it is giving me an error when I use my variable TNM_CLIN_STAGE_GROUP. Example below:
PROC SQL;
create table subset as
select ncdb.*
from ncdb
where YEAR_OF_DIAGNOSIS>2002
AND SEX = 2
AND LATERALITY IN (1,2,3)
AND HISTOLOGY = 8500
AND TNM_CLIN_STAGE_GROUP = 1;
quit;
ERROR: Expression using equals (=) has components that are of different data types.
When I run the same code, but take out the variable TNM_CLIN_STAGE_GROUP, the code works. Anyone know what the problem with that variable's name is?
That error indicates a difference in type. SAS has only two types, numeric and character, so the variable is probably character; verify the specific values, but in general it likely needs quotations (single or double, doesn't matter in this case).
If it is not a hardcoded value, but a value of another variable, you can use PUT to convert to character or INPUT to convert to numeric, whichever is easier to convert based on the data.
SAS in a data step will happily convert this for you, but in SQL and SQL-like (WHERE statements) it does not automatically convert character to numeric and vice versa; you must provide the correct type.
Before doing equality, check what you are trying to compare.
Check the structure of you ncbd table, in particulary field type of TNM_CLIN_STAGE_GROUP
You would see the real type, if its a varchar, you need to use single quote like #JChao suggest in is comment.
If its another type, so you need to adapt the comparator or use cast if you don t have choice.
I am developing an SSIS 2008 package and I am trying to create a Derived Column transformation. But when I go to the Expression editor and try this expression it gives me alot of errors. I have tried various differentiations of this but all have resulted in errors. I need one of you SQL experts to point out a better expression!
ISNULL(WITHDRAWAL_DATE)||TRIM(WITHDRAWAL_DATE)==""?NULL:CAST(WITHDRAWAL_DATE
AS DATETIME)
So I want this WITHDRAWAL_DATE input String datatype to be compared to an empty string--if it is empty I want it to become Null, otherwise to be cast as a date.
Thanks guys for your helps. I am so confused! WITHDRAWAL_DATE is a DATE data type input in the source XML file and now I have it as a STRING data type in my XSD file. Ultimately the problem is that some of the Withdrawal_Date fields in my XML source data are empty. So I want to insert Null values into my database for these records.
What data type do I need to specify in my XSD, XLST, and SQL output table? And it doesn't matter to me if I use Data Conversion task or Derived Column Xform but since I am new to these, could you send me expression syntax?
#BobS: when I ran with your updated solution, I received error:
The conversion of a datetime2 data type to a datetime data type resulted in an out-of-range value
So when I googled datetime2 datatype it looks like this is a new data type that supports larger time/date fields. So I modified my SQL table to use DATETIME2 instead for this field and modified your cast expression below to use DATETIME2 but then output of transform didn't change accordingly.
I also tried changing WITHDRAWAL_DATE to datetime for all files and then changing SQL Table to say NOT NULL for this field. But this also gave me errors.
I see a couple of issues that you should address with this Derived Column transformation.
First, you can't change the data type of a column, which is what your expression is trying to do. I'm not sure if you're actually trying to do this. But, if your output column is the same as the input column, then you will have to change it. To do this, in the Derived Column editor, the Derived Column Name should be a new column name and the Derived Column should be <add as new column>
The expression needs two changes. To assign a NULL value you use a null function with syntax NULL(data-type). And, the CAST function syntax is (DT_datatype)columnname. So, here's how your expression should look
ISNULL(WITHDRAWAL_DATE) || TRIM(WITHDRAWAL_DATE) == "" ? NULL(DT_DBTIMESTAMP) : (DT_DBTIMESTAMP)WITHDRAWAL_DATE
UPDATE: You should be able to use the expression above; but, I did change it to reference the DT_DBTIMESTAMP data type. The SSIS DT_DBTIMESTAMP data type matches the DATETIME SQL Server data type.
To learn what data type you should use for the source component, you can right-click on the source component and select Show Advanced Editor... Select the inputs and outputs tab. Navigate the tree view to find your column and view the associated data type. The Advanced Editor is available most (maybe all) data flow components.
UPDATE 2: IF your output data type for the Derived Column component is DT_DBTIMESTAMP2 instead of DT_DBTIMESTAMP, make sure you change both DT_DBTIMESTAMP references in your expresson. Before closing the Derived Column component, look at the Data Type column for your expression. You can't change it, but it will show the data type that the expression output will be. If it's not what you want, then there's still a problem with your expression.
For, the source files, you can't change the data type of the external columns. At least, I haven't been able to do it. In SSIS, you have to work with what is interpreted by the Source component. If you can alter the files, to change data type, then great. Then, use the Derived Column component to convert what is giving to what you need.
What errors are you getting? I suspect one of them is a cast error.