I am developing an SSIS 2008 package and I am trying to create a Derived Column transformation. But when I go to the Expression editor and try this expression it gives me alot of errors. I have tried various differentiations of this but all have resulted in errors. I need one of you SQL experts to point out a better expression!
ISNULL(WITHDRAWAL_DATE)||TRIM(WITHDRAWAL_DATE)==""?NULL:CAST(WITHDRAWAL_DATE
AS DATETIME)
So I want this WITHDRAWAL_DATE input String datatype to be compared to an empty string--if it is empty I want it to become Null, otherwise to be cast as a date.
Thanks guys for your helps. I am so confused! WITHDRAWAL_DATE is a DATE data type input in the source XML file and now I have it as a STRING data type in my XSD file. Ultimately the problem is that some of the Withdrawal_Date fields in my XML source data are empty. So I want to insert Null values into my database for these records.
What data type do I need to specify in my XSD, XLST, and SQL output table? And it doesn't matter to me if I use Data Conversion task or Derived Column Xform but since I am new to these, could you send me expression syntax?
#BobS: when I ran with your updated solution, I received error:
The conversion of a datetime2 data type to a datetime data type resulted in an out-of-range value
So when I googled datetime2 datatype it looks like this is a new data type that supports larger time/date fields. So I modified my SQL table to use DATETIME2 instead for this field and modified your cast expression below to use DATETIME2 but then output of transform didn't change accordingly.
I also tried changing WITHDRAWAL_DATE to datetime for all files and then changing SQL Table to say NOT NULL for this field. But this also gave me errors.
I see a couple of issues that you should address with this Derived Column transformation.
First, you can't change the data type of a column, which is what your expression is trying to do. I'm not sure if you're actually trying to do this. But, if your output column is the same as the input column, then you will have to change it. To do this, in the Derived Column editor, the Derived Column Name should be a new column name and the Derived Column should be <add as new column>
The expression needs two changes. To assign a NULL value you use a null function with syntax NULL(data-type). And, the CAST function syntax is (DT_datatype)columnname. So, here's how your expression should look
ISNULL(WITHDRAWAL_DATE) || TRIM(WITHDRAWAL_DATE) == "" ? NULL(DT_DBTIMESTAMP) : (DT_DBTIMESTAMP)WITHDRAWAL_DATE
UPDATE: You should be able to use the expression above; but, I did change it to reference the DT_DBTIMESTAMP data type. The SSIS DT_DBTIMESTAMP data type matches the DATETIME SQL Server data type.
To learn what data type you should use for the source component, you can right-click on the source component and select Show Advanced Editor... Select the inputs and outputs tab. Navigate the tree view to find your column and view the associated data type. The Advanced Editor is available most (maybe all) data flow components.
UPDATE 2: IF your output data type for the Derived Column component is DT_DBTIMESTAMP2 instead of DT_DBTIMESTAMP, make sure you change both DT_DBTIMESTAMP references in your expresson. Before closing the Derived Column component, look at the Data Type column for your expression. You can't change it, but it will show the data type that the expression output will be. If it's not what you want, then there's still a problem with your expression.
For, the source files, you can't change the data type of the external columns. At least, I haven't been able to do it. In SSIS, you have to work with what is interpreted by the Source component. If you can alter the files, to change data type, then great. Then, use the Derived Column component to convert what is giving to what you need.
What errors are you getting? I suspect one of them is a cast error.
Related
I have some data which was dumped from a PostgreSQL database (allegedly, using pg_dump) which needs to get imported into SQL Server.
While the data types are ok, I am running into an issue where there seems to be a placeholder for a NULL. I see a backslash followed by an uppercase N in many fields. Below is a snippet of the data, as viewed from within Excel. Left column has a Boolean data type, and the right one has an integer as the data type
Some of these are supposed to be of the Boolean datatype, and having two characters in there is most certainly not going to fly.
Here's what I tried so far:
Import via dirty read - keeping whatever datatypes SSIS decided each field had; to no avail. There were error messages about truncation on all of the boolean fields.
Creating a table for the data based on the correct data types, though this was more fun... I needed to do the same as in the dirty read, as the source would otherwise not load properly. There was also a need to transform the data into the correct data type for insertion into the destination data source; yet, I am getting truncation issues, when it most certainly shouldn't be.
Here is a sample expression in my derived column transformation editor:
(DT_BOOL)REPLACE(observation,"\\N","")
The data type should be Boolean.
Any suggestion would be really helpful!
Thanks!
Since I was unable to circumvent the SSIS rules in order to get my data into my tables without an error, I took the quick-and-dirty approach.
The solution which worked for me was to have the source data read each column as if it were a string, and the destination table had all fields be of the datatype VARCHAR. This destination table will be used as a staging table, once in SS, I can manipulate as needed.
Thank you #cha for your input.
I have a table with the field "Ticket_Number" whose data type is int. I need in an outer join to connect this field to the "KeyData" field in another log file table which has the data type "varchar(50)" and sometimes has data with text in it but also has the ticket numbers whose records in this table I need to connect to. I've tried using the "convert" function on the field and even on a subview already pulling only the "numeric-like" data. However, even when I try to connect to a view that's already pulling only the "Numeric-like" data, I continue to get a conversion failed error. What should I do?
There may be character value in the VARCHAR field that you are using for join condition .
If you are using SQL server 2012 version, use TRYCONVERT instead of CONVERT function like below.
On Ticket_Number= TRYCONVERT (int,KeyData)
Or you can eliminate those non-numeric values also by adding ISNUMERIC () function in the where clause like below.
Where ISNUMERIC (KeyData)=1
I have a table with a column1 nvarchar(50) null. I want to insert this into a more 'tight' table with a nvarchar(30) not null. My idea was to insert a derived column task between source and destination task with this expression: Replace column1 = (DT_WSTR,30)Column1
I get the "truncation may occur error" and I am not allowed to insert the data into the new tighter table.
Also I am 100% sure that no values are over 30 characters in the column. Moreover I do not have the possibility to change the column data type in the source.
What is the best way to create the ETL process?
JotaBe recommended using a data conversion transformation. Yes, that is another way to achieve the same thing, but it will also error out if truncation occurs. Your way should work (I tried it), provided the input data really is less than 30 characters.
You could modify your derived column expression to
(DT_WSTR,30)Substring([Column1], 1, 30)
Consider changing the truncation error disposition of the Derived Column component within your Data Flow. By default, a truncation will cause the Derived Column component to fail. You can configure the component to ignore or redirect rows which are causing a truncation error.
To do this, open the Derived Column Transformation editor and click the 'Configure Error Output...' button in the bottom-left of the dialog. From here, change the setting in the 'Truncation' column for any applicable columns as required.
Be aware that any data which is truncated for columns ignoring failure will not be reported by SSIS during execution. It sounds like you've already done this, but it's important to be sure you've analysed your data as it currently stands and taken into consideration any possible future changes to the nature of the data before disabling truncation reporting.
To do so you must use a Data Conversion Transformation, which allows to change the data type from the original nvarchar(50) to the desired nvarchar(30).
You'll get a new column with the required data type.
Of course, you can decide what to do in case of error: truncation, by configuring this component.
UPDATE
As there are people who have downvoted this answer, let's add 3 more comments:
this solution is checked and works. Create a table with a nvarchar(50) column, a new table with a nvarchar(30) column, add a data flow that uses a data conversion transform and it works witout a glitch. Please, chek it, I guarantee. Besides, as the OP states "Also I am 100% sure that no values are over 30 characters in the column" in his case there will be no truncation problems. However, I recommend treating the possible errors, just in case they happen.
from MSDN: "a package can perform the following types of data conversions: ... Set the column length of string data"
from MSDN: "If the length of an output column of string data is shorter than the length of its corresponding input column, the output data is truncated."
Heyo,
I have a problem with a conversion from a Varchar column to Money.
As far as I know, the money data type want a "." as delimiter for cents.
My strings are like this: 796.01. So actually it should work. But the result is - with this example - 79601,00.
I also tried the replace function, ( "," as search_expression and "." as replace_expression) but wihtout success.
I'm using the Derivec Column Task in SSIS with a SQL 2008 R2 Server.
Here is a screen:
Klick Image
Can you try like the given below..[amount] is the input column
(DT_I4)REPLACE(amount,"'","")
It is working for me.
1.Source column is of datatype 'string
After data conversion
If you go to Adavanced Editor for OLE DB source and under Input Output Properties, expand Output Columns, choose required column(for which you want to convert it into currency) and set it's Data Type property as currency [DT_CY] from drop down list. In this way OLEDB source itself will take care of conversion for output column.
There are few indications why it was not working in your case -
Data Conversion - If you are converting data to a date or a datetime data type, the date in the output column is in the ISO format, although the locale preference may specify a different format.
Data Conversion - When copying between columns with a string data type, the two columns must use the same code page.
Derived Column - If an expression references an input column that is overwritten by the Derived Column transformation, the expression uses the original value of the column, not the derived value.
Let's say I have a column of varchar(40) with data already and i change the datatype of that column to integer. does the data change at all in the columns (ie, does the data 'corrupt') or does it not matter and a table of (1,2,3) will still be (1,2,3) regardless of the datatype?
It will be the same, subject to
datatype conversion error (eg "foo" to int)
truncation (eg "foobar" in char(4))
If you attempt to change a column's datatype and the new type is incompatible with the old type, it will fail and nothing will change. You'll get an error like:
"Disallowed implicit conversion from data type <type> to data type <type2>".
Your first problem when you do something like this is that not all the data may meet the criteria to make the change. Those records need to be found and fixed before changing a data type.
The safest way to do something like this is to:
Make a backup
Find and fix any data that will not meet the criteria for the new datatype
Create an additional column in the correct datatype
Migrate the data
Drop the orginal column
Rename the new column to the old name