Conversion Failed Due To Data Overflow (Numeric) - sql

I am trying to move data from a .dbf file to a table in SQL Server 2008 and am getting the following error on multiple numeric columns:
OLE DB provider "MSDASQL" for linked server "(null)" returned message "Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done.".
Msg 7341, Level 16, State 2, Line 1
Cannot get the current row value of column "[MSDASQL].apryr" from OLE DB provider "MSDASQL" for linked server "(null)". Conversion failed because the data value overflowed the data type used by the provider.
It only happens on numeric columns and not on every numeric column. Character data is fine and there is no date/time data that could give any issues.
Here is a sample of the code I'm using:
insert into [table] select * from OPENROWSET('MSDASQL',
'DRIVER=Microsoft Visual FoxPro Driver;
SourceDB=[filepath];
SourceType=DBF',
'select *
from [file].dbf)
Since the data in the dbf file is customer data, I've been told I can't manually fix the garbage data in the file (assuming there is any) and everything has to be done through the SQL code. I have searched around the internet and haven't really found a solution to this problem. I'd appreciate any help.
Thank you.


Without knowing more specifics, the situation sounds simple enough: There is data in the dbf file that does not match the data(s) type in your SQL Server table. If that is the case, then you have two options:
Change your SQL Server table to accommodate the data in your dbf file.
Do not import data from the dbf file that is causing the issue.
In option #1, you could modify restrictive numeric or date-type fields to varchar or nvarchar fields. Then, you would want to modify any programs that might be assuming certain data types in the dbf file to accommodate varchar or nvarchar data. For instance you could use some kind of try-catch language that tests the conversion of data before letting a program have access to it.
If you decide to go with option #2, you can change your select query to be filter out data that does not meet the field requirements of your SQL Server table(s).
Good luck!


Check your field types on SQL Server table. Maybe some of them unable to take your DBF's BCD value.

Related

Issues with special characters from IBM DB2 to SQL in SSIS

I am trying to pull data from IBM db2 into a SQL table via SSIS.
However, the table (source) contains a special character in the column, like below;
"?Ǫ. note pads wearing thin"
As a result, my package fails with the following error;
[OLE DB Source [33]] Error: There was an error with OLE DB
Source.Outputs[OLE DB Source Output].Columns[VTTEXT] on OLE DB
Source.Outputs[OLE DB Source Output]. The column status returned was:
"Text was truncated or one or more characters had no match in the
target code page.".
The data is being loaded into a VARCHAR column within SQL, but the error seems specific to my data source.
How can I get around it?

When an nvarchar column can store any Unicode data the varchar column is restricted to an 8-bit codepage. Codepage incompatabilities are frequent issues, so Unicode is the cure for codepage problems.
ALTER TABLE yourtable
MODIFY thecolumn NVARCHAR(255);

Error when trying to load data from an OData Source Editor to a SQL db table

Several Errors of the same type when trying to load a SQL db table from an OData SharePoint connection.
[GFXBankAccountProcessing - DB GFX Account List [2]] Error: An error occurred while setting up a binding for the "BLCompanyID" column. The binding status was "DT_NTEXT". The data flow column type is "DBBINDSTATUS_UNSUPPORTEDCONVERSION". The conversion from the OLE DB type of "DBTYPE_IUNKNOWN" to the destination column type of "DBTYPE_WVARCHAR" might not be supported by this provider.
It is expected to load the table and proceed to the next function within the process. I believe it has something to do with the data conversion but I am not sure what to convert the data to. I have looked to try to compare datatypes that the table in the DB is requiring. It is a NVARCHAR but I am not sure why it would fail. BLCompanyID is only one column, some of the other columsn are succedding while that one and a few others are failing.

The problem is that the data type you are trying to store into the destination table is not the correct format. For instance, if the destination table requires an nvarchar(255) and you're trying to insert a DT_NTEXT, it will fail. You will need to convert the column to DT_WSTR with a length of 255.
Here's a quick reference that I have bookmarked to help me:
http://wiki.melissadata.com/index.php?title=FAQ%3ASSIS%3AData_Type_Conversions

Getting DataType error in SSIS - Package is failing

I have two packages that I built in SSIS. This package goes out to a file path, pulls in two excel files, and loads them to two tables in SQL Server db. One of the packages runs fine and loads its respective table. However, the other excel/csv file contains two date columns and the package is failing because the SQL Server table that it needs to load to has two DateTime columns. The error I am getting is:
-There was an error with OLE DB Destination.Inputs[OLE DB Destination Input].Columns[CustDate] on OLE DB Destination.Inputs[OLE DB Destination Input].
I have tried to cast the datetime column to Date in the SQL table, and it still doesn't work and throws the same error. I have added a data conversion task in SSIS to try and convert the two date fields to multiple datatypes, and I still get an error. Can someone please tell me what datatype I should convert these two fields in the SSIS package where it would work? The first package runs fine because it doesn't contain any date fields in the excel file or the table it loads to.
So far, I have tried to convert these two date fields to: string[DT_STR], Unicode string[DT_WSTR], date[DT_DATE] and none of them work. Can anyone offer up any suggestions? Thank you!
So far, I have tried to convert these two date fields to: string[DT_STR], Unicode string[DT_WSTR], date[DT_DATE] and none of them work. Can anyone offer up any suggestions? Thank you!

Made the following 2 changes. It will give you answer.
1) Select your excel file connection and add IMEX=1 to the end of the connection under properties. So your connection string will now look like
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\SSIS\Test.xls;Extended Properties="EXCEL 8.0;HDR=YES;IMEX=1";
2) Use the following expression in the derived column transformation
ISNULL([CustDate]) ? NULL(DT_DATE) : (LEN(TRIM([CustDate])) == 0 ? NULL(DT_DATE) : (DT_DATE)((DT_DBDATE)TRIM([CustDate])))

Can't convert String to Numeric/Decimal in SSIS

I have five or six OLE DB Sources with a String[DT_STR], with a length of 500 and 1252 (Latin) as Code Page.
The format of the column is like 0,08 or 0,10 etc etc. As you can see, it is separated with a comma.
All of them are equal except one of them. In this one source, I have a POINT as separation. On this it is working when I set the Data Type in the advanced editor of the OLE DB Source. On another (with comma separated) it is also working, if I set the Data Type in the advanced editor of the OLE DB Source. BUT the weird thing is, that it isn't working with the other sources although they are the same (sperated with comma).
I tested Numeric(18,2) and decimal(2).
Another try to solve the problem with the conversion task and/or the derived column task, failed.
I'm using SQL Server 2008 R2
Slowly, I think SSIS is fooling me :)
Has anyone an idea?
/// EDIT
Here a two screens:
Is working:
click
Isn't working:
click

I would not set the Data Type in the Advanced Editor of the OLE DB Source. I would convert the data in the SQL Code of the OLE DB Source, or in a Script Transformation e.g. using Decimal.TryParse , which would update a new column.
SSIS is unbeleivably fussy over datatypes and trying to mess with its internals is not productive.

Check that there are any spaces in between the commas, so that the SSIS is throwing an error trying to convert the blank space to a number. A blank space does not equal nothing in between spaces.
Redirect error rows and output the data to a file. Then you can examine the data that is being rejected by the SSIS and determine why it's causing error.
Reason for the error
1) Null’s are not properly handled either in the destination database or during SSIS package creation. It is quite possible that the source contains a null database but the destination is not accepting the NULL data leading to build generate above error.
2) Data types between source and destination does not match. For example, source column has varchar data and destination column have an int data type. This can easily generate above error. There are certain kind of datatypes which will automatically convert to another data type and will not generate the error but there are for incompatible datatypes which will generate The value could not be converted because of a potential loss of data. error.

The Issue arises when there is unhandled space or null. I have worked around using the Conditional (Ternary) Operator which checks the length:
LEN(TRIM([Column Name])) >= 1 ? (DT_NUMERIC,38,8)[Column Name] : 0

Import Package Error - Cannot Convert between Unicode and Non Unicode String Data Type

I have made a dtsx package on my computer using SQL Server 2008. It imports data from a semicolon delimited csv file into a table where all of the field types are NVARCHAR MAX.
It works on my computer, but it needs to run on the clients server. Whenever they create the same package with the same csv file and destination table, they receive the error above.
We have gone through the creation of the package step by step, and everything seems OK. The mappings are all correct, but when they run the package in the last step, they receive this error. They are using SQL Server 2005.
Can anyone advise where to begin looking for this problem?

The problem of converting from any non-unicode source to a unicode SQL Server table can be solved by:
add a Data Conversion transformation step to your Data Flow
open the Data Conversion and select Unicode for each data type that applies
take note of the Output Alias of each applicable column (they are named Copy Of [original column name] by default)
now, in the Destination step, click on Mappings
change all of your input mappings to come from the aliased columns in the previous step (this is the step that is easily overlooked and will leave you wondering why you are still getting the same errors)

At some point, you're trying to convert an nvarchar column to a varchar column (or vice-versa).
Moreover, why is everything (supposedly) nvarchar(max)? That's a code smell if I ever saw one. Are you aware of how SQL Server stores those columns? They use pointers to where the column is stored from the actual rows, since they don't fit within the 8k pages.

Non-Unicode string data types:
Use STR for text file and VARCHAR for SQL Server columns.
Unicode string data types:
Use W_STR for text file and NVARCHAR for SQL Server columns.
The problem is that your data types do not match, so there could be a loss of data during the conversion.

Two solutions:
1- if the type of the target column is [nvarchar] it should be change to [varchar]
2- Add a "Derived Column" component to the SSIS package and add a new column with the following expression:
(DT_WSTR, «length») [ColumnName]
Length is the length of the column in the target table and ColumnName is the name of the column in the target table.
finally at the mapping part you should use this new added column instead of the original column.

Not sure if this is a best practice with SSIS but sometimes I find their tools are a bit clunky when you want to do this type of activity.
Instead of using their components you can convert the data within your query
Instead of doing
SELECT myField = myNvarchar20Field
FROM myTable
You could do
SELECT myField = CONVERT(VARCHAR(20),myNvarchar20Field)
FROM myTable

This a solution that uses the IDE to fix:
Add a Data Conversion item to your dataflow as shown below;
Double click on the Data Conversion item, and set it as shown:
Now double click on the DB Destination item, Click on Mapping, and ensure that your input Column is actually the same as coming from the Copy of [your column name], which is in fact the Data Conversion output NOT the DB Source Output (be careful here). Here is a screenshot:
And thats it .. save and run ..

Mike, I had the same problem with SSIS in SQL Server 2005...
Apparently, the DataFlowDestination object will always attempt to validate the data coming in,
into Unicode. Go to that object, Advanced Editor, Component Properties pane, change the "ValidateExternalMetaData" property to False. Now, go to the Input and Output Properties pane, Destination Input, External Columns - set each column Data type and Length to match the database table it's going to. Now, when you close that editor, those column changes will be saved and not validated over, and it will work.

Follow the below steps to avoid (cannot convert between unicode and non-unicode string data types) this error
i) Add the Data conversion Transformation tool to your DataFlow.
ii) To open the DataFlow Conversion and select [string DT_STR] datatype.
iii) Then go to Destination flow, select Mapping.
iv) change your i/p name to copy of the name.

Get to the registry to configuration of the client and change the LANG.
For Oracle, go to HLM\SOFTWARE\ORACLE\KEY_ORACLIENT...HOME\NLS_LANG and change to appropriate language.

The dts data Conversion task is time taking if there are 50 plus columns!Found a fix for this at the below link
http://rdc.codeplex.com/releases/view/48420
However, it does not seem to work for versions above 2008. So this is how i had to work around the problem
*Open the .DTSX file on Notepad++. Choose language as XML
*Goto the <DTS:FlatFileColumns> tag. Select all items within this tag
*Find the string **DTS:DataType="129"** replace with **DTS:DataType="130"**
*Save the .DTSX file.
*Open the project again on Visual Studio BIDS
*Double Click on the Source Task . You would get the message
the metadata of the following output columns does not match the metadata of the external columns with which the output columns are associated:
...
Do you want to replace the metadata of the output columns with the metadata of the external columns?
*Now Click Yes. We are done !

Resolved - to the original ask:
I've seen this before. Easiest way to fix (don't need all those data conversion steps as ALL of the meta data is available from the source connection):
Delete the OLE DB Source & OLE DB Destinations
Make sure Delayed Validation is FALSE (you can set it to True later)
Recreate the OLE DB Source with your query, etc.
Verify in the Advanced Editor that all of the output data column types are correct
Recreate your OLE DB Destination, map, create new table (or remap to existing) and you'll see that SSIS got all the data types correct (same as source).
So much easier that the stuff above.

Not sure if this is still a problem but I found this simple solution:
Right-Click Ole DB Source
Select 'Edit'
Select Input and Output Properties Tab
Under "Inputs and Outputs", Expand "Ole DB Source Output" External Columns and Output Columns
In Output columns, select offending field, on the right-hand panel ensure Data Type Property matches that of the field in External Columns properties
Hope this was clear and easy to follow

Sometime we get this error when we select static character as a field in source query/view/procedure and the destination field data type in Unicode.
Below is the issue i faced:
I used the script below at source
and got the error message Column "CATEGORY" cannot convert between Unicode and non-Unicode string data types. as below:
error message
Resolution:
I tried multiple options but none worked for me. Then I prefixed the static value with N to make in Unicode as below:
SELECT N'STUDENT DETAIL' CATEGORY, NAME, DATEOFBIRTH FROM STUDENTS
UNION
SELECT N'FACULTY DETAIL' CATEGORY, NAME, DATEOFBIRTH FROM FACULTY

If anyone is still experiencing this issue, I found that it related to a difference in Oracle Client versions.
I have posted my full experience and solution here: https://stackoverflow.com/a/43806765/923177

1.add a Data Conversion tool from toolbox
2.Open it,It shows all coloumns from excel ,convert it to desire output. take note of the Output Alias of
each applicable column (they are named Copy Of [original column name] by default)
3.now, in the Destination step, click on Mappings

I changed ValidateExternalMetadata=False for each transformation task. It worked for me.