I created a ODBC connection through PowerQuery. The value column is causing the following error:
DataSource.Error: The ODBC driver returned a decimal value for column X, that we couldn't understand or is out of range.
It seems that the errors are caused mostly by values of around 0 with more than 6 decimal when the data is loaded directly to PowerQuery. If I load the connection directly to an excel table, there are no errors.
The SQL code I am using is simply selecting the desired columns with couple of conditions (SELECT, FROM, WHERE). I am not adept with SQL yet.
Is there a some kind of Format or Rounding I can apply to the selected value column directly in the query so that the data is transformed in a format that PowerQuery is able to read before being loaded to PowerQuery?
Or any other possible solutions to fix this error?
Any input is appreciated.
Thank you.
Related
I have two packages that I built in SSIS. This package goes out to a file path, pulls in two excel files, and loads them to two tables in SQL Server db. One of the packages runs fine and loads its respective table. However, the other excel/csv file contains two date columns and the package is failing because the SQL Server table that it needs to load to has two DateTime columns. The error I am getting is:
-There was an error with OLE DB Destination.Inputs[OLE DB Destination Input].Columns[CustDate] on OLE DB Destination.Inputs[OLE DB Destination Input].
I have tried to cast the datetime column to Date in the SQL table, and it still doesn't work and throws the same error. I have added a data conversion task in SSIS to try and convert the two date fields to multiple datatypes, and I still get an error. Can someone please tell me what datatype I should convert these two fields in the SSIS package where it would work? The first package runs fine because it doesn't contain any date fields in the excel file or the table it loads to.
So far, I have tried to convert these two date fields to: string[DT_STR], Unicode string[DT_WSTR], date[DT_DATE] and none of them work. Can anyone offer up any suggestions? Thank you!
So far, I have tried to convert these two date fields to: string[DT_STR], Unicode string[DT_WSTR], date[DT_DATE] and none of them work. Can anyone offer up any suggestions? Thank you!
Made the following 2 changes. It will give you answer.
1) Select your excel file connection and add IMEX=1 to the end of the connection under properties. So your connection string will now look like
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\SSIS\Test.xls;Extended Properties="EXCEL 8.0;HDR=YES;IMEX=1";
2) Use the following expression in the derived column transformation
ISNULL([CustDate]) ? NULL(DT_DATE) : (LEN(TRIM([CustDate])) == 0 ? NULL(DT_DATE) : (DT_DATE)((DT_DBDATE)TRIM([CustDate])))
I am working on a data migration project. In current task, I have to select data from n number of tables from Oracle, join them and insert the data into a single SQL table. The number of rows are in millions.
Issue: There is data in Oracle which when we are trying to insert in SQL is giving exception. For example the datatype of the Oracle column is VARCHAR2 whereas in SQL it's int. The data is numbers. But there are few columns which have special characters like ','. This is one such example which will fail when we are trying to insert into SQL table. It's failing for many such columns.
I am using SSIS for this task. I am moving all the error id's of the rows into an error table which are throwing such error as mentioned in above example.
Question: I want the column name for which the insertion is failing for each row. Is there an option in SSIS? On error I want to store the id and the column name in an Error table.
Tried to search on internet, but didn't get anything. In SSIS, we do have option to configure the rows having Error. But didn't find that giving column name option to insert into a error table.
Edit: The data will come on daily basis i.e. the SSIS package will be executed daily.
The Error Output contains many columns providing information about it.
The list of columns includes the columns in the component input, the ErrorCode and ErrorColumn columns added by previous error outputs, and the ErrorCode and ErrorColumn columns added by this component.
If you are using OLEDB Destination, you cannot redirect the error rows while using Fast load option. And since you mentioned that
The number of rows are in millions.
Then it is not recommended to use the Row-by-Row insertion.
If there are few columns, i suggest adding a Data Conversion Transformation and use its Error output to get the error information.
References and helpful links
Configuring Error Output Columns
SSIS how to redirect the rows in OLEDB Destination when the fast load option is turned on and maximum insert commit size set to zero
Error Handling in Data
Error Handling With OLE DB Destinations
I have five or six OLE DB Sources with a String[DT_STR], with a length of 500 and 1252 (Latin) as Code Page.
The format of the column is like 0,08 or 0,10 etc etc. As you can see, it is separated with a comma.
All of them are equal except one of them. In this one source, I have a POINT as separation. On this it is working when I set the Data Type in the advanced editor of the OLE DB Source. On another (with comma separated) it is also working, if I set the Data Type in the advanced editor of the OLE DB Source. BUT the weird thing is, that it isn't working with the other sources although they are the same (sperated with comma).
I tested Numeric(18,2) and decimal(2).
Another try to solve the problem with the conversion task and/or the derived column task, failed.
I'm using SQL Server 2008 R2
Slowly, I think SSIS is fooling me :)
Has anyone an idea?
/// EDIT
Here a two screens:
Is working:
click
Isn't working:
click
I would not set the Data Type in the Advanced Editor of the OLE DB Source. I would convert the data in the SQL Code of the OLE DB Source, or in a Script Transformation e.g. using Decimal.TryParse , which would update a new column.
SSIS is unbeleivably fussy over datatypes and trying to mess with its internals is not productive.
Check that there are any spaces in between the commas, so that the SSIS is throwing an error trying to convert the blank space to a number. A blank space does not equal nothing in between spaces.
Redirect error rows and output the data to a file. Then you can examine the data that is being rejected by the SSIS and determine why it's causing error.
Reason for the error
1) Null’s are not properly handled either in the destination database or during SSIS package creation. It is quite possible that the source contains a null database but the destination is not accepting the NULL data leading to build generate above error.
2) Data types between source and destination does not match. For example, source column has varchar data and destination column have an int data type. This can easily generate above error. There are certain kind of datatypes which will automatically convert to another data type and will not generate the error but there are for incompatible datatypes which will generate The value could not be converted because of a potential loss of data. error.
The Issue arises when there is unhandled space or null. I have worked around using the Conditional (Ternary) Operator which checks the length:
LEN(TRIM([Column Name])) >= 1 ? (DT_NUMERIC,38,8)[Column Name] : 0
We use DB2 as our main production database, but we use SQL Server for many other things i.e. to do integration between other customers and vendors via EDI etc.
I have a table in SQL with SO numbers and I try to make a lookup in DB2 to get all the invoices for the SO's in my table, so here's what I did.
Created a connection to the DB2 using the Microsoft® OLEDB Provider for DB2
Created a data fllow with a source using a SQL Server connection.
Added a Data Conversion Transformation trying to convert the INT so value to a decimal with a precision of 12, but I couldn't change a precision in a DT_DECIMAL, so the only datatype that I have the option to change the precision is DT_NUMERIC.
Added a lookup transformation to lookup the data withing DB2.
Now when i try to create the join between the source table and DB2 I get an error Cannot map the input column, 'so', to the lookup column, 'orno', because the data types do not match.
According to Microsoft this is not a bug and they suggest to use the DT_NUMERIC where you can change the precision.
If I try to convert the SO to a DT_DECIMAL without changing the precision I'd get the same error mentioned above.
Is there any way to work around the limitations from SSIS and change the precision in a DT_DECIMAL conversion so I could do the match?
Or any other suggestions?
The simple answer is to change the connection property in the DB2 connection to treat DECIMAL as NUMERIC.
See bellow
I have setup an SSIS package to take data from an excel file, and insert it into SQL.
The package takes the data from Excel, performs some Data Conversions, and then Inserts it into my SQL DB.
One of the columns in excel contains both text, and ints. Typically the values will be either: W2 or 1099.
In the data conversion data flow, this column is converted to: string[DT_STR], length: 50, Code Page: 1252 (ANSI - Latin I)
in SQL, the field it is going to is configured as a varchar(50).
For any row, where the column value in Excel is 1099, the data inserted into SQL, comes through as NULL.
I have no idea what could be causing this. I have similar conversions in other packages, that come through with no issues.
i have tried formatting the column in excel as text as well as re-creating the data conversion, with no luck.
Any suggestions of ways to get this to work properly appreciated. Please let me know if you need any additional information.
Is the data NULL in the Excel preview? You need to isolate whether this NULL is being provided by the Excel driver.
Anyway this sounds like the old IMEX=1 issue that has been around since forever.
http://microsoft-ssis.blogspot.com.au/2011/06/mixed-data-types-in-excel-column.html
Try adding IMEX=1 to your Excel connection string.