Is there any way to capture the error messages that occur during a bulk insert?
If I specify an error file, I get 2 seperate files, one that contains the record that errored, and one that contains the row.
The messages that are displayed for errors contain more information:
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 6 (temp_batch_date).
Is it possible to write these messages into a temp table so I can handle them accordingly?
Check out the SqlBulkCopy class to perform the operation and you should have programmatic access to any errors (exceptions) that are generated. This should allow you to attempt recovery/logging.
If you don't want to use Visual Studio (why not? it's free), you could also use PowerShell.
Handling these errors outside of SQL Server really opens up what/how you can workaround hurdles with your source data.
Related
I am working on a data migration project. In current task, I have to select data from n number of tables from Oracle, join them and insert the data into a single SQL table. The number of rows are in millions.
Issue: There is data in Oracle which when we are trying to insert in SQL is giving exception. For example the datatype of the Oracle column is VARCHAR2 whereas in SQL it's int. The data is numbers. But there are few columns which have special characters like ','. This is one such example which will fail when we are trying to insert into SQL table. It's failing for many such columns.
I am using SSIS for this task. I am moving all the error id's of the rows into an error table which are throwing such error as mentioned in above example.
Question: I want the column name for which the insertion is failing for each row. Is there an option in SSIS? On error I want to store the id and the column name in an Error table.
Tried to search on internet, but didn't get anything. In SSIS, we do have option to configure the rows having Error. But didn't find that giving column name option to insert into a error table.
Edit: The data will come on daily basis i.e. the SSIS package will be executed daily.
The Error Output contains many columns providing information about it.
The list of columns includes the columns in the component input, the ErrorCode and ErrorColumn columns added by previous error outputs, and the ErrorCode and ErrorColumn columns added by this component.
If you are using OLEDB Destination, you cannot redirect the error rows while using Fast load option. And since you mentioned that
The number of rows are in millions.
Then it is not recommended to use the Row-by-Row insertion.
If there are few columns, i suggest adding a Data Conversion Transformation and use its Error output to get the error information.
References and helpful links
Configuring Error Output Columns
SSIS how to redirect the rows in OLEDB Destination when the fast load option is turned on and maximum insert commit size set to zero
Error Handling in Data
Error Handling With OLE DB Destinations
I have five or six OLE DB Sources with a String[DT_STR], with a length of 500 and 1252 (Latin) as Code Page.
The format of the column is like 0,08 or 0,10 etc etc. As you can see, it is separated with a comma.
All of them are equal except one of them. In this one source, I have a POINT as separation. On this it is working when I set the Data Type in the advanced editor of the OLE DB Source. On another (with comma separated) it is also working, if I set the Data Type in the advanced editor of the OLE DB Source. BUT the weird thing is, that it isn't working with the other sources although they are the same (sperated with comma).
I tested Numeric(18,2) and decimal(2).
Another try to solve the problem with the conversion task and/or the derived column task, failed.
I'm using SQL Server 2008 R2
Slowly, I think SSIS is fooling me :)
Has anyone an idea?
/// EDIT
Here a two screens:
Is working:
click
Isn't working:
click
I would not set the Data Type in the Advanced Editor of the OLE DB Source. I would convert the data in the SQL Code of the OLE DB Source, or in a Script Transformation e.g. using Decimal.TryParse , which would update a new column.
SSIS is unbeleivably fussy over datatypes and trying to mess with its internals is not productive.
Check that there are any spaces in between the commas, so that the SSIS is throwing an error trying to convert the blank space to a number. A blank space does not equal nothing in between spaces.
Redirect error rows and output the data to a file. Then you can examine the data that is being rejected by the SSIS and determine why it's causing error.
Reason for the error
1) Null’s are not properly handled either in the destination database or during SSIS package creation. It is quite possible that the source contains a null database but the destination is not accepting the NULL data leading to build generate above error.
2) Data types between source and destination does not match. For example, source column has varchar data and destination column have an int data type. This can easily generate above error. There are certain kind of datatypes which will automatically convert to another data type and will not generate the error but there are for incompatible datatypes which will generate The value could not be converted because of a potential loss of data. error.
The Issue arises when there is unhandled space or null. I have worked around using the Conditional (Ternary) Operator which checks the length:
LEN(TRIM([Column Name])) >= 1 ? (DT_NUMERIC,38,8)[Column Name] : 0
I have a SQL Server 2008 database linked to an Access 2010 front end.
Some of the columns are varchar of varying lengths, from 50 to MAX. There is no default value and they can be null.
Recently, any of these columns longer than 255 started causing the write conflict error:
This record has been changed by another user since you started editing it. If you save the record, you will...
I can update other data, including varchar data, in the same row, with no problems and no errors.
This error occurs when I attempt to update the column directly from my table. It also occurs if I update the table programmatically via VBA or through a form.
It looks like this is because the column is NULL in SQL Server, but I've never seen this before with varchar columns. Is this a "feature" or a thing you just have to work around? It just started happening regardless, which is also confusing.
The leading cause of updatability problems in Office Access–linked tables is that Office Access is unable to verify whether data on the server matches with what was last retrieved by the dynaset being updated. If Office Access cannot perform this verification, it assumes that the server row has been modified or deleted by another user and it aborts the update.
There are several types of data that Office Access is unable to check reliably for matching values. These include large object types, such as text, ntext, image, and the varchar(max)(which is in your case), nvarchar(max), and varbinary(max) types.
Coming to your situation,i believe access forms may be bound to the data & produce a lot of locks on records and tables of SQL Server, those locks can prevent tables to be updated.So try to rectify Access forms which are bound to data and that are showing those updateability errors and make them work with unbound data, which allows you to read and update that unbound data to users without creating locks on the database server.
While refreshing Webi report I am getting an error:
A database error occured. The database error text is: (CS) "Unexpected behavior" . (WIS 10901)
All the objects are parsing in the universe and Server is also responding. What can be the possible reason?
We are also able to run query in the database using database client tool.
If the error message appears after the a long time it might just be a timeout issue.
Else, you could try to import a version of the report that works in CMS to your local drive, rename it and run again.
It can be caused by some special character in the data combined with the fact that the server language settings do not foresee such character and therefore Business Objects cannot parse it for presentation.
If that is the case you might need to configure an environment variable of the server (like NLS_LANG) setting it to a value such that those special characters in your data can be handled by Business Objects.
In my situation, the error appera when some objet from the data base has changed or does not exists anymore. So we need to delete this object in the Universe or be sure that the field exists in the data base with the same name and type.
I had same problem with my reports. After couple hour of "investigation", I found.
I create Object in my universe, and set inappropriate type of object data Number, when value in database have type Character.
It throw me oracle Error (ORA-01722), and Bussiness Object error (WIS 10901), though SQL copied from report creator interface, executed directly on database return proper data.
I have a SQL Server 2005 SP2 database which has a table with a poc_resp_city attribute which is nvarchar(35).
It was changed to nvarchar(80) 2 months ago without aligning the very same attribute in the data warehouse. (which still has nvarchar(35) )
The SSIS data loading package (after two months of proper working) now gives back package failure every time I run it with the following error:
There was an error with output column "poc_resp_city" (2250) on output
"OLE DB Source Output" (11). The column status returned was: "Text was
truncated or one or more characters had no match in the target code
page.". There was an error with output column "poc_resp_city"
(2250) on output "OLE DB Source Output" (11). The column status
returned was: "Text was truncated or one or more characters had no
match in the target code page.".
SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on
component "Source Table" (1) returned error code 0xC020902A. The
component returned a failure code when the pipeline engine called
PrimeOutput(). The meaning of the failure code is defined by the
component, but the error is fatal and the pipeline stopped executing.
There may be error messages posted before this with more information
about the failure.
Neither the package nor the databases were modified regarding this issue. I know that I could ignore this error or I could make the arrangements to make sure it's working but I want to provide a proper and acceptable answer why this error appears 2 months after the modification? Because maybe I miss an important step in this situation.
Important note: I don't have even a single record which has more than 35 characters so truncation never occurs. (this warning belongs to some kind of an SSIS validation step)
Now I think that maybe after a period of time, SSIS package recompiles itself and now it sees this misalignment in its metadata (35 =/= 80) and because TruncationRowDisposition attribute is set to RD_FailComponent, it fails the component.
And I would exclude the code page option because every database column is nvarchar, not varchar, so this shouldn't be the case.
Thanks!
You need to refresh size of column:
With right button push on OLE DB Source -> Show Advanced Editor
Choose Input and Output Properties tab -> Ole DB Source Output -> Output Columns
In the right panel Length row insert your new size.
Push OK
Or you can copy your query from OLE DB Source, delete OLE DB Source, insert new OLE DB Source and paste query. This gonna automaticily refresh your columns.
Just remember what there are probably more element in Dataflow where you need to edit length of your column, like Data Converion...