SSIS Package won't fail, even with bad input data - sql-server-2005

I have an SSIS package that won't error out even with bad input. In my case this package reads from a flat file and puts records into a SQL Server table. Pretty straightforward, nothing fancy going on here.
The flat file is defined as being ragged right, 80 characters per row, maybe 10 columns in total. Problem: sometimes the flat file isn't padded out to 80 characters, so we get variable length rows, instead of spaces filling out the rest. We want to fail the package when that happens.
In my flat file source component I have the error output section set up so that if any column is truncated the component will fail. Yet for some reason all steps are green when I run the package from Visual Studio, even though no rows are imported. Here's the output I get:
SSIS package "SSIS Package 01.dtsx" starting.
Information: 0x4004300A at Import data to Table01, DTS.Pipeline: Validation phase is beginning.
Information: 0x4004300A at Import data to Table01, DTS.Pipeline: Validation phase is beginning.
Information: 0x40043006 at Import data to Table01, DTS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Import data to Table01, DTS.Pipeline: Pre-Execute phase is beginning.
Information: 0x402090DC at Import data to Table01, Flat File Source [1]: The processing of file "C:\import_files\sampledata.dat" has started.
Information: 0x4004300C at Import data to Table01, DTS.Pipeline: Execute phase is beginning.
Warning: 0x8020200F at Import data to Table01, Flat File Source [1]: There is a partial row at the end of the file.
Information: 0x402090DE at Import data to Table01, Flat File Source [1]: The total number of data rows processed for file "C:\import_files\sampledata.dat" is 0.
Information: 0x402090DF at Import data to Table01, OLE DB Destination [5467]: The final commit for the data insertion has started.
Information: 0x402090E0 at Import data to Table01, OLE DB Destination [5467]: The final commit for the data insertion has ended.
Information: 0x40043008 at Import data to Table01, DTS.Pipeline: Post Execute phase is beginning.
Information: 0x402090DD at Import data to Table01, Flat File Source [1]: The processing of file "C:\import_files\sampledata.dat" has ended.
Information: 0x40043009 at Import data to Table01, DTS.Pipeline: Cleanup phase is beginning.
Information: 0x4004300B at Import data to Table01, DTS.Pipeline: "component "OLE DB Destination" (5467)" wrote 0 rows.
....so no records are imported, and there are warnings about partial rows, but the package completes successfully. When this package is fed a good input file, 80 characters per row, it imports as many rows as are in the file with no problem.
The strange thing is we have other SSIS packages that fail when variable-length rows are used as input. I've looked and compared those packages to this and for the life of me cannot see what they might be doing different.
If you have any ideas or leads I could follow I'd be grateful. Thanks!

Also another fix is:
Insert a SET NOCOUNT ON statement at the start of the SQL command
I found this out at:
http://louiebao.net/ssis-package-succeeded-without-errors-but-wrote-0-rows/

Ok, we've got this one figured out. What happened is the final column in each row expects a CR/LF ColumnDelimiter.
So when the first row is processed and comes up short, SSIS doesn't acknowledge the CR/LF simply because it comes too soon. It instead continues on and processes the second row as part of the first. In this case the second row is all that's left and we get the truncation message referred to above, because a CR/LF is never encountered at character 80.
In the case of the other SSIS package that does fail, a number of rows are close enough to the expected line width that when the package tries stuffing them into a single field a truncation error on that field is triggered.

Related

Import Flat File to MS SQL

I am trying to import a CSV into MSSQL 2008 by using the flat file import method but I am getting an Overflow error. Any ideas on how to go around it?
I used the tool before for files containing up to 10K-15K records but this file has 75K records in it....
These are the error messages
Messages
Error 0xc020209c: Data Flow Task 1: The column data for column "ClientBrandID" overflowed the disk I/O buffer.
(SQL Server Import and Export Wizard)
Error 0xc0202091: Data Flow Task 1: An error occurred while skipping data rows.
(SQL Server Import and Export Wizard)
Error 0xc0047038: Data Flow Task 1: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Source - Shows_csv" (1) returned error code 0xC0202091. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
(SQL Server Import and Export Wizard)
This could be a format problem of the csv file e.g. the delimiter. Check if the delimiters are consistent within the file.
It could also be a problem of blank lines. I had a similar problem a while ago. I've solved it by removing all blank lines in the csv file. Worth a try anyway.
You may have one or more bad data elements. Try loading a small subset of your data to determine if it's a small number of bad records or a large one. This will also tell you if your loading scheme is working and your datatypes match.
Sometimes you can quickly spot data issues if you open the csv file in excel.
Another possible reason for this error is that input file has wrong encoding. So, when you manually check data, it seems fine. For example, in my case correct files were in 8-bit ANSI, and wrong files in UTF-16 - you can tell the difference by looking at files size, wrong files were twice bigger than correct files.

BIDS Package Error's on Truncate while EXPORTING to flat file

I have a BIDS package. The final "Data Flow Task" exports a SQL table to Flat File. I receive a truncation error on this process. What would cause a truncation error while exporting to flat file? The error was occurring within the "OLE DB" element under the Data Flow tab for the "Data Flow Task".
I have set the column to ignore truncation errors and the export works fine.
I understand truncation errors. I understand why they would happen when you are importing data into a table. I do not understand why this would happen when outputting to a flat file.
This might be occurring for many reasons. Please make sure some of the steps listed below:
1) Check the source Data types that has to match with destination data type. If there are different it might through Truncation Error.
2) Check if there are blocks :- You can check this by creating Data viewer before the Destination and see the data come through.

SQL Server 2005 SSIS Buffer Overflow

I'm trying to import a UTF-8 encoded file (not sure if this is relevant) containing 10 columns - the delimiter is รพ. I've added a flat file source and set up all the columns as usual and set them to DT_WSTR.
However, it's failing to read in the file as I keep getting the error message below, [advertiser-id] buffer overflow. Advertiser ID is the first column in the file. I've checked the file and all the data is correct, no missing delimiters or columns anywhere. But strangely, it works if there's 1193 rows of data or less, regardless of which rows they are from the file.
Any ideas on why it's work for 1193 but no more? I've tried setting the max rows in the buffer to small numbers and big numbers but still getting the same message.
`SSIS package "pck_NetsightV3_Import_Doubleclick.dtsx" starting.
Information: 0x4004300A at Transfer Creative Match Table, DTS.Pipeline: Validation phase is beginning.
Information: 0x40043006 at Transfer Creative Match Table, DTS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Transfer Creative Match Table, DTS.Pipeline: Pre-Execute phase is beginning.
Information: 0x402090DC at Transfer Creative Match Table, Flat File Source [1]: The processing of file "\\man2\systems\Team\datafiles\netsight\doubleclick\latest\doubleclick_matchtables_creative.csv" has started.
Information: 0x4004300C at Transfer Creative Match Table, DTS.Pipeline: Execute phase is beginning.
Error: 0xC020209C at Transfer Creative Match Table, Flat File Source [1]: The column data for column "Advertiser-ID" overflowed the disk I/O buffer.
Error: 0xC0202091 at Transfer Creative Match Table, Flat File Source [1]: An error occurred while skipping data rows.
Error: 0xC0047038 at Transfer Creative Match Table: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Flat File Source" (1) returned error code 0xC0202091. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
Error: 0xC0047021 at Transfer Creative Match Table: SSIS Error Code DTS_E_THREADFAILED. Thread "SourceThread0" has exited with error code 0xC0047038. There may be error messages posted before this with more information on why the thread has exited.
Error: 0xC0047039 at Transfer Creative Match Table: SSIS Error Code DTS_E_THREADCANCELLED. Thread "WorkThread0" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown. There may be error messages posted before this with more information on why the thread was cancelled.
Error: 0xC0047021 at Transfer Creative Match Table: SSIS Error Code DTS_E_THREADFAILED. Thread "WorkThread0" has exited with error code 0xC0047039. There may be error messages posted before this with more information on why the thread has exited.
Information: 0x40043008 at Transfer Creative Match Table, DTS.Pipeline: Post Execute phase is beginning.
Information: 0x402090DD at Transfer Creative Match Table, Flat File Source [1]: The processing of file "\\man2\systems\Team\datafiles\netsight\doubleclick\latest\doubleclick_matchtables_creative.csv" has ended.
Information: 0x402090DF at Transfer Creative Match Table, OLE DB Destination [156]: The final commit for the data insertion has started.
Information: 0x402090E0 at Transfer Creative Match Table, OLE DB Destination [156]: The final commit for the data insertion has ended.
Information: 0x40043009 at Transfer Creative Match Table, DTS.Pipeline: Cleanup phase is beginning.
Information: 0x4004300B at Transfer Creative Match Table, DTS.Pipeline: "component "OLE DB Destination" (156)" wrote 0 rows.
Task failed: Transfer Creative Match Table
SSIS package "pck_NetsightV3_Import_Doubleclick.dtsx" finished: Success.
The program '[7916] pck_NetsightV3_Import_Doubleclick.dtsx: DTS' has exited with code 0 (0x0).`

Errors in SQL Server while importing CSV file despite varchar(MAX) being used for each column

I'm trying to insert a large CSV file (several gigs) into SQL Server, but once I go through the Import Wizard and finally try to import the file I get the following error report:
Executing (Error)
Messages
Error 0xc02020a1: Data Flow Task 1: Data conversion failed. The data
conversion for column ""Title"" returned status value 4 and status
text "Text was truncated or one or more characters had no match in the
target code page.".
(SQL Server Import and Export Wizard)
Error 0xc020902a: Data Flow Task 1: The "Source -
Train_csv.Outputs[Flat File Source Output].Columns["Title"]" failed
because truncation occurred, and the truncation row disposition on
"Source - Train_csv.Outputs[Flat File Source Output].Columns["Title"]"
specifies failure on truncation. A truncation error occurred on the
specified object of the specified component.
(SQL Server Import and Export Wizard)
Error 0xc0202092: Data Flow Task 1: An error occurred while processing
file "C:\Train.csv" on data row 2.
(SQL Server Import and Export Wizard)
Error 0xc0047038: Data Flow Task 1: SSIS Error Code
DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on Source - Train_csv
returned error code 0xC0202092. The component returned a failure code
when the pipeline engine called PrimeOutput(). The meaning of the
failure code is defined by the component, but the error is fatal and
the pipeline stopped executing. There may be error messages posted
before this with more information about the failure.
(SQL Server Import and Export Wizard)
I created the table to insert the file into first, and I set each column to hold varchar(MAX), so I don't understand how I can still have this truncation issue. What am I doing wrong?
In SQL Server Import and Export Wizard you can adjust the source data types in the Advanced tab (these become the data types of the output if creating a new table, but otherwise are just used for handling the source data).
The data types are annoyingly different than those in MS SQL, instead of VARCHAR(255) it's DT_STR and the output column width can be set to 255. For VARCHAR(MAX) it's DT_TEXT.
So, on the Data Source selection, in the Advanced tab, change the data type of any offending columns from DT_STR to DT_TEXT (You can select multiple columns and change them all at once).
This answer may not apply universally, but it fixed the occurrence of this error I was encountering when importing a small text file. The flat file provider was importing based on fixed 50-character text columns in the source, which was incorrect. No amount of remapping the destination columns affected the issue.
To solve the issue, in the "Choose a Data Source" for the flat-file provider, after selecting the file, a "Suggest Types.." button appears beneath the input column list. After hitting this button, even if no changes were made to the enusing dialog, the Flat File provider then re-queried the source .csv file and then correctly determined the lengths of the fields in the source file.
Once this was done, the import proceeded with no further issues.
I think its a bug, please apply the workaround and then try again: http://support.microsoft.com/kb/281517.
Also, go into Advanced tab, and confirm if Target columns length is Varchar(max).
The Advanced Editor did not resolve my issue, instead I was forced to edit dtsx-file through notepad (or your favorite text/xml editor) and manually replace values in attributes to
length="0" dataType="nText" (I'm using unicode)
Always make a backup of the dtsx-file before you edit in text/xml mode.
Running SQL Server 2008 R2
Goto Advanced tab----> data type of column---> Here change data type from DT_STR to DT_TEXT and column width 255. Now you can check it will work perfectly.
Issue:
The Jet OLE DB provider reads a registry key to determine how many rows are to be read to guess the type of the source column.
By default, the value for this key is 8. Hence, the provider scans the first 8 rows of the source data to determine the data types for the columns. If any field looks like text and the length of data is more than 255 characters, the column is typed as a memo field. So, if there is no data with a length greater than 255 characters in the first 8 rows of the source, Jet cannot accurately determine the nature of the data type.
As the first 8 row length of data in the exported sheet is less than 255 its considering the source length as VARCHAR(255) and unable to read data from the column having more length.
Fix:
The solution is just to sort the comment column in descending order.
In 2012 onwards we can update the values in Advance tab in the Import wizard.

Importing csv file to table

I'm trying to import a csv file to a table in sql server 2005 with the wizard.
but when i import the file is always give me this errors :
Executing (Error) Messages Error 0xc02020a1: Data Flow Task 1: Data
conversion failed. The data conversion for column "Column 15" returned
status value 4 and status text "Text was truncated or one or more
characters had no match in the target code page.". (SQL Server Import
and Export Wizard)
Error 0xc020902a: Data Flow Task 1: The "output column "Column 15"
(70)" failed because truncation occurred, and the truncation row
disposition on "output column "Column 15" (70)" specifies failure on
truncation. A truncation error occurred on the specified object of the
specified component. (SQL Server Import and Export Wizard)
Error 0xc0202092: Data Flow Task 1: An error occurred while processing
file "C:\PEP_ENTITIES_71.csv" on data row 1. (SQL Server Import and
Export Wizard)
Error 0xc0047038: Data Flow Task 1: SSIS Error Code
DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Source
- PEP_ENTITIES_71_csv" (1) returned error code 0xC0202092. The component returned a failure code when the pipeline engine called
PrimeOutput(). The meaning of the failure code is defined by the
component, but the error is fatal and the pipeline stopped executing.
There may be error messages posted before this with more information
about the failure. (SQL Server Import and Export Wizard)
What i'm doing wrong? This file was a export from a query, then i delete all records and try to import.
The flat file import task will default lengths to VARCHAR(50). You need to go to the Advanced property of the flat file connection page and change the length manually to match the destination table. There should also be an option to Suggest Types which may match the metadata that you're using, but it does a sampling of rows from the file so it may not be as accurate as just setting types manually.
It may be that you're trying to import data from the CSV which is too large for the field you're importing it in to. Perhaps you need to increase the size of your fields in your SQL table?
Have you tried to import a single, very small line of data from the CSV to see if that works? If it does, excessively large data somewhere in the rest of the sheet may be the problem.