SQL Bulk Upload/Copy with Format File Error - sql

Question

So, I have a table with the following structure;
CREATE TABLE [dbo].[LoadAgencyInfo](
       [Agency] [varchar](4) NULL,
       [Fund] [varchar](1) NULL,
       [Account] [varchar](14) NULL,
       [LName] [varchar](20) NULL,
       [FName] [varchar](9) NULL,
       [SSN] [varchar](9) NULL,
       [Prior] [varchar](3) NULL,
       [BalDue] [varchar](8) NULL,
       [DRSBal] [varchar](8) NULL,
       [Fill1] [varchar](16) NULL,
       [FileDate] [varchar](6) NULL,
       [Fill2] [varchar](3) NULL,
       [P1] [varchar](2) NULL,
       [P2] [varchar](2) NULL
) ON [PRIMARY]
GO
I then have a flat file that I intend to bulk upload to the database. The file format is as follows and is just a normal text file, see below; there are several different files but they all follow this format.
Sample Data
2000S10000252      VYUSBN              RUDXOD   0099882260000006824100000000                181802      
2300S20000350      TEVXXXX             TGXC     0025045640000000652800000000                052153      
 
The file is supposed to follow the format below based on character length. Each cell should fill with the correct information based on where the characters are in the file/record. The records with a width of zero fill should fill with zeros. The records with skip this field should skip over the particular field and not put any data into the cell.
File Break Down
Field Name
Character Start
Character End
Width
Agency
1
4
4
Fund
5
5
1
Account
6
19
14
LName
20
39
20
FName
40
48
9
SSN
49
57
9
Prior
58
60
3 (Zero fill, doesn’t come from text file default to 000)
BalDue
61
68
8
DrsBal
69
76
8 (Zero fill, doesn’t come from text file default to 00000000)
Fill1
77
92
16 (Skip this field)
FileDate
93
98
6
Fill2
99
105
7 (skip this field)
 
The flat file is supposed to follow a strict guideline in terms of character length, for example the first four characters belong to the agency cell, and the next character belongs to the fund cell, etc. Notice there aren’t any delimited characters so I am using a SQL format file to help process the records and input them into the database. However no matter what I do the appropriate cells are either not filled in or partially filled in and when I attempt to skip over fields the fields I am skipping get populated. I have tried changing the datatype to nvarchar but still doesn’t work. Sometimes the data gets garbbled and doesn’t translate properly.
See the format file below;
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
       <FIELD ID="1" xsi:type="CharFixed" MAX_LENGTH="4" />
       <FIELD ID="2" xsi:type="CharFixed" MAX_LENGTH="1"/>
       <FIELD ID="3" xsi:type="CharFixed" MAX_LENGTH="14"/>
       <FIELD ID="4" xsi:type="CharFixed" MAX_LENGTH="20"/>
       <FIELD ID="5" xsi:type="CharFixed" MAX_LENGTH="9"/>
       <FIELD ID="6" xsi:type="CharFixed" MAX_LENGTH="9"/>
       <FIELD ID="7" xsi:type="CharFixed" MAX_LENGTH="3"/>
       <FIELD ID="8" xsi:type="CharFixed" MAX_LENGTH="8"/>
       <FIELD ID="9" xsi:type="CharFixed" MAX_LENGTH="8"/>
</RECORD>
<ROW>
       <COLUMN SOURCE="1" NAME="Agency" xsi:type="SQLINT" LENGTH="4"/>
       <COLUMN SOURCE="2" NAME="Fund" xsi:type="SQLNVARCHAR" LENGTH="1"/>
       <COLUMN SOURCE="3" NAME="Account" xsi:type="SQLNVARCHAR" LENGTH="14"/>
       <COLUMN SOURCE="4" NAME="LName" xsi:type="SQLNVARCHAR" LENGTH="20"/>
       <COLUMN SOURCE="5" NAME="FName" xsi:type="SQLNVARCHAR" LENGTH="9"/>
       <COLUMN SOURCE="6" NAME="SSN" xsi:type="SQLINT" LENGTH="9"/>
       <COLUMN SOURCE="7" NAME="Prior" xsi:type="SQLINT" LENGTH="3"/>
       <COLUMN SOURCE="8" NAME="BalDue" xsi:type="SQLINT" LENGTH="8"/>
       <COLUMN SOURCE="9" NAME="DRSBal" xsi:type="SQLINT" LENGTH="8"/>
</ROW>
</BCPFORMAT>
 
Any help as to why this isn’t working would be very beneficial

Answer 1

So, I’ve made some adjustments to the SQL format xml file and to the SQL command itself as you can see below. The issue now is that all of the appropriate fields are populated when I run the command, but the last cell in the first row gets all the subsequent row’s data as there are over 2000+ rows. What should happen is after the sixth character in the FileDate field the data should move to the next row instead of putting all of the subsequent rows in the FileDate field. I have tried updating the ROWTERMINATOR field with 0x0a, 'n, 'r, 'r'n, 'n'r but nothing seems to work. I am also puzzled as to why the command isn’t respecting the MAX_LENGTH attribute as I have it set to 6 and why none of the row terminators aren’t working? I’ve been at this for days and it is driving me wild.
The source file is plan text from a unix machine and only has lf as the row terminator. I cant post the image of the data because I dont have 10 rep points.
XML File
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharFixed" LENGTH="4" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="2" xsi:type="CharFixed" LENGTH="1" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharFixed" LENGTH="14" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharFixed" LENGTH="20" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharFixed" LENGTH="9" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="6" xsi:type="CharFixed" LENGTH="9" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="7" xsi:type="CharFixed" LENGTH="3" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="8" xsi:type="CharFixed" LENGTH="8" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="9" xsi:type="CharFixed" LENGTH="8" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="10" xsi:type="CharFixed" LENGTH="16" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="11" xsi:type="CharTerm" MAX_LENGTH='6' TERMINATOR="0x0a" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="Agency" xsi:type="SQLCHAR" />
<COLUMN SOURCE="2" NAME="Fund" xsi:type="SQLCHAR" />
<COLUMN SOURCE="3" NAME="Account" xsi:type="SQLCHAR" />
<COLUMN SOURCE="4" NAME="LName" xsi:type="SQLCHAR" />
<COLUMN SOURCE="5" NAME="FName" xsi:type="SQLCHAR" />
<COLUMN SOURCE="6" NAME="SSN" xsi:type="SQLCHAR" />
<COLUMN SOURCE="7" NAME="Prior" xsi:type="SQLCHAR" />
<COLUMN SOURCE="8" NAME="BalDue" xsi:type="SQLCHAR" />
<COLUMN SOURCE="9" NAME="DRSBal" xsi:type="SQLCHAR" />
<COLUMN SOURCE="10" NAME="Fill1" xsi:type="SQLCHAR" />
<COLUMN SOURCE="11" NAME="FileDate" xsi:type="SQLNCHAR" />
</ROW>
</BCPFORMAT>