Delimiters with bcp import - sql

Question

I am trying to import bulk data from text files. The statement that i use is below:
DECLARE #s as varchar(300)
SET #s='bcp [hc]..[QRYM_COMPANIES] in ' + char(34) + 'C:\SQL Files\health canada allfiles\comp.txt' + char(34) + ' -C ACP -c -t \", -r\n -e' + char(34) + 'C:\SQL Files\health canada allfiles\err\comp.err.txt' + char(34) + ' -b250 -m50 -SServer -Usa -P' + char(34) + 'mypassword' + char(34) + ' -h' + char(34) + 'TABLOCK' + char(34)
PRINT #s
EXEC master..xp_cmdshell #s
Now, normally I would have put -t, but my data contains commas in some of the data fields which results in the data being shifted. When I try to use -t \" I get the error "String data, right truncation", so that is why I am having to use the combination of -t ,\"
The problem here is that I get quotations in my transferred data. Below is a sample row of my data:
"20042","NSL95","4797","NU SKIN INTERNATIONAL, INC.","DIN OWNER","Y","N","N","N","","75 WEST CENTER STREET","PROVO","UTAH","UNITED STATES","84601",""
The table description is below. To avoid the "string truncation error" i have added a 1 to the left field size.
Can someone help me fix this problem?
Thanks
CREATE TABLE [health_canada].[dbo].[QRYM_COMPANIES] (
DRUG_CODE VARCHAR(18) NOT NULL, --NUMERIC(8) NOT NULL,
MFR_CODE VARCHAR(15),
COMPANY_CODE VARCHAR(16), --NUMERIC(6),
COMPANY_NAME VARCHAR(180),
COMPANY_TYPE VARCHAR(140),
ADDRESS_MAILING_FLAG VARCHAR(11),
ADDRESS_BILLING_FLAG VARCHAR(11),
ADDRESS_NOTIFICATION_FLAG VARCHAR(11),
ADDRESS_OTHER VARCHAR(11),
SUITE_NUMERIC VARCHAR(120),
STREET_NAME VARCHAR(180),
CITY_NAME VARCHAR(160),
PROVINCE VARCHAR(140),
COUNTRY VARCHAR(140),
POSTAL_CODE VARCHAR(120),
POST_OFFICE_BOX VARCHAR(115)
) ON [PRIMARY]
GO

Answer 1

Consider to use a format file with your bcp call. It is easy to configure, and you can fine tune it for your file. Just add -f and the path to use it: http://msdn.microsoft.com/en-us/library/ms162802.aspx
Steps
Add the format file to the disk (make sure that the SQL server can reach the path).
Update paths in query.
Run query.
I used this file:
This code imported your example data without quotations with the format file:
DECLARE #importTable VARCHAR(128) = 'hc.dbo.QRYM_COMPANIES';
DECLARE #importPath VARCHAR(8000) = '\\YOURPATH\comp.txt';
DECLARE #formatFilePath VARCHAR(8000) = '\\YOURPATH\formatFile.txt';
DECLARE #separator VARCHAR(10) = '';
DECLARE #export VARCHAR(8000) = 'bcp "' + #importTable + '" in "' + #importPath + '" -T -f "' + #formatFilePath + '" -C RAW -S ' + ##SERVERNAME;
PRINT(#export)
EXEC xp_cmdshell #export
Output:
SELECT * FROM hc.dbo.QRYM_COMPANIES:
Add this format file to your disk:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="0" xsi:type="CharTerm" TERMINATOR='"' MAX_LENGTH="1" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="18" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="15" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="16" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="180" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="140" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="6" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="11" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="7" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="11" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="8" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="11" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="9" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="11" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="10" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="120" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="11" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="180" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="12" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="160" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="13" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="140" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="14" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="140" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="15" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="120" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="16" xsi:type="CharTerm" TERMINATOR='"\r\n' MAX_LENGTH="115" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="DRUG_CODE" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="2" NAME="MFR_CODE" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="3" NAME="COMPANY_CODE" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="4" NAME="COMPANY_NAME" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="5" NAME="COMPANY_TYPE" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="6" NAME="ADDRESS_MAILING_FLAG" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="7" NAME="ADDRESS_BILLING_FLAG" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="8" NAME="ADDRESS_NOTIFICATION_FLAG" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="9" NAME="ADDRESS_OTHER" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="10" NAME="SUITE_NUMERIC" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="11" NAME="STREET_NAME" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="12" NAME="CITY_NAME" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="13" NAME="PROVINCE" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="14" NAME="COUNTRY" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="15" NAME="POSTAL_CODE" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="16" NAME="POST_OFFICE_BOX" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
You can auto generate new format files with bcp if you need it for other files/tables (no need to do this now if the format file works)
Important:
After format file export:
Replace: TERMINATOR="\"," to TERMINATOR='","'
Replace: TERMINATOR="\r\n" to TERMINATOR='"\r\n'*
Add to <record> (will remove first quotation): <FIELD ID="0" xsi:type="CharTerm" TERMINATOR='"' MAX_LENGTH="1" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
Auto generate query:
DECLARE #importTable VARCHAR(128) = 'hc.dbo.QRYM_COMPANIES';
DECLARE #formatFilePath VARCHAR(8000) = '\\YourFilePath\formatFile.txt';
DECLARE #bcpCall VARCHAR(8000);
SET #bcpCall = 'bcp "' + #importTable + '" format nul -c -t", -x -f "' + #formatFilePath + '" -T -S ' + ##SERVERNAME;
PRINT #bcpCall
EXEC xp_cmdshell #bcpCall