BULK INSERT SQL, Need to separate the data - sql

Question

I'm loading a Notepad txt file into a SQL Table. I'm trying to use the BULK INSERT command but I keep getting this error:
Msg 4863, Level 16, State 1, Line 8
Bulk load data conversion error (truncation) for row 1, column 3 (column3).
The txt file has each column separated by a | symbol. I just need each set of text between each | | to be in it's own column.
For example:
|100|AA|BCD|200|
I need each of those to be separated into a column in a table. My code may be too simple right now but any help would be appreciated.
CREATE TABLE BMData2 (
column1 varchar(30),
column2 varchar(30),
column3 character(3),
column4 varchar(10),
column5 varchar(10),
column6 varchar(10),
column7 varchar(10),
column8 varchar(10),
column9 varchar(10),
column10 varchar(10),
column11 varchar(10),
column12 varchar(10),
column13 varchar(10),
column14 varchar(10),
column15 varchar(10),
column16 varchar(10),
column17 varchar(10),
column18 varchar(10),
column19 varchar(10),
column20 varchar(10),
column21 varchar(10),
column22 varchar(10),
column23 varchar(10),
column24 varchar(10),
column25 varchar(10),
column26 varchar(10),
)
BULK INSERT BMData
FROM '\\DBV\march_june\All march june Data.txt'
WITH
(Fieldterminator = '|',
ROWTERMINATOR = '\n');
Line of data looks like this:
AB|1410|MTH|ART|20150401|3|1600|1600|1556|2048|2048|2101|0|0|168|185|-4|13|17|1630|2054|ARTPROJECT|34|7|144||0|0|0|0|0|0|0|0|0||0|0|0|0|||0|0|0|0|||0|0|0|0|||0|0|0|0|||0|0|0|0|
I really only need 5 of these "data points" i.e. |data point| but my data comes like this and there are so many rows that the clean up is impossible in Excel or Notepad.

Answer 1

Remove the column headers in your text file.
BULK INSERT (Transact-SQL)
The FIRSTROW attribute is not intended to skip column headers.
Skipping headers is not supported by the BULK INSERT statement. When
skipping rows, the SQL Server Database Engine looks only at the field
terminators, and does not validate the data in the fields of skipped
rows.
Edit: Your sample data does not match the table you present. You table has 26 columns but the sample data has about double.
Create a test file with the following test data (no column headers):
Note that the last column has a NULL value. It is after the last | and before the NewLine character.
AB|1410|MTH|ART|20150401|3|1600|1600|1556|2048|2048|2101|0|0|168|185|-4|13|17|1630|2054|ARTPROJECT|34|7|144|
AB|1410|MTH|ART|20150401|3|1600|1600|1556|2048|2048|2101|0|0|168|185|-4|13|17|1630|2054|ARTPROJECT|34|7|144|
AB|1410|MTH|ART|20150401|3|1600|1600|1556|2048|2048|2101|0|0|168|185|-4|13|17|1630|2054|ARTPROJECT|34|7|144|

Answer 2

Use xml format file instead. It works much better. Here is sample based on #Ricardo C sample data.
1. Prepare xml file and save (c:\temp\bulk.xml in this example).
<?xml version="1.0" encoding="utf-8" ?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<!--list all fields in your .txt here-->
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="6" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="7" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="8" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="9" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="10" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="11" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="12" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="13" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="14" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="15" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="16" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="17" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="18" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="19" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="20" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="21" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="22" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="23" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="24" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="25" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="510" />
<FIELD ID="26" xsi:type="CharTerm" TERMINATOR='\r\n' MAX_LENGTH="510" />
<!-- the last is end of row -->
</RECORD>
<ROW>
<!-- List only what you need -->
<COLUMN SOURCE="1" NAME="Id" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="2" NAME="Id1" xsi:type="SQLINT"/>
<COLUMN SOURCE="3" NAME="Name" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
Usage:
insert dbo.myTable
SELECT *
FROM OPENROWSET(BULK 'C:\temp\bulk.txt',
FORMATFILE='C:\temp\bulk.xml'
) AS t1;

Answer 3

I would use PowerShell to pre-process the original file and extract only the columns I need and put the extracted data to a new file.
Let's say the original source file is located at c:\temp\src.txt as follows(put your column name in the first line for easy process):
col1|col2|col3|col4|col5|col6|col7|col8|col9|col10
1|2|3|4|5|6|7|8|9|10
11|12|13|14|15|16|17|18|19|20
21|22|23|24|25|26|27|28|29|30
31|32|33|34|35|36|37|38|39|40
41|42|43|44|45|46|47|48|49|50
51|52|53|54|55|56|57|58|59|60
61|62|3|64|65|66|67|68|69|70
Now let's say I only need to get col2, col4, col5 and col9 data. So here is the PowerShell code
[string[]]$col_wanted = 'col2', 'col4', 'col5', 'col9'; #only need four columns out of 10 columns
$csv = import-csv -path c:\temp\src.txt -Delimiter '|';
$t=($col_wanted -join "|") + "`r`n";
foreach ($c in $csv)
{
$col_wanted | % -begin {[string]$s='';} -process {$s+=$c.$_+'|';} -end {$s = $s.Substring(0, $s.Length-1) + "`r`n"}
$t += $s
}
$t | Out-File -FilePath c:\temp\target.txt -Force;
If we open c:\temp\target.txt, we will see the result like this:
col2|col4|col5|col9
2|4|5|9
12|14|15|19
22|24|25|29
32|34|35|39
42|44|45|49
52|54|55|59
62|64|65|69
Now you can use bulk insert to do the data importing, but since we have the first row as column header, in the bulk insert, we need to set firstrow = 2, i.e. like the following
bulk insert MyTable
from 'c:\temp\target.txt'
with (FIELDTERMINATOR ='|', firstrow=2);