Bulk insert csv file with semicolon as delimiter - sql

I'm trying to import data from semicolon separated csv file into a SQL Server database. Here is the table structure
CREATE TABLE [dbo].[waste_facility]
(
[Id] INT IDENTITY (1, 1) NOT NULL,
[postcode] VARCHAR (50) NULL,
[name] VARCHAR (50) NULL,
[type] VARCHAR (255) NULL,
[street] VARCHAR (255) NULL,
[suburb] VARCHAR (255) NULL,
[municipality] VARCHAR (255) NULL,
[telephone] VARCHAR (255) NULL,
[website] VARCHAR (255) NULL,
[longtitude] DECIMAL (18, 8) NULL,
[latitude] DECIMAL (18, 8) NULL,
PRIMARY KEY CLUSTERED ([Id] ASC)
);
The csv file is shown below:
Location Coordinate;Feature Extent;Projection;Postcode;Name Of Facility;Type Of Facility;Street;Suburb;Municipality;Telephone Number;Website;Easting Coordinate;Northing Coordinate;Longitude Coordinate;Latitude Coordinate;Google Maps Direction
-37.9421182892,145.3193857967;"{""coordinates"": [145.3193857967, -37.9421182892], ""type"": ""Point""}";MGA zone 55;3156;Cleanaway Lysterfield Resource Recovery Centre;Recovery Centre;840 Wellington Road;LYSTERFIELD;Yarra Ranges;9753 5411;https://www.cleanaway.com.au/location/lysterfield/;352325;5799275;145.31938579674124;-37.94211828921733;https://www.google.com.au/maps/dir//-37.94211828921733,145.31938579674124/#your+location,17z/data=!4m2!4m1!3e0
-38.0529529215,145.2433557709;"{""coordinates"": [145.2433557709, -38.0529529215], ""type"": ""Point""}";MGA zone 55;3175;Smart Recycling (South Eastern Depot);Recycling Centre;185 Dandenong-Hastings Rd;LYNDHURST;Greater Dandenong;8787 3300;https://smartrecycling.com.au/;345876;5786853;145.24335577090602;-38.05295292152536;https://www.google.com.au/maps/dir//-38.05295292152536,145.24335577090602/#your+location,17z/data=!4m2!4m1!3e0
-38.0533129717,145.267610135;"{""coordinates"": [145.267610135, -38.0533129717], ""type"": ""Point""}";MGA zone 55;3976;Hampton Park Transfer Station (Outlook Environmental);Transfer Station;274 Hallam Road;HAMPTON PARK;Casey;9554 4502;https://www.suez.com.au/en-au/who-we-are/suez-in-australia-and-new-zealand/our-locations/waste-management-hampton-park-transfer-station;348005;5786853;145.2676101350274;-38.053312971691255;https://www.google.com.au/maps/dir//-38.053312971691255,145.2676101350274/#your+location,17z/data=!4m2!4m1!3e0
-38.1243050577,145.2183465487;"{""coordinates"": [145.2183465487, -38.1243050577], ""type"": ""Point""}";MGA zone 55;3977;Frankston Regional Recycling and Recovery Centre;Recycling Centre;20 Harold Road;SKYE;Frankston;1300 322 322;https://www.frankston.vic.gov.au/Environment-and-Waste/Waste-and-Recycling/Frankston-Regional-Recycling-and-Recovery-Centre-FRRRC/Accepted-Items-at-FRRRC;343833;5778893;145.21834654873447;-38.12430505770815;https://www.google.com.au/maps/dir//-38.12430505770815,145.21834654873447/#your+location,17z/data=!4m2!4m1!3e0
-38.0973208774,145.4920399066;"{""coordinates"": [145.4920399066, -38.0973208774], ""type"": ""Point""}";MGA zone 55;3810;Pakenham Waste Transfer Station (Future Recycling);Transfer Station;30-32 Exchange Drive;PAKENHAM;Cardinia;13Recycling;https://www.futurerecycling.com.au/;367776;5782313;145.4920399066473;-38.09732087738631;https://www.google.com.au/maps/dir//-38.09732087738631,145.4920399066473/#your+location,17z/data=!4m2!4m1!3e0
There are some columns that I don't need, so I create a format file to import the data. The format file is shown as below
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharFixed" LENGTH="50"/>
<FIELD ID="12" xsi:type="CharFixed" LENGTH="50"/>
<FIELD ID="13" xsi:type="CharFixed" LENGTH="50"/>
<FIELD ID="2" xsi:type="CharFixed" LENGTH="50" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharFixed" LENGTH="50" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharFixed" LENGTH="255" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharFixed" LENGTH="255" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="6" xsi:type="CharFixed" LENGTH="255" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="7" xsi:type="CharFixed" LENGTH="255" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="8" xsi:type="CharFixed" LENGTH="255" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="9" xsi:type="CharFixed" LENGTH="255" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="14" xsi:type="CharFixed" LENGTH="50"/>
<FIELD ID="15" xsi:type="CharFixed" LENGTH="50"/>
<FIELD ID="10" xsi:type="CharFixed" LENGTH="41"/>
<FIELD ID="11" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="41"/>
<FIELD ID="16" xsi:type="CharFixed" LENGTH="50"/>
</RECORD>
<ROW>
<COLUMN SOURCE="2" NAME="postcode" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="3" NAME="name" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="4" NAME="type" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="5" NAME="street" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="6" NAME="suburb" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="7" NAME="municipality" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="8" NAME="telephone" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="9" NAME="website" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="10" NAME="longtitude" xsi:type="SQLDECIMAL" PRECISION="18" SCALE="8"/>
<COLUMN SOURCE="11" NAME="latitude" xsi:type="SQLDECIMAL" PRECISION="18" SCALE="8"/>
</ROW>
</BCPFORMAT>
Then I tried both bulk insert and bcp in - neither of them works.
Here is the bulk insert command
USE [waste-facility-locations];
BULK INSERT [dbo].[waste_facility]
FROM 'E:\onboardingIteration\waste-facility-locations.csv'
WITH (FORMATFILE = 'E:\onboardingIteration\waste_facility_formatter.xml',
FIRSTROW = 2,
LASTROW = 6,
FIELDTERMINATOR = ';',
ROWTERMINATOR = '\n',
ERRORFILE = 'E:\onboardingIteration\myRubbishData.log');
But unlucky some error file were generated. Here is what myRubbishData.log error says:
Row 2 File Offset 1993 ErrorFile Offset 0 - HRESULT 0x80004005
And the actual row stored in myRubbishData.txt:
;Pakenham Waste Transfer Station (Future Recycling);Transfer Station;30-32 Exchange Drive;PAKENHAM;Cardinia;13Recycling;https://www.futurerecycling.com.au/;367776;5782313;145.4920399066473;-38.09732087738631;https://www.google.com.au/maps/dir//-38.09732087738631,145.4920399066473/#your+location,17z/data=!4m2!4m1!3e0;Pakenham Waste Transfer Station (Future Recycling);Transfer Station;30-32 Exchange Drive;PAKENHAM;Cardinia;13Recycling;https://www.futurerecycling.com.au/;367776;5782313;145.4920399066473;-38.09732087738631;https://www.google.com.au/maps/dir//-38.09
As you can see, it seems like rows are not correctly separated. So I tried to change the row delimiter to "\n","\r","\n\r","\r\n", none of them work.
And I tried bcp. It did not work either.
Here is the bcp command I used:
bcp [waste-facility-locations].[dbo].[waste_facility] in "E:\onboardingIteration\waste-facility-locations.csv" -f "E:\onboardingIteration\waste_facility_formatter.xml" -T -S "(LocalDB)\MSSQLLocalDB" -F 2 -t ";" -r "\n"
Then I get an error said somehow the same thing
SQLState = S1000, NativeError = 0
Error = [Microsoft][ODBC Driver 17 for SQL Server]Unexpected EOF encountered in BCP data-file
0 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total : 1
One interesting things is, if I create a new excel and choose "Get data" option to import the csv file, the file can be literally correctly parsed.
Basically I can't find what I did wrong here. Can someone help me on this one?

The SQL Server import facilities are very intolerant of bad data and even just formatting variations or options. In my career, I have literally spent thousands of work-hours trying to develop and debug import procedures for customers. I can tell you right now, that trying to fix this with SQL alone is both difficult and time-consuming.
When you have this problem (bad data and/or inconsistent formatting) it is almost always easier to find or develop a more flexible tool to pre-process the data into the rigid standard that SQL expects. So I would say that if Excel can parse it then just use Excel automation to pre-process them and then use SQL to import the Excel output. If that's not practical for you, then I'd advise writing your own tool in some client language (C#, Vb, Java, Python, etc.) to pre-process the files.
You can do it in SQL (and I have done it many times), but I promise you that it is a long complicated trek.
SSIS has more flexible error-handling for problems like this, but if you are not already familiar and using it, it has a very steep learning curve and your first SSIS project is likely to be very time-consuming also.

Related

How to insert bulk csv file into SQL Server

I have a csv file that have columns like this
"Hold","EmpID","Source","Shard-Exists","Type"
but my DB table look like this
//Note: My Id is auto increment
"Id","Hold","EmpID","Source","Type","CurrentDate"
I'm just wondering how can bulk insert my csv file into the database table without the shard-Exist column and also passing the Current Date automatically.
Any help or suggestion will be really appreciated
TRUNCATE TABLE dbo.Actors;
GO
-- import the file
BULK INSERT dbo.Actors
FROM 'C:\Documents\Skyvia\csv-to-mssql\actor.csv'
WITH
(
FORMAT='CSV',
FIRSTROW=2
)
GO
You should be able to use a format file to accomplish the 'skip columns in table' task. I'll modify the example from the MS docs.
<?xml version="1.0"?>
<BCPFORMAT xmlns="https://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="7"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="25" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="6" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="30" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="2" NAME="Hold" xsi:type="SQLINT"/>
<COLUMN SOURCE="3" NAME="EmpID" xsi:type="SQLINT"/>
<COLUMN SOURCE="4" NAME="Source" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="6" NAME="Type" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
Note, in the <ROW> portion that I'm not specifying anything for the ID or CurrentDate columns. It's also noteworthy that there's no SOURCE="5"; that's how the Shard-Exists field in the source data is being skipped.
As to auto-generating a value for CurrentDate, my recommendation would be to add a default constraint to your table. that can be done like so:
ALTER TABLE dbo.Actors
ADD CONSTRAINT DF_Actors__CurrentDate
DEFAULT (getdate()) FOR CurrentDate;

How to import semicolon delimited text file into SQL Server

I'm trying to import a semicolon delimited file into SQL Server. I used bcp to try to create an XML file, but I'm getting errors.
The text file (data) looks like this:
customer_id;remed_date;assumed_closed;exempt_ind;refresh_date;Target_date;due_date
2;06/06/2015;True;False;06/13/2015;06/13/2020;
3;08/02/2014;False;False;;08/02/2019;
The XML file came out like this:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR=";" MAX_LENGTH="12"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR=";" MAX_LENGTH="11"/>
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR=";" MAX_LENGTH="1"/>
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR=";" MAX_LENGTH="1"/>
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR=";" MAX_LENGTH="11"/>
<FIELD ID="6" xsi:type="CharTerm" TERMINATOR=";" MAX_LENGTH="11"/>
<FIELD ID="7" xsi:type="CharTerm" TERMINATOR="0x0A" MAX_LENGTH="11"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="Customer_ID" xsi:type="SQLINT"/>
<COLUMN SOURCE="2" NAME="Remed_Date" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="3" NAME="Assumed_Closed" xsi:type="SQLBIT"/>
<COLUMN SOURCE="4" NAME="Exempt_Ind" xsi:type="SQLBIT"/>
<COLUMN SOURCE="5" NAME="Refresh_Date" xsi:type="SQLDATE"/>
<COLUMN SOURCE="6" NAME="Target_Date" xsi:type="SQLDATE"/>
<COLUMN SOURCE="7" NAME="Due_Date" xsi:type="SQLDATE"/>
</ROW>
</BCPFORMAT>
The table looks like this:
[ODS_Customer_ID] [int] NOT NULL,
[Remed_Date] [date] NOT NULL,
[Assumed_Closed] [bit] NOT NULL,
[Exempt_Ind] [bit] NOT NULL,
[Refresh_Date] [date] NOT NULL,
[Target_Date] [date]
[Due_Date] [date]
When I try to run a BULK INSERT:
BULK INSERT MXB.dbo.RefreshSuppression
FROM '\\SRVR1\Data\MXB\Automated\BSA_AML_Suppression.txt'
(FORMATFILE = '\\SRVR1\Scripts\MXB\Weekly\MXBRefreshSupp.xml');
I get this error:
Msg 4864, Level 16, State 1, Line 26
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 1 (ODS_Customer_ID).
Msg 4866, Level 16, State 8, Line 26
The bulk load failed. The column is too long in the data file for row 1, column 7. Verify that the field terminator and row terminator are specified correctly.
Msg 7399, Level 16, State 1, Line 26
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 26
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
Any ideas?
For message 4864, you do not need to convert your file to an XML to perform a bulk insert. Try:
BULK INSERT MXB.dbo.RefreshSuppression
FROM '\\SRVR1\Data\MXB\Automated\BSA_AML_Suppression.txt'
WITH (
FIELDTERMINATOR = ';',
ROWTERMINATOR = '\n'
)
GO
More options on bulk import can be found here.
For message 4866, you are missing your 7th column in your file.
Messages 7330 & 7399 appear to be related to your linked server, not your bulk upload. You can find the microsoft support article for them here.
Your data only has six columns. There should be three dates after the boolean 'Exempt_Ind', but I only count two.

Bulk insert not recognizing row terminators

I'm preparing format files to import text-qualified files into sql-server based on This article
Sample of data for import:
"1000000"|"1100000"|"2017-02-26 00:00:00"|"CAT1"|"Item from CAT1"
"1000001"|"1100000"|"2017-02-26 00:00:01"|"CAT2"|"Item from CAT2"
"1000002"|"1100001"|"2017-02-26 00:01:02"|"CAT2"|"Item from CAT2"
"1000003"|"1100002"|"2017-02-26 01:02:03"|"CAT3"|"Item from CAT3"
My format file:
13.0
6
1 SQLCHAR 0 0 "\"" 0 FIRST_QUOTE SQL_Latin1_General_CP1_CI_AS
2 SQLINT 0 4 "\"|\"" 1 transaction_id ""
3 SQLINT 1 4 "\"|\"" 2 user_id ""
4 SQLDATETIME 0 8 "\"|\"" 3 create_date ""
5 SQLCHAR 2 10 "\"|\"" 4 category SQL_Latin1_General_CP1_CI_AS
6 SQLCHAR 2 50 "\"\r\n" 5 item SQL_Latin1_General_CP1_CI_AS
Which results in:
The bulk load failed. The column is too long in the data file for row 1, column 6. Verify that the field terminator and row terminator are specified correctly.
I'm fairly certain that file contains \r\n (Checked with Hex editor shows 0x0d,0x0a), although ignoring text qualifiers and format file i was able to import it manually only with
Bulk insert <table_name> from '\\path\to\file' with (fieldterminator='|', rowterminator='\n')
I tried poking at your format file, but it just confirmed that I am not any good with that version of the format file.
Switching over to an xml format file was easy enough though.
cat.xml:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR='"' COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR='"|"' COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR='"|"' COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR='"|"' COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR='"|"' COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="6" xsi:type="CharTerm" TERMINATOR='"\r\n' COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="2" NAME="transaction_id" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="3" NAME="user_id" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="4" NAME="create_date" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="5" NAME="category" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="6" NAME="item" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
sql:
bulk insert dbo.cat
from 'c:\cat.txt'
with (
formatfile = 'c:\cat.xml'
, firstrow = 1
);
results:
input file, showing row terminator in notepad++:

Bulk Insert using format file for varying number of columns between datafile and actual database table

This is the Schema for a table
Create table dbo.Project
(
ProjectID (int,not null)
ManagerID (int,not null)
CompanyID(int, not null)
Title (nvarchar(50),not null)
StartDate(datetime,not null)
EndDate(datetime,null)
ProjDescription(nvarchar(max))
)
I created a datafile called bob.dat from this table which has around 15 rows with the following bcp command
bcp "Select ProjectID,ManagerID,CompanyID,Title,StartDate from CATS.dbo.Project" queryout "C:\Documents\bob.dat" -Sbob-pc -T -n
Also a format/mapping file called bob.fmt was created using the following bcp command
bcp CATS.dbo.Project format nul -f C:\Documents\bob.fmt -x -Sbob-pc -T -n
Then i created a copy of the table Project.
Create table dbo.ProjectCopy
(
ProjectID (int,not null)
ManagerID (int,not null)
CompanyID(int, not null)
Title (nvarchar(50),not null)
StartDate(datetime,not null)
EndDate(datetime,null)
ProjDescription(nvarchar(max))
)
What i want to do now is use the bob.dat and bob.format file to populate this table ProjectCopy using the following Bulk Insert statement.
BULK INSERT CATS.dbo.ProjectCopy
FROM 'C:\Documents\bob.dat'
WITH (FORMATFILE = 'C:\Documents\bob.fmt',
LASTROW=5,
KEEPNULLS,
DATAFILETYPE='native');
GO
SELECT * FROM CATS.dbo.ProjectCopy
GO
So basically the data file does not contain any data for the columns EndDate and ProjDescription. I want these two columns to be remained as null. Unfortunately i get the
following error when i run the bulk insert statement.
Msg 4863, Level 16, State 4, Line 2
Bulk load data conversion error (truncation) for row 1, column 6 (EndDate).
Msg 7399, Level 16, State 1, Line 2
The OLE DB provider "BULK" for linked server "(null)" reported an error.
The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 2
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
(0 row(s) affected)
Anyone got any clue how this could be fixed?
just to inform you all i have already been to this sections and the solution provided there
didn't work out for me.
BULK INSERT with inconsistent number of columns,
Can't identify reason for BULK INSERT errors,
BULK INSERT with inconsistent number of columns
First of all, for a creating a datafile called bob.dat you need to add two columns: EndDate and ProjDescription. In addition, for the bulk copy operation using Unicode-characters must be added the argument -W.
Example:
bcp "Select ProjectID,ManagerID,CompanyID,Title,StartDate, NULL AS EndDate, NULL AS ProjDescription from CATS.dbo.Project" queryout "C:\Users\Pawan\Documents\bob.dat" -Sbob-pc -T -n -w
The original format file:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
<FIELD ID="2" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
<FIELD ID="3" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
<FIELD ID="4" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="100" COLLATION="Cyrillic_General_CI_AS"/>
<FIELD ID="5" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="48"/>
<FIELD ID="6" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="48"/>
<FIELD ID="7" xsi:type="NCharTerm" TERMINATOR="\r\0\n\0" COLLATION="Cyrillic_General_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="ProjectID" xsi:type="SQLINT"/>
<COLUMN SOURCE="2" NAME="ManagerID" xsi:type="SQLINT"/>
<COLUMN SOURCE="3" NAME="CompanyID" xsi:type="SQLINT"/>
<COLUMN SOURCE="4" NAME="Title" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="5" NAME="StartDate" xsi:type="SQLDATETIME"/>
<COLUMN SOURCE="6" NAME="EndDate" xsi:type="SQLDATETIME"/>
<COLUMN SOURCE="7" NAME="ProjDescription" xsi:type="SQLNVARCHAR"/>
</ROW>
</BCPFORMAT>
But you want a way to fill the data only upto StartDate. Therefore, this file needs to be changed:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
<FIELD ID="2" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
<FIELD ID="3" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
<FIELD ID="4" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="100" COLLATION="Cyrillic_General_CI_AS"/>
<FIELD ID="5" xsi:type="NCharTerm" TERMINATOR="\r\0\n\0" MAX_LENGTH="48"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="ProjectID" xsi:type="SQLINT"/>
<COLUMN SOURCE="2" NAME="ManagerID" xsi:type="SQLINT"/>
<COLUMN SOURCE="3" NAME="CompanyID" xsi:type="SQLINT"/>
<COLUMN SOURCE="4" NAME="Title" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="5" NAME="StartDate" xsi:type="SQLDATETIME"/>
</ROW>
</BCPFORMAT>

SQL Server: BULK INSERT from file that contains dates in format YYYYMMDD [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
bulk insert a date in YYYYMM format to date field in MS SQL table
I am using SQL Server 2012 Express.
I have a problem with using BULK INSERT from file that contains dates in format YYYYMMDD (can't change the format, it's an output from different software). I use a format file, as I have an identity column that should be skipped when inserting values. Table structure and BULK INSERT command are here:
CREATE TABLE [dbo].[daily](
[id] [bigint] IDENTITY(1,1) NOT NULL,
[ticker] [varchar](15) NOT NULL,
[ddate] [date] NOT NULL,
[dopen] [decimal](16, 8) NOT NULL,
[dhigh] [decimal](16, 8) NOT NULL,
[dlow] [decimal](16, 8) NOT NULL,
[dclose] [decimal](16, 8) NOT NULL,
[dvol] [int] NOT NULL,
[dopenint] [int] NOT NULL
);
BULK INSERT daily
FROM 'C:\IBM.TXT'
WITH (
FORMATFILE = 'C:\dailyformat.xml',
TABLOCK,
FIRSTROW = 2);
Tab-delimited data file (IBM.TXT) is here:
Symbol Date Open High Low Close Total Volume Total Open Interest
IBM 19620102 2.57000000 2.57000000 2.54000000 2.54000000 11704 0
IBM 19620103 2.54000000 2.56000000 2.54000000 2.56000000 8778 0
IBM 19620104 2.56000000 2.56000000 2.54000000 2.54000000 7878 0
IBM 19620105 2.53000000 2.53000000 2.48000000 2.49000000 11029 0
IBM 19620108 2.49000000 2.49000000 2.42000000 2.44000000 16431 0
IBM 19620109 2.45000000 2.50000000 2.45000000 2.47000000 14855 0
Format file (dailyformat.xml), generated by bcp, is here:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="15"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="11"/>
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="41"/>
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="41"/>
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="41"/>
<FIELD ID="6" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="41"/>
<FIELD ID="7" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="12"/>
<FIELD ID="8" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="12"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="ticker" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="2" NAME="ddate" xsi:type="SQLDATE"/>
<COLUMN SOURCE="3" NAME="dopen" xsi:type="SQLDECIMAL" PRECISION="16" SCALE="8"/>
<COLUMN SOURCE="4" NAME="dhigh" xsi:type="SQLDECIMAL" PRECISION="16" SCALE="8"/>
<COLUMN SOURCE="5" NAME="dlow" xsi:type="SQLDECIMAL" PRECISION="16" SCALE="8"/>
<COLUMN SOURCE="6" NAME="dclose" xsi:type="SQLDECIMAL" PRECISION="16" SCALE="8"/>
<COLUMN SOURCE="7" NAME="dvol" xsi:type="SQLINT"/>
<COLUMN SOURCE="8" NAME="dopenint" xsi:type="SQLINT"/>
</ROW>
</BCPFORMAT>
I get the following error:
Msg 206, Level 16, State 2, Line 1
Operand type clash: numeric is incompatible with date.
Please help to fix it without inserting date as number to a temporary table and then converting it to date by selecting it into other table. There should be some fix to my code.
Thanks!
Problem solved. I used
INSERT INTO daily
SELECT a.* FROM OPENROWSET(
BULK 'C:\IBM.TXT',
FORMATFILE = 'C:\dailyformat.xml',
FIRSTROW = 2
) as a
Try calling SET DATEFORMAT ymd before your 'BULK INSERT'