SQL bulk insert csv with currency sign separator ¤ - sql

I have a .csv file with a currency sign field separator (¤), when I execute this query to bulk load it to a table it raise an error.
The file is UTF-8 encoded.
BULK INSERT dbo.test
FROM 'file.csv'
WITH (DATA_SOURCE = 'MyAzureBlobStorage',
FIRSTROW = 2,
CODEPAGE = 65001, --UTF-8 encoding
FIELDTERMINATOR = '¤', --CSV field delimiter
ROWTERMINATOR = '\n' --Use to shift the control to next row
);
The error I get is:
The bulk load failed. The column is too long in the data file for row 1, column 1. Verify that the field terminator and row terminator are specified correctly.
This is working fine with a semicolon as the separator.

Related

Bulk insert error "Bulk load data conversion error (type mismatch or invalid character for the specified codepage)"

I'm trying to do bulk insert from csv file with the following script:
BULK INSERT x.tableName
FROM 'PATH OF FILE'
WITH (
FIRSTROW = 2,
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
ERRORFILE = 'C:\CSVDATA\SchoolsErrorRows.csv',
TABLOCK )
My issue is that I have a column in the database of type Datetime2 and the format of the column in the csv file like this : 2020-12-01 20:15:00 +0300
I got this error:
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 2, column 1 (Date).
How can I solve this issue?

Bulk insert SQL command cannot insert first row

I'm using a bulk insert command for SQL Server but for some reason the first row isn't being inserted. Why can't I insert data from the first row? Is bulk insert expecting headers as default and how can i circumvent this? If I add a dummy row and set WITH to FIRSTROW = 2 then the first row is inserted without a problem but I don't think this is a nice solution.
Error code:
Msg 4864, Level 16, State 1, Line 1
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 1 (table_id).
Command:
BULK INSERT TableData
FROM 'C:\Users\Oscar\file.csv'
WITH (FIELDTERMINATOR = ';',
ROWTERMINATOR = '\n',
KEEPNULLS,
KEEPIDENTITY)
Sample data:
1;Text 1;1;0;;
2;Text 2;1;0;;
3;Text 3;1;0;;
4;Text 4;1;0;;
5;Text 5;1;0;;

The script is probably utf-8 and you're trying to load it from cmd with cp-1252 or something, the UTF-Bom at the beginning freaks out the interpreter.
Look with a hexeditor and you'll see it.
Save as ANSI and try again.

I've had problems with Line Feed/Carriage Return Line Feed, this might be the issue here as well. For Line Feed I had to use a Row terminator of 0x0a:
BULK INSERT TableData
FROM 'C:\Users\Oscar\file.csv'
WITH (
FIELDTERMINATOR = ';',
ROWTERMINATOR = '0x0a',
KEEPNULLS,
KEEPIDENTITY)

SQL Server Bulk Insert with condition if the line is empty, do not read it

I just want to ask if there is a way to use the bulk insert feature but to check if the last line is empty and skip it.
I have a text file that is being populated with data but the last line will always be empty cause when it repopulates, it will start from there and the the end of the previous line that is already populated.
My query so far looks like this:
BULK INSERT #TEMP
FROM 'C:\Test\Test.txt'
WITH (FIELDTERMINATOR ='\t',
ROWTERMINATOR = '\r',
FIRSTROW = 2, KEEPNULLS)
It will then be input into a temp table but the query will not go this far because of the last line in the text file. Is there a setting to say, skip the last line if its empty?

If you're expecting one line to be nonimportable then include
MAXERRORS =1
In the command and BULK INSERT should import the other ones fine.
BULK INSERT #TEMP
FROM 'C:\Test\Test.txt'
WITH (FIELDTERMINATOR ='\t',
ROWTERMINATOR = '\r',
MAXERRORS =1,
FIRSTROW = 2, KEEPNULLS)
Unless there's an unexpected problem in another line (in which case you'll want to trap the error anyway)
https://msdn.microsoft.com/en-us/library/ms188365.aspx

Well i found my answer here:Difference between CR LF, LF and CR line break types?
What i did is changed the from ROWTERMINATOR = '\r' to ROWTERMINATOR = '0x0a'
You can read up on the link about the differences in end of line chars which soved my issue. I hope this works for others as well! Best of luck guys!

SQL Server Bulk Import ROWTERMINATOR vbCrLf (\n) not working

I know there are other threads on here regarding this problem and none of the recommended fixes are working. I have verified that the CRLF at the end of each record is a hex '0D0A'. I can do a replace in VBS on vbCrLf and it replaces every one of them.
Here is a sample of my tab-delimited text file:
01/16/2013 11:00 HS01 DocLast, DocFirst PA-C Occurred ML 11/20/2012 15:31
01/07/2013 09:40 HS01 DocLast, DocFirst PA-C Canceled ML 11/20/2012 15:36 Patient Canceled 20130103-57935
I am executing this code against the text file in a stored procedure in SQL Server 2008:
set #sqlcmd = '
BULK INSERT #temp_import_records
FROM ''' + #import_file + '''
WITH
(
ROWTERMINATOR = ''\n''
)'
I am trying to insert this text into a temp table with 20 columns. This data only has 10 fields. With my code, both of these records are being installed into the same record in the temp table. I have tried setting the rowterminator to '0D0A' and '0x0A' and neither of them worked.
What am I doing wrong?

For anybody landing here now, you can try using HEX codes
ROWTERMINATOR = '0x0d0a' or ROWTERMINATOR = '0x0a'

The \n character is only the LF (0A) part of the two character CR+LF (0D0A) row terminator. Use \r for the first part:
ROWTERMINATOR = ''\r\n''

I suggest also escaping the termination like so:
ROWTERMINATOR = ''\\n''

Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 4 (Year)

I'm getting the conversion error when I try to import a text file to my database. Below is the error message I received:
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 4 (Year).
Here is my query code:
CREATE TABLE Students
(
StudentNo Integer NOT NULL Primary Key,
FirstName VARCHAR(40) NOT NULL,
LastName VARCHAR(40) NOT NULL,
Year Integer,
GPA Float NULL
);
Here is the sample data from text file:
100,Christoph,Van Gerwen,2011
101,Anar,Cooke,2011
102,Douglis,Rudinow,2008
I think I know what the problem is..Below is my bulk insert code:
use xta9354
bulk insert xta9354.dbo.Students
from 'd:\userdata\xta9_Students.txt'
with (fieldterminator = ',',rowterminator = '\n')
With the sample data, there is no ',' after the Year attribute even tho there is still another attribute Grade after the Year which is NULL
Can someone please tell me how to fix this?

Try using a format file since your data file only has 4 columns. Otherwise, try OPENROWSET or use a staging table.
myTestFormatFiles.Fmt may look like:
9.0
4
1 SQLINT 0 3 "," 1 StudentNo ""
2 SQLCHAR 0 100 "," 2 FirstName SQL_Latin1_General_CP1_CI_AS
3 SQLCHAR 0 100 "," 3 LastName SQL_Latin1_General_CP1_CI_AS
4 SQLINT 0 4 "\r\n" 4 Year "
(source: microsoft.com)
This tutorial on skipping a column with BULK INSERT may also help.
Your statement then would look like:
USE xta9354
GO
BULK INSERT xta9354.dbo.Students
FROM 'd:\userdata\xta9_Students.txt'
WITH (FORMATFILE = 'C:\myTestFormatFiles.Fmt')

In my case, I was dealing with a file that was generated by hadoop on a linux box. When I tried to import to sql I had this issue. The fix wound up being to use the hex value for 'line feed' 0x0a. It also worked for bulk insert
bulk insert table from 'file'
WITH (FIELDTERMINATOR = ',', ROWTERMINATOR = '0x0a')

We use the bulk insert as well. The file we upload is sent from an external party. After a while of troubleshooting, I realized that their file had columns with commas in it. Just another thing to look for...

The above options works for Google big query file also. I exported a table data to goodle cloud storage and downloaded from there. While loading the same to sql server was facing this issue and could successfully load the file after specifying the row delimiter as
ROWTERMINATOR = '0x0a'
Pay attention to header record as well and specify
FIRSTROW = 2
My final block for data file export from google bigquery looks like this.
BULK INSERT TABLENAME
FROM 'C:\ETL\Data\BigQuery\In\FILENAME.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '0x0a',--Files are generated with this row terminator in Google Bigquery
TABLOCK
)

My guess is it's an encoding problem, for instance your file is UTF-8 but SQL will not read it the way it should, so it attempts to insert 100ÿ or something along these lines into your table.
Possible fixes:
Specify Codepage
Change the Encoding of the source using Powershell
Code samples:
1.
BULK INSERT myTable FROM 'c:\Temp\myfile.csv' WITH (
FIELDTERMINATOR = '£',
ROWTERMINATOR = '\n',
CODEPAGE = 'ACP' -- ACP corresponds to ANSI, also try UTF-8 or 65001 for Unicode
);
2.
get-content "myfile.csv" | Set-content -Path "myfile.csv" -Encoding String
# String = ANSI, also try Ascii, Oem, Unicode, UTF7, UTF8, UTF32

Added MSSQLSERVER full access to the folder, diskadmin and bulkadmin server roles.
In my c# application, when preparing for the bulk insert command,
string strsql = "BULK INSERT PWCR_Contractor_vw_TEST FROM '" + strFileName + "' WITH (FIELDTERMINATOR = ',', ROWTERMINATOR = '\\n')";
And I get this error - Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 8 (STATUS).
I looked at my logfile and found that the terminator becomes ' ' instead of '\n'.
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error:
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)". Query :BULK INSERT PWCR_Contractor_vw_TEST FROM 'G:\NEWSTAGEWWW\CalAtlasToPWCR\Results\parsedRegistration.csv' WITH (FIELDTERMINATOR = ',', **ROWTERMINATOR = ''**)
So I added extra escape to the rowterminator - string strsql = "BULK INSERT PWCR_Contractor_vw_TEST FROM '" + strFileName + "' WITH (FIELDTERMINATOR = ',', ROWTERMINATOR = '\\n')";
And now it inserts successfully.
Bulk Insert SQL - ---> BULK INSERT PWCR_Contractor_vw_TEST FROM 'G:\\NEWSTAGEWWW\\CalAtlasToPWCR\\Results\\parsedRegistration.csv' WITH (FIELDTERMINATOR = ',', ROWTERMINATOR = '\n')
Bulk Insert to PWCR_Contractor_vw_TEST successful... ---> clsDatase.PerformBulkInsert