My procedure for bulk insert is below:
ALTER PROCEDURE [dbo].[usp_impt]
AS
BEGIN
Declare #SQL1 varchar(150), #path varchar(100),
#pathtable varchar(100), #date datetime
set #date = getdate()
-- set path for files
set #path= '\\ff\avc\ce\ed_imp\'
set #pathtable = #path + 'ABC20140723.csv'
-- Delete data from tables
delete from table1
-- set sql
set #SQL1 = "BULK INSERT dbo.table1 FROM '" + #pathtable
+ "' WITH (FIRSTROW = 3,MAXERRORS = 0,FIELDTERMINATOR = ',')"
-- Bulk insert
exec(#sql1)
end
It works fine except when my data has "Google,Inc", it is converted to "Google" "Inc".
I want to write FIELDTERMINATOR = '","' instead of ',', however, I don't know how to put it into my #sql1 string?
Also, is it recommended I write a format file? My csv file has 200 columns and rows. Do I need to write each row? Thanks for any advice.
Short answer is you can't do this with BULK INSERT as is. BULK INSERT can't tell what comma is a delimiter and what comma is part of the data. Your source data apparently doesn't have the string field values quoted (i.e. surrounded by quotes). For comma separated fields, if ANY of the data contains a comma, it must be surrounded by quotation marks or BULK INSERT will view it as a field terminator. You will have to update your source data to surround strings with quotation marks before inserting.
I ran into this frequently while running data loads from text files and csv files. The only solution was to get the source of the data to add quotes or to use a different delimiter, such as a pipe ( | ) or to switch to fixed width fields. In either case, the source data has to be fixed.
Related
First of all, this problem doesn't exist when the text is just English, but when I insert Arabic text, I got the problem.
Look at my code
CREATE PROCEDURE insertToPinTableCardActivation
(#callerID VARCHAR (200),
#vAccount VARCHAR (200))
AS
BEGIN
DECLARE #textEnglish NVARCHAR(1000) --missing length previously
SET #textEnglish = 'Dear customer, your Card linked to account number '+ #vAccount --missing set keyword
SET #textEnglish = #textEnglish + ' is now activated. Thank you for banking with us.'
SET #textEnglish = 'عزيزي الزبون، تم تفعيل بطاقة الصراف الآلي التابعة لحسابكم رقم ' + #vAccount
INSERT INTO pinData([CallerID], [body], [processed])
VALUES (#callerID, #textEnglish, 0)
END
The code creates a string of mix Arab and English, and then insert it to a table.
My problem is that look what it is being inserted to the table
even though i already made the field body as nvarchar
could you help please
Update 1
I am inserting from my sql server when executing the stored procedure
update 2
if i go to the table and insert the data manually in arab, the arab letters shows correctly
IF you want to insert Unicode string literals, you must prepend your string literal with an N prefix - try this:
SET #textEnglish = N'...(insert your Arabic text here)...'
Otherwise, your text is reverted back to a non-Unicode format before being stored - and that's why you lose the Arabic text....
And also: if you're concatenating with VARCHAR parameters, I'd recommend using an explicit CAST to NVARCHAR (include a length when casting!):
SET #textEnglish = N'Dear customer, your Card linked to account number ' + CAST(#vAccount AS NVARCHAR(100))
What i'm trying to do is read a text file and then use BULK INSERT to create a table.
This is an example of how the text file looks
TIME DATE USER_NAME VALUE
11:10:04 10/02/15 Irene I. Moosa
There are a lot of rows and i mean a lot but sometimes the time is empty or the end character is not just a simple enter and I'm trying to compensate for it
Is something like this possible:
BULK INSERT #TEMP FROM 'C:\QPR_Logs\Audit\MetricsServerAudit.txt'
WHERE [TIME] IS NOT NULL WITH (FIELDTERMINATOR =' ', ROWTERMINATOR = '\n')
Something like that if it reads a null value that it just skips the line?
For the end character I'm not exactly sure what to use.
Has anyone got any suggestions?
Try OPENROWSET. since you have custom row/column terminators, you might require a format file.
select t1.*
from openrowset(bulk 'c:\folder\file1.csv'
, formatfile = 'c:\folder\values.fmt'
, firstrow = 2) as t1
where t1.[TIME] is not null
I have this type of data in a column in my table,,,
QCIM1J77477, 4650125572, QCCR1J77891, 4650312729, QCCR1J74974 --- remove without comma
or
QCIM1E107498,QCIM1E109835,4650092399/ QCCR1E91190, -- remove 4650092399
I want only that string which starts from QC,remove apart from QC,
so please tell me how can I achive it?
Beneath a piece of t-sql script that creates a temporary table #t with temporary variables. Here the temporary table exists untill you break your session, temporary variables exist during the execution of the script. I have a drop table statement at the bottom. Figure out yourself what you want with the table data and whether you want the data put in somewhere else, for example in a not-temporary table :).
I assume you want all the pieces of the string that contain 'QC' as seperate values. If you want your data back as it was originally, that is multiple strings per one column, then you could also do a group by trick. Then you do need a unique identifier of some sort, like name, id, guid of each row or identity.
create table #t ([QCs] nvarchar(100))
declare #str nvarchar(500)
set #str = 'QCIM1E107498,QCIM1E109835,4650092399/ QCCR1E91190'
--replace the above temporary variable with the column you are selecting
declare #sql nvarchar(4000)
select #sql = 'insert into #t select '''+ replace(#str,',',''' union all select ''') + ''''
print #sql
exec ( #sql )
select
QCs
,PATINDEX('%QC%',QCs) as StartPosition
,SUBSTRING(QCs,PATINDEX('%QC%',QCs),12) as QCsNew
from #t where QCs like '%QC%'
drop table #t
With PATINDEX you find the position where in the string 'QC' starts, and with SUBSTRING you tell the dbms to give back (here) 12 characters starting from the found StartPosition.
Beneath what the result looks like. QCsNew is your desired result.
QCs StartPosition QCsNew
QCIM1E107498 1 QCIM1E107498
QCIM1E109835 1 QCIM1E109835
4650092399/ QCCR1E91190 13 QCCR1E91190
I can't seem to figure out how this is happening.
Here's an example of the file that I'm attempting to bulk insert into SQL server 2005:
***A NICE HEADER HERE***
0000001234|SSNV|00013893-03JUN09
0000005678|ABCD|00013893-03JUN09
0000009112|0000|00013893-03JUN09
0000009112|0000|00013893-03JUN09
Here's my bulk insert statement:
BULK INSERT sometable
FROM 'E:\filefromabove.txt
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR= '|',
ROWTERMINATOR = '\n'
)
But, for some reason the only output I can get is:
0000005678|ABCD|00013893-03JUN09
0000009112|0000|00013893-03JUN09
0000009112|0000|00013893-03JUN09
The first record always gets skipped, unless I remove the header altogether and don't use the FIRSTROW parameter. How is this possible?
Thanks in advance!
I don't think you can skip rows in a different format with BULK INSERT/BCP.
When I run this:
TRUNCATE TABLE so1029384
BULK INSERT so1029384
FROM 'C:\Data\test\so1029384.txt'
WITH
(
--FIRSTROW = 2,
FIELDTERMINATOR= '|',
ROWTERMINATOR = '\n'
)
SELECT * FROM so1029384
I get:
col1 col2 col3
-------------------------------------------------- -------------------------------------------------- --------------------------------------------------
***A NICE HEADER HERE***
0000001234 SSNV 00013893-03JUN09
0000005678 ABCD 00013893-03JUN09
0000009112 0000 00013893-03JUN09
0000009112 0000 00013893-03JUN09
It looks like it requires the '|' even in the header data, because it reads up to that into the first column - swallowing up a newline into the first column. Obviously if you include a field terminator parameter, it expects that every row MUST have one.
You could strip the row with a pre-processing step. Another possibility is to select only complete rows, then process them (exluding the header). Or use a tool which can handle this, like SSIS.
Maybe check that the header has the same line-ending as the actual data rows (as specified in ROWTERMINATOR)?
Update: from MSDN:
The FIRSTROW attribute is not intended
to skip column headers. Skipping
headers is not supported by the BULK
INSERT statement. When skipping rows,
the SQL Server Database Engine looks
only at the field terminators, and
does not validate the data in the
fields of skipped rows.
I found it easiest to just read the entire line into one column then parse out the data using XML.
IF (OBJECT_ID('tempdb..#data') IS NOT NULL) DROP TABLE #data
CREATE TABLE #data (data VARCHAR(MAX))
BULK INSERT #data FROM 'E:\filefromabove.txt' WITH (FIRSTROW = 2, ROWTERMINATOR = '\n')
IF (OBJECT_ID('tempdb..#dataXml') IS NOT NULL) DROP TABLE #dataXml
CREATE TABLE #dataXml (ID INT NOT NULL IDENTITY(1,1) PRIMARY KEY CLUSTERED, data XML)
INSERT #dataXml (data)
SELECT CAST('<r><d>' + REPLACE(data, '|', '</d><d>') + '</d></r>' AS XML)
FROM #data
SELECT d.data.value('(/r//d)[1]', 'varchar(max)') AS col1,
d.data.value('(/r//d)[2]', 'varchar(max)') AS col2,
d.data.value('(/r//d)[3]', 'varchar(max)') AS col3
FROM #dataXml d
You can use the below snippet
BULK INSERT TextData
FROM 'E:\filefromabove.txt'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = '|', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
ERRORFILE = 'E:\ErrorRows.csv',
TABLOCK
)
To let SQL handle quote escape and everything else do this
BULK INSERT Test_CSV
FROM 'C:\MyCSV.csv'
WITH (
FORMAT='CSV'
--FIRSTROW = 2, --uncomment this if your CSV contains header, so start parsing at line 2
);
In regards to other answers, here is valuable info as well:
I keep seeing this in all answers: ROWTERMINATOR = '\n'
The \n means LF and it is Linux style EOL
In Windows the EOL is made of 2 chars CRLF so you need ROWTERMINATOR = '\r\n'
Given how mangled some data can look after BCP importing into SQL Server from non-SQL data sources, I'd suggest doing all the BCP import into some scratch tables first.
For example
truncate table Address_Import_tbl
BULK INSERT dbo.Address_Import_tbl
FROM 'E:\external\SomeDataSource\Address.csv'
WITH (
FIELDTERMINATOR = '|', ROWTERMINATOR = '\n', MAXERRORS = 10
)
Make sure all the columns in Address_Import_tbl are nvarchar(), to make it as agnostic as possible, and avoid type conversion errors.
Then apply whatever fixes you need to Address_Import_tbl. Like deleting the unwanted header.
Then run a INSERT SELECT query, to copy from Address_Import_tbl to Address_tbl, along with any datatype conversions you need. For example, to cast imported dates to SQL DATETIME.
I want to insert text with single quote
Eg john's to table in sql server 2005 database
Escape single quote with an additional single as Kirtan pointed out
And if you are trying to execute a dynamic sql (which is not a good idea in the first place) via sp_executesql then the below code would work for you
sp_executesql N'INSERT INTO SomeTable (SomeColumn) VALUES (''John''''s'')'
The answer really depends on how you are doing the INSERT.
If you are specifying a SQL literal then you need to use the double-tick approach:
-- Direct insert
INSERT INTO Table1 (Column1) VALUES ('John''s')
-- Using a parameter, with a direct insert
DECLARE #Value varchar(50)
SET #Value = 'John''s'
INSERT INTO Table1 (Column1) VALUES (#Value)
-- Using a parameter, with dynamic SQL
DECLARE #Value varchar(50)
SET #Value = 'John''s'
EXEC sp_executesql 'INSERT INTO Table1 (Column1) VALUES (#p1)', '#p1 varchar(50)', #Value
If you are doing the INSERT from code, use parameters:
// Sample ADO.NET
using (SqlConnection conn = new SqlConnection(connectionString)) {
conn.Open();
using (SqlCommand command = conn.CreateCommand()) {
command.CommandText = "INSERT INTO Table1 (Column1) VALUES (#Value)";
command.Parameters.AddWithValue("#Value", "John's");
command.ExecuteNonQuery();
}
}
If your data contains user-input, direct or indirect, USE PARAMETERS. Parameters protect against SQL Injection attacks. Never ever build up dynamic SQL with user-input.
This worked for me:
INSERT INTO [TABLE]
VALUES ('text','''test.com''', 1)
Basically, you take the single quote you want to insert and replace it with two. So if you want to insert a string of text ('text') and add single quotes around it, it would be ('''text'''). Hope this helps.
INSERT INTO Table1 (Column1) VALUES ('John''s')
Or you can use a stored procedure and pass the parameter as -
usp_Proc1 #Column1 = 'John''s'
If you are using an INSERT query and not a stored procedure, you'll have to escape the quote with two quotes, else its OK if you don't do it.
This answer works in SQL Server 2005, 2008, 2012.
At times the value has MANY single quotes. Rather than add a single quote next to each single quote as described above with 'John''s'. And there are examples using the REPLACE function to handle many single quotes in a value.
Try the following. This is an update statement but you can use it in an INSERT statement as well.
SET QUOTED_IDENTIFIER OFF
DECLARE #s VARCHAR(1000)
SET #s = "SiteId:'1'; Rvc:'6'; Chk:'1832'; TrEmp:'150'; WsId:'81'; TtlDue:'-9.40'; TtlDsc:'0'; TtlSvc:'0'; TtlTax:'-0.88'; TtlPay:'0'; TipAmt:'0.00'; SvcSeq:'09'; ReTx:'N'; TraceId:'160110124347N091832';"
UPDATE TransactionPaymentPrompt
set PromptData = #s
from TransactionPaymentPrompt tpp with (nolock)
where tpp.TransactionID = '106627343'
You asked how to escape an Apostrophe character (') in SQL Server. All the answers above do an excellent job of explaining that.
However, depending on the situation, the Right single quotation mark character (’) might be appropriate.
(No escape characters needed)
-- Direct insert
INSERT INTO Table1 (Column1) VALUES ('John’s')
• Apostrophe (U+0027)
Ascii Apostrophe on Wikipedia
• Right single quotation mark (U+2019)
Unicode Right single quotation on Wikipedia