i am trying to bulk insert into Db using sql server 2005
Below is the code.
declare #path varchar(500)
set #path = 'E:\Support\test.csv';
Create table #mytable( name varchar(max), class varchar(max), roll varchar(max) )
BULK INSERT #mytable FROM #path <-- Error line
WITH ( FIELDTERMINATOR = ',', ROWTERMINATOR = '\n' );
Go
select * from #mytable
drop table #mytable
Problem: issue is that my file path is dynamic and comes from a variable instead of hard coding which is not working
If i change the error line to below it works
BULK INSERT #mytable FROM 'E:\Support\test.csv';
Please advise how to fix this
Try to use Dynamic SQL:
declare #sql varchar(max)
set #sql = 'BULK INSERT #mytable FROM ''' + #path + ''' WITH ...
exec (#sql)
DECLARE #path varchar(50) = 'D:\ARQUIVOS_CARGAS\CABOS\FILE.prn'
DECLARE #SQL_BULK VARCHAR(MAX)
SET #SQL_BULK = 'BULK INSERT #TAB FROM ''' + #path + ''' WITH
(
CODEPAGE = ''ACP'',
FIRSTROW = 1,
FIELDTERMINATOR = ''tab'',
ROWTERMINATOR = ''0x0a'',
KEEPNULLS
)'
EXEC (#SQL_BULK)
Related
I am trying to do a bulk insert but the #CSVPath is not resolving.
declare #path varchar(255)
set #path = 'C\CSVPath.csv';
BULK INSERT #mytable FROM #CSVPath <-- Error line
WITH ( FIELDTERMINATOR = ',', ROWTERMINATOR = '\n' );
I've tried
FROM ''' + #CSVpath + '''
If I hard code the path it works. If I wrap it all in a SET statement and execute it works.
declare #sql varchar(max)
set #sql = 'BULK INSERT #mytable FROM ''' + #CSVPath + ''' WITH ...
exec (#sql)
However, I cannot do it this way and need to it the first method but it doesn't seem to be resolving and cannot figure how to get it to work.
properly When running query dynamics, the compiler stores the values of the variable, but does not normally name the table or address through the variable.
BULK INSERT #mytable
FROM 'C:\test.csv'
WITH
(
FIRSTROW = 2, -- as 1st one is header
FIELDTERMINATOR = ',', -- field delimiter
ROWTERMINATOR = '\n', -- next row
TABLOCK
)
I create a SQL procedure to import data from txt file. However there are lots of files(about 80 files) and i cannot list its name. Its name formatted like 'DATA_XXXXXXX.TXT'
DECLARE #sql1 nvarchar(max) = N'BULK INSERT dbo.Student FROM '''
+ 'D:\NEW_FOLDER\DATA_20190222'
+ '.TXT'
+ ''' WITH
(
FIELDTERMINATOR = ''|'',
MAXERRORS = 10000
);';
EXEC sys.sp_executesql #sql1;
I want all data on theses file should be loaded into table. How can I do that?
You could do it with a while loop, insert the values into a temptable and increment the file name variable with dynamic SQL:
CREATE TABLE #TEMP_FILENAMES
(
FILENAME VARCHAR(50)
)
INSERT INTO #TEMP_FILENAMES
VALUES('20190222')
INSERT INTO #TEMP_FILENAMES
VALUES('20190223')
DECLARE INT #YEARMMDD
WHILE EXISTS(SELECT * FROM #TEMP_FILENAMES)
BEGIN
SET #YEARMMDD = (SELECT TOP 1 FILENAME FROM #TEMP_FILENAMES)
DECLARE #sql1 nvarchar(max) = N'BULK INSERT dbo.Student FROM '''
+ 'D:\NEW_FOLDER\DATA_' + #YEARMMDD
+ '.TXT'
+ ''' WITH
(
FIELDTERMINATOR = ''|'',
MAXERRORS = 10000
);';
EXEC sys.sp_executesql #sql1;
DELETE FROM #TEMP_FILENAMES WHERE FILENAME = #YEARMMDD
END
I cobbled together the code below. I can write file paths into a table but I can't use the paths in the table to bulk load CSV files in a folder. Can some expert here take a look and let me know what's wrong? TIA.
IF OBJECT_ID('tempdb..#DirectoryTree') IS NOT NULL
DROP TABLE #DirectoryTree;
CREATE TABLE #DirectoryTree (
id int IDENTITY(1,1)
,subdirectory nvarchar(512)
,depth int
,isfile bit);
INSERT #DirectoryTree (subdirectory,depth,isfile)
EXEC master.sys.xp_dirtree 'C:\my_path\CSV Files\',1,1;
SELECT * FROM #DirectoryTree
WHERE isfile = 1 AND RIGHT(subdirectory,4) = '.csv'
ORDER BY id;
GO
DROP TABLE ALLFILENAMES
--CREATE TABLE ALLFILENAMES(id VARCHAR(999),subdirectory VARCHAR(255),depth VARCHAR(1),isfile VARCHAR(1))
Select * INTO ALLFILENAMES
From #DirectoryTree
--code above is fine; problems start here
--cursor loop
--bulk insert won't take a variable name, so make a sql and execute it instead:
Declare #sql varchar(8000)
set #sql = 'BULK INSERT BULKACT FROM ''' + 'ALLFILENAMES.subdirectory' + ''' '
+ ' WITH (
DATAFILETYPE = ''char'',
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''\n'',
FIRSTROW = 2
) '
print #sql
exec (#sql)
The problem is with the Bulk Insert. Here is the error message that I get: Msg 4860, Level 16, State 1, Line 28
Cannot bulk load. The file "ALLFILENAMES.subdirectory" does not exist.
So, 'ALLFILENAMES' is the name of the table and 'subdirectory' is the name of the field that contains all paths to all CSV files.
you need to do a select from that ALLFILENAMES table. You can't just specified the tablename + column name like that and expect it to work
also you need to specify the full path in the FROM file name
and you may use temp table for ALLFILENAMES instead of permanent tble
Declare #sql varchar(max)
select #sql = isnull(#sql , '')
+ 'BULK INSERT BULKACT FROM ''C:\my_path\CSV Files\' + ALLFILENAMES.subdirectory + ''' '
+ ' WITH (
DATAFILETYPE = ''char'',
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''\n'',
FIRSTROW = 2
); ' + char(13)
from ALLFILENAMES
print #sql
and there is a WITH (FORMAT = 'CSV'); option for bulk insert from CSV file
Thanks for the help, Sqiurrel. I got the code below to work, and add some comments. It's really ugly though. I guess SQL Server really isn't designed for these kinds of things...
----------------------------------------------------------
-- Create 5 tables and all fields in tables
DECLARE #intFlag INT
SET #intFlag = 1
WHILE (#intFlag <=5)
BEGIN
declare #cmd nvarchar(1000),
#MyTableName nvarchar(100)
print str(#intFlag)
set #MyTableName = 'CSV' + replace(str(#intFlag),' ','')
print #MyTableName
set #cmd = 'CREATE TABLE dbo.' + quotename(#MyTableName, '[') + '(Name varchar(255), Address varchar(255), Age varchar(255), Work varchar(255));';
print #cmd
exec(#cmd)
SET #intFlag = #intFlag + 1
END
GO
----------------------------------------------------------
IF OBJECT_ID('tempdb..#DirectoryTree') IS NOT NULL
DROP TABLE #DirectoryTree;
CREATE TABLE #DirectoryTree (
id int IDENTITY(1,1)
,subdirectory nvarchar(512)
,depth int
,isfile bit);
INSERT #DirectoryTree (subdirectory,depth,isfile)
EXEC master.sys.xp_dirtree 'C:\your_path_here\',1,1;
SELECT * FROM #DirectoryTree
WHERE isfile = 1 AND RIGHT(subdirectory,4) = '.csv'
ORDER BY id;
GO
DROP TABLE ALLFILENAMES
Select * INTO ALLFILENAMES
From #DirectoryTree
----------------------------------------------------------
--cursor loop
--bulk insert won't take a variable name, so make a sql and execute it instead:
Declare #sql varchar(max)
select #sql = isnull(#sql , '')
+ 'BULK INSERT ' + ALLFILENAMES.subdirectory + ' FROM ''C:\your_path_here\' + ALLFILENAMES.subdirectory + ''' '
+ ' WITH (
DATAFILETYPE = ''char'',
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''\n'',
FIRSTROW = 2
); ' + char(13)
from ALLFILENAMES
print #sql
exec (#sql)
----------------------------------------
I'm trying run BULK INSERT in a loop. Loop through each file in some directory ends with no of particular file. Below is my solution
DECLARE #startFlag INT
DECLARE #endFlag INT
DECLARE #fileName varchar(50)
SET #startFlag = 1
SET #endFlag = 10
WHILE (#startFlag <= #endFlag)
BEGIN
SET #fileName = 'c:\path to file\filename_' + cast(#startFlag as varchar) + '.csv'
BULK
INSERT dbo.Intraday
FROM #fileName
WITH
(
FIELDTERMINATOR = '|',
ROWTERMINATOR = '\n'
)
SET #startFlag = #startFlag + 1
END
GO
but seems don't work. Is there anything I've overlooked or another missing stuff I can fix this issue?
You can't use variables or expressions all the places you might like in TSQL. You'll have to use dynamic SQL:
declare #fileName nvarchar(2000) = 'foo.csv'
SET #fileName = 'foo'
declare #sql nvarchar(max) = N'
BULK
INSERT dbo.Intraday
FROM '''+#fileName+'''
WITH
(
FIELDTERMINATOR = ''|'',
ROWTERMINATOR = ''\n''
)';
exec (#sql);
you can not use veritable name after From. you have to provide the name of file after from clause not variable. so you need to make complete bulk insert statement dynamically. please refer below sample code -
declare #sql nvarchar(max)
DECLARE #fileName varchar(50)
set #fileName ='C:\Input.txt'
set #sql = 'BULK
INSERT dbo.Intraday
FROM ''' + #fileName + '''
WITH
(
FIELDTERMINATOR = ''|'',
ROWTERMINATOR = ''\n''
)'
exec(#sql)
I'm using dynamic SQL for bulk insert with a parameter (Bulk insert using stored procedure).
DECLARE #sql NVARCHAR(4000) = 'BULK INSERT TblValues FROM ''' + #FileName + ''' WITH ( FIELDTERMINATOR ='','', ROWTERMINATOR =''\n'' )';
EXEC(#sql);
But... How to avoid SQL injection?
You could use QUOTENAME to surround the file name in single quotes:
DECLARE #sql NVARCHAR(4000) = 'BULK INSERT TblValues FROM ' + QUOTENAME(#FileName,'''') + ' WITH ( FIELDTERMINATOR ='','', ROWTERMINATOR =''\n'' )';
EXEC (#sql);
One way would be to retrieve the file name versus pass it in... something like
DECLARE #fileLocation VARCHAR(128) = '\\some\folder\location'
IF OBJECT_ID('tempdb..#FileNames') IS NOT NULL DROP TABLE #FileNames
CREATE TABLE #FileNames(
id int IDENTITY(1,1)
,subdirectory nvarchar(512)
,depth int
,isfile bit)
INSERT #FileNames(subdirectory,depth,isfile)
EXEC xp_dirtree #fileLocation, 1, 1
Then, in #FileNames will be all the files in that directory (where isfile = 1 of course). Then you can simply query the file name(s) from the temp table.