I am doing a bulk insert that I have to skip the last row. Otherwise , I got an error saying "Bulk Insert: Unexpected end-of-file (EOF) encountered in data file."
If I set ROWTERMINATOR='\r\n', then I got 0 rows imported.
I wonder if there is any code that can help me skip the lastrow of the txt file? (The last row is dynamic.) My company currently doesn't have SSIS installed.
My code for bulk insert is
Declare #SQL1 varchar(150), #path varchar(100),
#pathtable varchar(100), #date datetime
set #date = getdate()
-- set path for files
set #path= 'C:\imp\'
set #pathtable = #path + 'importfile.txt'
delete from IDX
-- set sql
set #SQL1 = "BULK INSERT dbo.table FROM '" + #pathtable
+ "' WITH (FIRSTROW = 2, MAXERRORS = 0)"
-- Bulk insert
exec(#sql1)
The issue is that the last row contains a row count from the export process. If you're able to modify the export process, make sure you use the SQL command:
SET NOCOUNT ON;
If you're using a GUI to export the data there should be a place to modify the T-SQL used or an option to set nocount on.
This will prevent the last row from writing out to your file.
If you cannot modify the export process... You can get crazy and right either a console application to read the data and remove the last line or a CLR that does basically that very task.. Open the file, remove the last line, save the file then call your stored procedure above to bulk insert your data.
You need to use single quotes ' multiple times, You have used double quotes " which are treated as identifiers in sql server.
Your query should look like this...
Declare #SQL1 varchar(150)
, #path varchar(100)
, #pathtable varchar(100)
, #date datetime
SET #date = getdate();
SET #path= 'C:\imp\'
SET #pathtable = #path + 'importfile.txt'
SET #SQL1 = 'BULK INSERT dbo.table
FROM ''' + #pathtable + '''
WITH (
FIRSTROW = 2
, MAXERRORS = 0
)';
Now if you print this SQL statement it would look like this...
PRINT #SQL1
RESULT:
BULK INSERT dbo.table
FROM 'C:\imp\importfile.txt'
WITH (
FIRSTROW = 2
, MAXERRORS = 0
)
Related
The following code gives an error (its part of a T-SQL stored procedure):
-- Bulk insert data from the .csv file into the staging table.
DECLARE #CSVfile nvarchar(255);
SET #CSVfile = N'T:\x.csv';
BULK INSERT [dbo].[TStagingTable]
-- FROM N'T:\x.csv' -- This line works
FROM #CSVfile -- This line will not work
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n',
FIRSTROW = 2
)
The error is:
Incorrect syntax near the keyword 'with'.
If I replace:
FROM #CSVfile
with:
FROM 'T:\x.csv'
... then it works nicely.
As I know only literal string is required in the from. In that case you have to write a dynamic query to use bulk insert
declare #q nvarchar(MAX);
set #q=
'BULK INSERT [TStagingTable]
FROM '+char(39)+#CSVfile+char(39)+'
WITH
(
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''\n'',
FIRSTROW = 1
)'
exec(#q)
Have you tried with dynamic SQL?
SET #SQL = "BULK INSERT TmpStList FROM '"+#PathFileName+"' WITH (FIELDTERMINATOR = '"",""') "
and then
EXEC(#SQL)
Ref.: http://www.sqlteam.com/article/using-bulk-insert-to-load-a-text-file
you have to engage in string building & then calling EXEC() or sp_executesql BOL
has an example:
DECLARE #bulk_cmd varchar(1000)
SET #bulk_cmd = 'BULK INSERT AdventureWorks2008R2.Sales.SalesOrderDetail
FROM ''<drive>:\<path>\<filename>''
WITH (ROWTERMINATOR = '''+CHAR(10)+''')'
EXEC(#bulk_cmd)
A string literal is required.
http://msdn.microsoft.com/en-us/library/ms188365.aspx
You could use dynamic sql to generate the string literal.
Most of the time the variable i'm looking for in a file name is the date, and this one works perfectly for bulk inserting files with date, for use such as in a daily job. Change as per your need, date format, table name, file path, file name and delimiters.
DECLARE #DT VARCHAR (10)
DECLARE #INSERT VARCHAR (1000)
SET #DT = (CONVERT(VARCHAR(10),GETDATE()-1,120))
SET #INSERT = 'BULK INSERT dbo.table FROM ''C:\FOLDER\FILE'+#DT+'.txt'''+' WITH (FIRSTROW=2, FIELDTERMINATOR=''\t'', ROWTERMINATOR=''\n'')'
EXEC (#INSERT);
Can you try FROM ' + #CSVfile + '
I am trying to write code for my client that they can use in the future without my help. They will install SQLServer Express on one of their desktops (like I have been using) so that the designated person can run the pre-written queries on locally stored data for specific information they need on a daily basis. I don't want to get into why we are using SQL Server Express and managing our data in this manner, but trust that we've looked into alternatives and this is the most feasible option.
I would like to make executing the import query as simple as possible for them, as they are not familiar with SQL or coding in general. At a high level, I have about 15 tab-delimited files automatically loaded into a folder on my local drive every day with the filename consistent except for the pre-fix, which is the date that the file was loaded. For example, 20180912-xxx, 20180912-yyy, etc. for all of the files loaded on Sept 12.
I run a bulk import that creates a Database for Sept 12 ("sep12"), and then loads the tables into the database; then I use the #date variable in the import statements to define the file location. For example:
CREATE DATABASE aug29
USE aug29
DECLARE #date VARCHAR(15)
SET #date = '20180829'
#import = 'BULK INSERT dbo.Table FROM ''\\Drive\Documents\' + #Date + '-xxx.txt''
WITH (FIRSTROW = 2, FIELDTERMINATOR = ''\t'', ROWTERMINATOR = ''\n'')'
EXEC(#import)
As you can see, there is manual edit needed for the 1st, 2nd, and 4th lines. I've simplified the code to only require 1 manual edit, by defining 1 variable as the date (ex. #dateinput = '9/12/2018'), and then the other variables define themselves accordingly (ex. #DBName = 'sep12', #FilePrefix = '20180912', etc.).
My question is this:
Can I prompt the user to enter this date before running the code? Something like
Click Execute > message prompt pops up > User enters a date > click OK > code runs with the inputted value stored as the variable
Create a Stored Procedure?
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author:
-- Create date:
-- Description:
-- =============================================
CREATE PROCEDURE [dbo].[ImportData]
#date varchar(15)
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Insert statements for procedure here
DECLARE #query AS varchar(MAX)
SET #query = 'CREATE DATABASE aug29
USE aug29
BULK INSERT dbo.Table FROM ''\\Drive\Documents\''' + #date + '''-xxx.txt''
WITH (FIRSTROW = 2, FIELDTERMINATOR = ''\t'', ROWTERMINATOR = ''\n'')'
EXEC(#query)
END
Then on SSMS open your database > Programmability > Stored Procedures > Right Click on the newly created Stored Procedure (ImportData or whichever you name it) > Execute Stored Procedure.
Users can then input the value for the #date parameter.
You could make your script completely independent of user input by using some of SQL Server's built in functions like I am doing below:
--Variable to use for dynamic sql
DECLARE #sqlStatement varchar(MAX) = '';
--Returns the month as its full name like June, or July
DECLARE #fullMonthValue varchar(100) = DATENAME(month, DATEADD(month, MONTH(GETDATE()) -1, CAST('2008-01-01' AS datetime)));
--Get the database name how you gave in your example, for example today is August 30th, this will result in aug30
DECLARE #databaseName varchar(100) = LOWER(SUBSTRING(#fullMonthName, 1, 3)) + CAST(DAY(GETDATE()) AS varchar(3));
--Now get the current date as string for use in your bulk insert
DECLARE #today = CAST(GETDATE() AS Date);
--cast the current date to varchar (string) and remove the hyphens
DECLARE #stringDate = REPLACE(CAST(#today AS varchar(100)), '-', ''); --Need to remove the hyphens
--Set the sql statement for creating the database
SET #sqlStatment = 'Create DataBase ' + #databaseName;
--Execute the sqlStatement to create the database
EXEC(#sqlStatement);
--At this point #stringDate is already the format you want for your example variable of #date
--Just put your USE statement into your dynamic sql string
#import = 'USE ' + #databaseName + 'BULK INSERT dbo.Table FROM ''\\Drive\Documents\'' + #stringDate + '-xxx.txt''
WITH (FIRSTROW = 2, FIELDTERMINATOR = ''\t'', ROWTERMINATOR = ''\n'')';
EXEC(#import);
I'm using a bulk insert script to import a number of flat files into SQL Server
A few files end with
-----------------------------------
So what I want to do is or skip last row(s) or remove ------------------ in the bulk insert. Is one of these options possible?
SET #s = N'BULK INSERT ' + #t + '
FROM ''' + #f + '''
WITH (FIELDTERMINATOR = ''|'',
ROWTERMINATOR = ''0x0a'',
FIRSTROW=2) '
lastrow = 1 doesn't work
The only way I can think of is to first bulk insert the whole file into a single column table (as varchar(max)). Than you can identify the last row, and use that value in your actual bulk insert.
This is not a very straight forward approach, but I don't think there is another (unless you write a custom solution in C# or java or whatever). Maybe you can use SQLCMD to first read the number of lines in the file, but I don't know how.
Please note there is a connect item which Microsoft has closed. On that page Microsoft suggests using an openrowset solution, could be worthwhile to try, but I doubt it would work in your situation.
use script like next:
SET NOCOUNT ON;
IF OBJECT_ID('tempdb..#csvData') IS NOT NULL DROP TABLE #csvData
IF OBJECT_ID('tempdb..#csvRowCount') IS NOT NULL DROP TABLE #csvRowCount
CREATE TABLE #csvRowCount
(
v1 nvarchar(max) -- get only first column - for all rows in file
)
BULK INSERT #csvRowCount
FROM 'C:\TEMP\HR_FOR_LOAD\PP_SICKLIST.CSV'
WITH
(
Firstrow=2,
FIELDTERMINATOR = '\t',
ROWTERMINATOR = '\n'
);
declare #textRowCount int=(select count(1) from #csvRowCount)
CREATE TABLE #csvData
(
v1 nvarchar(2000),
v2 nvarchar(2000),
v3 nvarchar(2000)
--etc
)
declare #sql varchar(max)
set #sql = '
BULK INSERT #csvData
FROM ''C:\TEMP\HR_FOR_LOAD\PP_SICKLIST.CSV''
WITH
(
Firstrow=2,
FIELDTERMINATOR = ''\t'',
ROWTERMINATOR = ''\n'',
Lastrow = '+cast(#textRowCount as varchar(100))+'
);'
exec (#sql)
select
v1,
v2,
v3
from #csvData
I need to bulk insert a text file, which is always stored in the same folder. The file name is 'employee_date', where the date part is not always the actual date. It is a dynamic date that the user changes every day (with the format YYYYMMDDHHMM).
What I need is a query that bulk inserts the data in that text file (that is already formatted). My problem is bulk insert does not work with variables or with *.txt or employes*.txt.
I need a query that bulk insert only the file which name is like EMPLOYEE_YYYYMMDDHHMM.txt that can be executed every day and always insert the file from that folder, doesn't matter what the date is in the file name.
Here is something you can modify to fit your requirements. I had a similar task where we got files SFTP to us from a Linux system and I needed to upload this data into a SQL Server. Here's the basic layout... understanding your file locations, permissions on the folder, SQL permissions, etc all need to be taken into consideration. Including if you can run command shell code in your environment.
CREATE procedure [dbo].[file_upload]
as
DECLARE #dt VARCHAR(10) --date variable but stored as VARCHAR for formatting of file name
DECLARE #fileLocation VARCHAR(128) = 'E:\SomeFolder\' --production location which is
DECLARE #sql NVARCHAR(4000) --dynamic sql variable
DECLARE #fileName VARCHAR(128) --full file name variable
--This stores the file names into a temp table to be used in a cursor.
--The bottom part is handleing some date formatting i needed. You can change to what your files look like
IF OBJECT_ID('tempdb..#FileNames') IS NOT NULL DROP TABLE #FileNames
CREATE TABLE #FileNames (
id int IDENTITY(1,1)
,subdirectory nvarchar(512)
,depth int
,isfile bit
,fileDate date null
,fileTime time null)
INSERT #FileNames (subdirectory,depth,isfile)
EXEC xp_dirtree #fileLocation, 1, 1
UPDATE #FileNames SET
fileDate = CAST(SUBSTRING(subdirectory,LEN(subdirectory) - 19,10) AS DATE)
,fileTime = CAST(REPLACE(SUBSTRING(subdirectory,LEN(subdirectory) - 8,5),'-',':') AS TIME)
--here's the cursor to loop through all the files
DECLARE c CURSOR FOR
select subdirectory from #FileNames
OPEN c
FETCH NEXT FROM c INTO #fileName
--For each file, bulk insert or what ever you want...
WHILE ##FETCH_STATUS = 0
BEGIN
--set the dynamic with the appropriate delimiters, if you want to keep headers, etc.
SET #sql = 'BULK INSERT Server.dbo.someTable FROM '''+ #fileLocation + #fileName +''' WITH (FIELDTERMINATOR = ''|'',FIRSTROW=2,KEEPNULLS,ROWTERMINATOR = ''0x0a'')'
EXEC(#sql)
--do some other stuff like logging, updating, etc...
END
CLOSE c
DEALLOCATE c
This should do it for you. Just adjust as needed.
DECLARE #intFlag INT
SET #intFlag = 1
WHILE (#intFlag <=48)
BEGIN
PRINT #intFlag
declare #fullpath1 varchar(1000)
select #fullpath1 = '''your_path_here\employee_' + convert(varchar, getdate()- #intFlag , 112) + '.txt'''
declare #cmd1 nvarchar(1000)
select #cmd1 = 'bulk insert [dbo].[your_table_name] from ' + #fullpath1 + ' with (FIELDTERMINATOR = ''\t'', FIRSTROW = 2, ROWTERMINATOR=''0x0a'')'
exec (#cmd1)
SET #intFlag = #intFlag + 1
END
GO
The following code gives an error (its part of a T-SQL stored procedure):
-- Bulk insert data from the .csv file into the staging table.
DECLARE #CSVfile nvarchar(255);
SET #CSVfile = N'T:\x.csv';
BULK INSERT [dbo].[TStagingTable]
-- FROM N'T:\x.csv' -- This line works
FROM #CSVfile -- This line will not work
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n',
FIRSTROW = 2
)
The error is:
Incorrect syntax near the keyword 'with'.
If I replace:
FROM #CSVfile
with:
FROM 'T:\x.csv'
... then it works nicely.
As I know only literal string is required in the from. In that case you have to write a dynamic query to use bulk insert
declare #q nvarchar(MAX);
set #q=
'BULK INSERT [TStagingTable]
FROM '+char(39)+#CSVfile+char(39)+'
WITH
(
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''\n'',
FIRSTROW = 1
)'
exec(#q)
Have you tried with dynamic SQL?
SET #SQL = "BULK INSERT TmpStList FROM '"+#PathFileName+"' WITH (FIELDTERMINATOR = '"",""') "
and then
EXEC(#SQL)
Ref.: http://www.sqlteam.com/article/using-bulk-insert-to-load-a-text-file
you have to engage in string building & then calling EXEC() or sp_executesql BOL
has an example:
DECLARE #bulk_cmd varchar(1000)
SET #bulk_cmd = 'BULK INSERT AdventureWorks2008R2.Sales.SalesOrderDetail
FROM ''<drive>:\<path>\<filename>''
WITH (ROWTERMINATOR = '''+CHAR(10)+''')'
EXEC(#bulk_cmd)
A string literal is required.
http://msdn.microsoft.com/en-us/library/ms188365.aspx
You could use dynamic sql to generate the string literal.
Most of the time the variable i'm looking for in a file name is the date, and this one works perfectly for bulk inserting files with date, for use such as in a daily job. Change as per your need, date format, table name, file path, file name and delimiters.
DECLARE #DT VARCHAR (10)
DECLARE #INSERT VARCHAR (1000)
SET #DT = (CONVERT(VARCHAR(10),GETDATE()-1,120))
SET #INSERT = 'BULK INSERT dbo.table FROM ''C:\FOLDER\FILE'+#DT+'.txt'''+' WITH (FIRSTROW=2, FIELDTERMINATOR=''\t'', ROWTERMINATOR=''\n'')'
EXEC (#INSERT);
Can you try FROM ' + #CSVfile + '