First of all, can i pass parameters for Path in BCP command?
I wrote this query in Exec SQL task
EXEC xp_cmdshell 'bcp "SELECT * FROM TLC.dbo.World_Hosts" queryout `"C:\Users\akshay.kapila\Desktop\TLC\Foreachlearn\Dest\?.txt" -T -c -t '`
I have given ? in path. I want specific countries in place of that. They are held in variable "Country".I am using Foreach loop which creates rather it should create a file ex Aus.txt,In.txt everytime loop runs for that specific value.
Can i use this way. If not, how can i pass variable value to Path in BCP command?
You can use variable as the SQLSourceType in your Execute SQL Task.
Create a variable to hold your bcp command, it may look like:
"EXEC xp_cmdshell 'bcp \"SELECT ''abc'' as output\" queryout \"" + #[User::strFileName] + "\" -T -c -t '"
Here #[User::strFileName] is the dynamic file you want to generate.
Then in the Execute SQL Task, change SQLSourceType to variable, and select the variable you just generated.
I have been playing around with database backup automation scripts and in particular the one at this link:
http://support.microsoft.com/kb/2019698
I got everything working fine and even added automated compression using 7zip, logging, and with the help of vbscript an email scheduled notification. However, even without all that, you can see this is a bit heavy. Its now easily reaching 400 lines of code.
I am really not comfortable having all my stuff in one block like this and I want to separate it out. So I can have say a compression file called BackupCompress.sql, and an log file called BackupLogReport.sql all of which would be called from inside the main Backup.sql script.
The Backup.sql script is in turn run from a Backup.bat file which is set to run in the scheduler.
All of this works like a charm. But I am at a loss as to how to call BackupCompress.sql from within BackupLogReport.sql and pass in parameters and get a return value.
In the Backup.bat file I use this command to spin everything up and pass parameters to it:
SQLCMD -S %SQLDATABASE% -d master -i %BACKUP_FOLDER%\Backup.sql -v Pram1="%Pram1%"
In the Backup.sql file I get those parameters simply by:
DECLARE #Param1 NVARCHAR(256) = '$(Param)'
from then on as my script runs it uses whatever I want to pass in.
I tried using standard sql stored procedure logic to call another procedure like this:
EXEC BackupCompress.sql
#AnotherParam = #Param1
I also tried:
EXECUTE sp_executesql BackupCompress.sql #Param1
Finally I tried:
SET #cmd = 'SQLCMD -S ' + ##ServerName + ' -d master -i $(BACKUP_FOLDER)\BackupCompress.sql -v Param1 = ' + #Param1
EXEC xp_cmdshell #cmd, no_output
but it doesn't work and my files which were being compressed simply don't get compressed. I get no error message. everything else continues to work fine.
EDIT: I was getting an error message on the last one but I fixed it - however, I still don't get my little zip file. I even put print's into the file to see if it was actually be executed but it does not seem to be.
EDIT2: Another option I have tried, almost works, but cant figure out how to pass parameters from within the sql file to the other file... As a result it generates an error saying it cant find the file as it's treating the path as a literal string instead of the variable value I want to pass.
:!!SQLCMD -S ##ServerName -d master -i #CFG_BACKUP_PATH\BackupCompress.sql -v Param1 = #Param1
xp_cmdshell can return values. These values can be captured into a table variable that you could use to "see" the results, and perhaps determine where the problem lies:
DECLARE #cmd VARCHAR(255);
DECLARE #Param1 NVARCHAR(256) = '$(Param)';
DECLARE #Results TABLE
(
ResultsText NVARCHAR(MAX)
);
SET #cmd = 'SQLCMD -S ' + ##ServerName + '-d master -i $(BACKUP_FOLDER)\$(BackupCompress.sql) -v Param1 = ' + #Param1;
SET #cmd = 'DIR \';
INSERT INTO #Results (ResultsText)
EXEC xp_cmdshell #cmd;
SELECT *
FROM #Results;
You need to ensure xp_cmdshell is enabled for the instance, by executing:
EXEC sp_configure 'xp_cmdshell',1;
Below is an example of the BCP Statement.
I'm not accustomed to using BCP so your help and candor is greatly appreciated
I am using it with a format file as well.
If I execute from CMD prompt it works fine but from SQL I get the error.
The BCP statement is all on one line and the SQL Server Agent is running as Local System.
The SQL server, and script are on the same system.
I ran exec master..xp_fixeddrives
C,45589
E,423686
I've tried output to C and E with the same result
EXEC xp_cmdshell 'bcp "Select FILENAME, POLICYNUMBER, INSURED_DRAWER_100, POLICY_INFORMATION, DOCUMENTTYPE, DOCUMENTDATE, POLICYYEAR FROM data.dbo.max" queryout "E:\Storage\Export\Data\max.idx" -fmax-c.fmt -SSERVERNAME -T
Here is the format file rmax-c.fmt
10.0
7
1 SQLCHAR 0 255 "$#Y#$" 1 FILENAME
2 SQLCHAR 0 40 "" 2 POLICYNUMBER
3 SQLCHAR 0 40 "" 3 INSURED_DRAWER_100
4 SQLCHAR 0 40 "" 4 POLICY_INFORMATION
5 SQLCHAR 0 40 "" 5 DOCUMENTTYPE
6 SQLCHAR 0 40 "" 6 DOCUMENTDATE
7 SQLCHAR 0 8 "\r\n" 7 POLICYYEAR
Due to formating in this post the last column of the format file is cut off but reads SQL_Latin1_General_CP1_CI_AS for each column other that documentdate.
Does the output path exist? BCP does not create the folder before trying to create the file.
Try this before your BCP call:
EXEC xp_cmdshell 'MKDIR "E:\Storage\Export\Data\"'
First, rule out an xp_cmdshell issue by doing a simple 'dir c:*.*';
Check out my blog on using BCP to export files.
I had problems on my system in which I could not find the path to BCP.EXE.
Either change the PATH variable of hard code it.
Example below works with Adventure Works.
-- BCP - Export query, pipe delimited format, trusted security, character format
DECLARE #bcp_cmd4 VARCHAR(1000);
DECLARE #exe_path4 VARCHAR(200) =
' cd C:\Program Files\Microsoft SQL Server\100\Tools\Binn\ & ';
SET #bcp_cmd4 = #exe_path4 +
' BCP.EXE "SELECT FirstName, LastName FROM AdventureWorks2008R2.Sales.vSalesPerson" queryout ' +
' "C:\TEST\PEOPLE.TXT" -T -c -q -t0x7c -r\n';
PRINT #bcp_cmd4;
EXEC master..xp_cmdshell #bcp_cmd4;
GO
Before changing the path to \110\ for SQL Server 2012 and the name of the database to [AdventureWorks2012], I received the following error.
After making the changes, the code works fine from SSMS. The service is running under NT AUTHORITY\Local Service. The SQL Server Agent is disabled. The output file was created.
Please check, the file might be opened in another application or program.
If it is the case, bcp.exe cannot overwrite the existing file contents.
In my case, I solved The problem in the following way:
my command was :
bcp "select Top 1000 * from abc.dbo.abcd" queryout FileNameWithDirectory -c -t "|" -r "0x0a" -S 192.111.1.111 -U xx -P xxxxx
My FileNameWithDirectory was too long. like "D:\project-abc\R&D\abc-608\FilesNeeded\FilesNeeded\DataFiles\abc.csv".
I change into a simpler directory like : "D:\abc.csv"
Problem solved.
So I guess the problem occurred due to file name exceeding. thus the file was not found.
If it works from the command line but not from the SQL Agent, I think it is an authentication issue.
The SQL Server Agent is running under a account. Make sure that the account has the ability to read the format file and generate the output file.
Also, make sure the account has the ability to execute the xp_cmdshell stored procedure.
Write back with your progress ...
I received this after I shared my output folder, even when there were no files open.
I created a new, unshared folder for output and all was fine.
(might help someone ;-))
In my case this fix was simply running in administrator mode.
This error can be due to insufficient write permissions to the target folder.
This is a common issue, since the user writing the query might have access to a folder, but the SQL Server Agent or logged-in server account which actually invokes bcp.exe may not.
Destination path has to already exist (except for file name).
Remove no_output from your command, if you use one offcourse
SET #sql = 'BCP ....'
EXEC master..xp_cmdshell #sql , no_output
EXEC master..xp_cmdshell #sql
In case anyone else runs into the same problem: I had ...lesPerson" queryout' rather than ...lesPerson" queryout '
If your code is writing the data file, and then reading it with BCP, make sure that you CLOSE THE DATA FILE before trying to read it!
Failure to do so gives: 'Unable to open host data-file'.
Python example:
# Management of temporary bulk insert file.
def openBulkInsertFile(self) :
self.bulkInsertFile = open('c:/tmp/bulkInsertContent.txt', 'w', newline='')
self.csvWriter = csv.writer(self.bulkInsertFile)
def closeBulkInsertFile(self) :
self.bulkInsertFile.close()
When using a Job in SQL the user that uses the SQL express server is the current user logged, you should give write permission to that user in the folder where the Batch writes the output.
This happens usually only with bcp, when using type commands the ownership goes to the computer(Administrator) and the command runs with out problem.
So if you have a long command in your job just look for the bcp parts.
for whatever reason, when I get to the step where I generate a txt file with data from a query using the BCP utility, it hangs on the file creation. then if i try to query the database for those tables, it really won't let me.
does anyone know why this would happen? the query is actually very simple:
SET #cmdQueryout = 'bcp "SELECT X FROM Database.dbo.Details WHERE DetailsId = (SELECT MAX(DetailsId) FROM Database.dbo.Details WHERE CommitDateTime IS NOT NULL AND LEFT(PolicyNumber, 3) != ''NYD'') ORDER BY X, Y, Z" queryout "' + #detailFilePath + '" -c -T'
EXEC master..xp_cmdshell #cmdQueryout
I can see it created the first file but there's no data in it and it stops there.
I can open the file but if I try to delete, it won't let me because BCP is using the file.
The query should not take more than a few seconds to run so why would it stop like this?
EDIT - If I run this by itself in another query window, it works.
But if it's in a SQL job and in a transaction, it does not work.
Found out the issue.
I was using a trusted connection when I needed to specify a username & password.
-Uusername -Ppassword
instead of -T.
Im using Sql2008 trying to run this BCP command but it never creates the file.
-- Export query
DECLARE #qry2 VARCHAR(1000)
SET #qry2 = 'SELECT * FROM #SkippedProductsTable'
-- Folder we will be putting the file in
DECLARE #incomingfolder VARCHAR(1000)
SET #incomingfolder = 'c:\Logs'
DECLARE #bcpCommand VARCHAR(2000)
SET #bcpCommand = 'bcp "'+#qry2+'" queryout "'+#incomingfolder+'\SkippedProducts-'+CAST(#StoreMatchCode AS VARCHAR)+'-'+'.txt" -c -T'
PRINT #bcpCommand
EXEC MASTER..xp_cmdshell #bcpCommand, no_output
The created command looks like:
bcp "SELECT * FROM #SkippedProductsTable" queryout "c:\Logs\SkippedProducts-1330-.txt" -c -T
Can anyone suggest what could be going wrong? I've never used BCP before and not really sure where to start looking.
As a start I know that the folder deffinately exists at that location
I think the problem is the SELECT.
You are SELECTing from a table variable that is not declared in the query, so there's nothing for BCP to do.
Table variables only persist for the context they are called in, so even if you have one in a query, and you have dynamic sql or a subproc within that first query, they won't be able to see the table variable.
See this for more info.