Heyy,
I'm trying to use BCP to export a SP result to a text file using this query:
EXEC xp_cmdshell 'bcp "exec asmary..usp_Contract_SelectByEmpId -1,1" queryout "C:\test.txt" -w -C OEM -t$ -T -r ~ -S heba\HEBADREAMNET '
The output of this query is telling this error:
Error = [Microsoft][SQL Server Native Client 10.0][SQL Server]Incorrect syntax near the keyword 'where'.
even thought I'm sure that the stored procedure "usp_Contract_SelectByEmpId" is working correctly.
Anyone faced that kind of error before?
As Lynn suggested, check your stored procedure. It looks like the issue is within that.
Ensure any plain SELECT works (e.g., C: drive is database server's local drive, not necessarily your own local drive).
If the first two items work fine, then add SET FMTONLY OFF as follows:
EXEC xp_cmdshell 'bcp "set fmtonly off exec asmary..usp_Contract_SelectByEmpId -1,1" queryout "C:\test.txt" -w -C OEM -t$ -T -r ~ -S heba\HEBADREAMNET'
I have to admit that when I tried similar on my computer it failed with 'Function sequence error', and I found that it is related to a SQL Server 2008 bug fixed in 2011.
Please note also that even without SET FMTONLY OFF everything works with BCP library (odbcbcp.dll/odbcbcp.lib). So, you can have much more generic ODBC-wide bcp solution if you write your own wrapper executable (for instance, in C or C++).
I also found the following at http://msdn.microsoft.com/en-us/library/ms162802.aspx
The query can reference a stored procedure as long as all tables referenced inside the stored procedure exist prior to executing the bcp statement. For example, if the stored procedure generates a temp table, the bcp statement fails because the temp table is available only at run time and not at statement execution time. In this case, consider inserting the results of the stored procedure into a table and then use bcp to copy the data from the table into a data file.
Please see also my later separate reply - I think the whole concept of using stored procedure for BCP/queryout is wrong.
Try this.
DECLARE #strbcpcmd NVARCHAR(max)
SET #strbcpcmd = 'bcp "EXEC asmary..usp_Contract_SelectByEmpId -1,1" queryout "C:\test.txt" -w -C OEM -t"$" -T -S'+##servername
EXEC master..xp_cmdshell #strbcpcmd
Sorry for flooding your question with multiple answers, but I wanted to find out how much heavier (performance-wise) the use of stored procedure is compared to plain SELECT. And I got a very important information from
http://social.msdn.microsoft.com/Forums/en-US/transactsql/thread/b8340289-7d7e-4a8d-b570-bec7a0d73ead/
This forced me to create another (separate) answer. The post I refer to invalidates the whole concept.
In a few words: stored procedure might be called several (3) times in order to figure out the structure of the resultset, then the actual data.
Therefore (and especially if calling from SQL Server connection rather than client), I think it makes a lot more sense to have a stored procedure or function, which will return SELECT statement. Then you can have another generic stored procedure or function to create and execute full BCP command with that statement embedded. I am pretty sure that in this case BCP might use a lot better execution plan. Unfortunately, I cannot verify that in practice, because of BCP bug in SQL Server 2008 R2 I mentioned in my previous post.
N.B. Please be careful creating dynamic queries and escape all explicit literal strings (i.e. repeat all single quotes twice) in order to avoid notorious SQL injection. Unfortunately, there is another pitfall: you should ensure you are not escaping your queries twice or more times.
Related
Background Information:
In Python, I might write something like this if I want to apply the same logic to different values in a list.
database_list = ["db_1", "db_2", "db_3"]
for x in range(0,len(database_list),1):
print("the database name is " + database_list[x])
What I am trying to do:
What I am trying to do in SSMS, is pull a list of DB objects for each database. I created a stored procedure to pull exactly what I want, but I have to run it against each database, so 10 databases mean running it 10 times.
My goal is to do this with a T-SQL query instead of Python.
I tried doing something like this:
exec sp_MSforeachdb 'USE ?; EXEC [dbo].[my_stored_procedure]';
The problem with this is, [dbo].[my_stored_procedure] has to exist in every database I want to do this in.
How can I create the stored procedure in 1 database, but execute it for all databases or a list of databases that I choose?
I know what you are trying to do and if it's what I think (you seem reluctant to actually say!) you can do the following:
In the master database, create your procedure. Normally you wouldn't do this, but in this case you must prefix it sp_
use master
go
create procedure sp_testproc as
select top 10 * from sys.tables
go
Now if you run this, it will return tables from the master database.
If you switch context to another database and exec master.dbo.sp_testproc, it will still return tables from the master database.
In master, run
sys.sp_MS_marksystemobject sp_testproc
Now switch context to a different database and exec master.dbo.sp_testproc
It will return tables from the database you are using.
Try creating your sproc in master and naming it with an sp_ prefix:
USE master
GO
CREATE PROCEDURE sp_sproc_name
AS
BEGIN
...
END
GO
-- You *may* need to mark it as a system object
EXEC sys.sp_MS_marksystemobject sp_sprocname
See: https://nickstips.wordpress.com/2010/10/18/sql-making-a-stored-procedure-available-to-all-databases/
It should then be available in all dbs
Create the stored procedure in the Master database with the sp_ prefix, and use dynamic SQL in the stored procedure so it resolves object names relative to the current database, rather than the database which contains the stored procedure.
EG
use master
go
CREATE OR ALTER PROCEDURE [dbo].[sp_getobjects]
AS
exec ('
select *
from [sys].[objects]
where is_ms_shipped = 0
order by type, name
')
go
use AdventureWorks2017
exec sp_getobjects
#LunchBox - it's your single stored procedure (that you create in one database) that is actually going to need to contain the "exec sp_MSforeach ...." command, and instead of the command to be executed being "EXEC ", it will need to be the actual SQL that you were going to put into the stored proc.
Eg. (inside your single stored procedure)
EXEC sp_MSforeachdb 'USE ?; SELECT * FROM <table>; UPDATE <another table> SET ...';
Think of the stored procedure (that you put into one database) as being no different than your Python code file - if you had actually wanted to achieve the same thing in Python, you would have either needed to create the stored proc in each database, or build the SQL statement string in Python and execute it against each database.
I understand what you thought you might be able to achieve with SQL, but stored procedures really don't work the way you were expecting. Even when you're in the context of a different database, but you run EXEC <different_db>.stored_proc, that stored proc ends up running in the context of the database in which it exists (not your context database).
Now, the only one issue you may come up against is that the standard sp_MSforeachdb stored proc has a limit of 2000 characters for the command that can be executed (although, it does have multiple "command" parameters, this may not be practical if you were planning on running a very large code block, perhaps with variables that carry all the way through). If this is something that might impact what you're intending to do, you could do a search online for "sp_MSforeachdb alternatives" - there seem to be a handful that people have created where the command parameter can contain a larger string.
I've been working on exporting a table to a file, and had problems with the bcp (bulk copy program) part of the procedure locking up. The code worked fine when I ran it as a script, but would generate locked processes when I wrapped it in a stored procedure.
I seem to have found the solution; COMMIT. Namely, I had to wrap the code which truncated and inserted into the table which bcp would be picking up the data from within a BEGIN TRANSACTION...COMMIT. Now the procedure works
I think it is to do with the command
exec master.dbo.xp_cmdshell #bcp
going outside of the SQL session to the OS. Am I correct, or is there a better explanation?
I know this is redundant, but I'd like to Call Query from another Query. I know I can just add it to first one, but the scripts are getting long and at times I don't want to run all of the queries.
I've been looking and my best guess is maybe just using command shell. I was just wondering if there was another way.
Declare #CommandDos VarChar(150) = 'sqlcmd -E -S Server-i h:\SQL\SomeThing.sql'
EXEC master..xp_cmdshell #CommandDos
Code re-use.
Perhaps use functions, i.e. put the query you want called into a function.
Functions can be Scalar, Table-valued, Deterministic, or Nondeterministic.
Maybe you can create stored procedures with the queries, then call them inside another one if needed.
What do you think about it?
I have a .sql script with a lot of action queries that work on some staging tables. This script needs to be run twice with some other commands in-between i.e.:
Load the staging table from source A
Use do_stuff.sql to process it
Move the results somewhere.
Repeat Steps 1-3 for source B.
The brute force approach would be to just copy & paste dostuff.sql as needed. While this would technically work, is there a better way?
I'm hoping there's a command like RunThisSQL 'C:\do_stuff.sql' that I haven't discovered yet.
Update
Well, it's been about 5 years and I just re-discovered this old question. I did this recently and made a cursor to loop thru a master table. For each record in that master table, the script runs through an inner script using variables set by the master table.
https://www.mssqltips.com/sqlservertip/1599/sql-server-cursor-example/
If you use visual studio you can create "Sql Server Database" project. Withing the project you can create script that let you execute your *.sql files in a manner
/*
Post-Deployment Script Template
--------------------------------------------------------------------------------------
This file contains SQL statements that will be appended to the build script.
Use SQLCMD syntax to include a file in the post-deployment script.
Example: :r .\myfile.sql
Use SQLCMD syntax to reference a variable in the post-deployment script.
Example: :setvar TableName MyTable
SELECT * FROM [$(TableName)]
--------------------------------------------------------------------------------------
*/
see also. http://candordeveloper.com/2013/01/08/creating-a-sql-server-database-project-in-visual-studio-2012/
Try using xp_cmdshell.
EXEC xp_cmdshell 'sqlcmd -S ' + #ServerName + ' -d ' + #DBName + ' -i ' +#FileName
xp_cmdshell and concatenation do not play together nicely, often resulting in an "Incorrect syntax near '+'" error. So further to Jeotics solution above you will need to make a variable of the entire string you pass to xp_cmdshell (including quotes around anything that may contain a space (eg filepath\filename). This is mentioned in the Microsoft documentation for xp_cmdshell here. Other issues you will have to contend with are the default set up for SQL Server which has xp_cmdshell disabled as outlined here and granting permission to non-system administrators to use xp_cmdshell outlined here. The documentation generally advises against giving xp_cmdshell rights to too many people owing to it being a vehicle for those with malintent but if, like me, you have minimal and trustworthy database users then it seems like a reasonable solution. One last issue that requires correct configuration is the SQL Server Agent as outlined here. Documentation outlines that SQL Agent is responsible for background scheduling (such as back ups) and performance of command line statements, etc..
DECLARE
#Server nvarchar (50)
,#Database nvarchar(50)
,#File nvarchar(100)
,#cmd nvarchar(300);
SET #Server = server_name;
SET #Database = database_name;
SET #File = 'C:\your file path with spaces';
SET #cmd = 'sqlcmd -S ' + #Server + ' -d ' + #Database + ' i "' + #File + '"';
EXEC xp_cmdshell #cmd;
There are some security issues with enabling xp_cmdshell in SQL Server. You can create a CLR Stored procedure, which executes the passed file content. This CLR stored procedure is especially for this purpose, not like xp_cmdshell, which can do anything over the command prompt.
issues with enabling xp_cmdshell
Create CLR stored procedure
We need to write a trigger so that when rows inserted into a sql server table meeting certain conditions, they are then written to a windows OS flat file in CSV format.
Are there any commands short of running a xp_cmdshell that would allow us to do this?
Any reason you wouldn't instead do a scheduled process with SSIS?
Depending on your transaction rate, I'd be hesistant to put this in a trigger since it means evaluations and possible export on every record entered. If you get a high frequency, you could shoot yourself in the foot...
Even if the transaction rate is fairly low, you could potentially still have problems like blocking/locking if the physical writes take long. You introduce several possible classes of errors (file i/o errors, write-fail means trigger fails means insert fails, etc.).
With a scheduled process you only take a hit on an infrequent basis and you don't potentially lock your table while the trigger is doing something external.
Use Bulk Copy Program (BCP) to create CSV files. For example:
BCP master..sysobjects out c:\sysobjects.txt -c -t, -T –S<servername>
The basic format for the BCP command for creating a CSV file is as follows:
BCP out
The switches used here are:
-c Output in ASCII with the default field terminator (tab) and row terminator (crlf)"
-t override the field terminator with ","
-T use a trusted connection. -U & –P may be used for username/password
-S connect to this server to execute the command
Here's another example:
declare #sql varchar(8000)
select #sql = 'bcp master..sysobjects out
c:\bcp\sysobjects.txt -c -t, -T -S'+ ##servername
exec master..xp_cmdshell #sql
Here's a description of how to use BCP: http://msdn.microsoft.com/en-us/library/ms162802.aspx