We need to write a trigger so that when rows inserted into a sql server table meeting certain conditions, they are then written to a windows OS flat file in CSV format.
Are there any commands short of running a xp_cmdshell that would allow us to do this?
Any reason you wouldn't instead do a scheduled process with SSIS?
Depending on your transaction rate, I'd be hesistant to put this in a trigger since it means evaluations and possible export on every record entered. If you get a high frequency, you could shoot yourself in the foot...
Even if the transaction rate is fairly low, you could potentially still have problems like blocking/locking if the physical writes take long. You introduce several possible classes of errors (file i/o errors, write-fail means trigger fails means insert fails, etc.).
With a scheduled process you only take a hit on an infrequent basis and you don't potentially lock your table while the trigger is doing something external.
Use Bulk Copy Program (BCP) to create CSV files. For example:
BCP master..sysobjects out c:\sysobjects.txt -c -t, -T –S<servername>
The basic format for the BCP command for creating a CSV file is as follows:
BCP out
The switches used here are:
-c Output in ASCII with the default field terminator (tab) and row terminator (crlf)"
-t override the field terminator with ","
-T use a trusted connection. -U & –P may be used for username/password
-S connect to this server to execute the command
Here's another example:
declare #sql varchar(8000)
select #sql = 'bcp master..sysobjects out
c:\bcp\sysobjects.txt -c -t, -T -S'+ ##servername
exec master..xp_cmdshell #sql
Here's a description of how to use BCP: http://msdn.microsoft.com/en-us/library/ms162802.aspx
Related
I'm creating a stored procedure that will delete all the data in my database and then insert the data from my sql file. The reason I am using the delete and insert instead of a restore is because a restore requires that no one is connected to the database where as deleting and inserting allows people to still be connected.
Stored Procedure:
CREATE PROCEDURE DropAndRestore
-- Add the parameters for the stored procedure here
#filepath nvarchar(200)
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Insert statements for procedure here
Exec sp_MSFOREACHTABLE 'delete from ?
RESTORE DATABASE [landofbeds] -- These lines are what needs to be replaced
FROM DISK = #FilePath --
END
GO
The reason I am using the delete and insert instead of a restore is
because a restore requires that no one is connected to the database
where as deleting and inserting allows people to still be connected
If all you need is minimum downtime you can restore your database in db_copy. Then drop your db and rename db_copy to db.
Yes you should disconnect all the users to be able to drop your db, but it will take minimum time, while if you delete your data the table will still be unavailable for the whole duration of the delete, and as delete is always fully logged your users will wait.
To launch your script you can use xp_cmdshell that calls sqlcmd with -i but it's not a good idea. You have no control on your script execution and if something goes wrong you will have even more downtime for your users.
Does your tables have FK defined?
Exec sp_MSFOREACHTABLE 'delete from ?
will try to delete everything in order it decides and you may end up with errors when you try to delete rows that are referenced in other tables.
To execute your sql file from Stored procedure .. you can use xp_cmdshell. See steps below
First Create a Batch File (C:\testApps\test.bat) and execute your sql file from there..
e.g.
osql -S TestSQlServer -E -I C:\testApps\test.sql > C:\testApps\tlog.txt
Then add this line to your Calling Stored procedure
exec xp_cmdshell 'C:\testApps\test.bat'
Execute your procedure
**Please note you will need to enable xp_cmdshell
You can use bulk insert like this:
BULK INSERT landofbeds.dbo.SalesOrderDetail
FROM '\\computer\share\folder\neworders.txt'
According to some recommendations i use bcp Utility to write SQL server Table to .cvs file so i could later transfer the data to informix table with the same structure .
My SQLServer Stored Procedure :
ALTER PROCEDURE [dbo].[TestCVS]
AS
BEGIN
declare #sql nvarchar(4000)
select #sql = 'bcp "select * from ML..gmp4vacationbalance" queryout c:\ss\Tom.cvs -c -t, -T -S' + ##servername
exec master..xp_cmdshell #sql
END
I have four questions concerning this procedure :
1- How to allow this procedure to write to remote server instead of local server ##servername,because it's not secure to allow specific server to access my sql server ?
2-How to allow to filter the query under specific condition :
say i want to write query like this :
select * from ML..gmp4vacationbalance where balance_date = #date AND emp_num = #empNum
3-when i execute the procedure i get data like this:
Why the third column appear corrupted like this , it's varchar desc written in arabic ?
4-When i want to delimit by pipe | instead of comma , like this
select #sql = 'bcp "select * from ML..gmp4vacationbalance" queryout c:\ss\Tom.cvs -c -t| -T -S' + ##servername
I get the following error :
Question 1: Writing from a remote server
I assume you meant to say "from" not "to". Specify the server name in form ServerName\InstanceName instead of using ##servername, you will need to have permissions to access the other server (since you are using a trusted connection -T)
Question 2: How to add parameters to the BCP statement
BCP is a command line utility not part of Transact-SQL. You can't add parameters. You can format the command line executed. You'll have to make your parameters #date and #empNum strings concatenate them with the rest of the SQL string for the command line
Question 3: Wrong characters in output
Instead of -c use -w to output Unicode characters
Question 4: Pipe not working
A common problem with BCP, simply quote the pipe like this -t"|" to make that the record separator
I have a .sql script with a lot of action queries that work on some staging tables. This script needs to be run twice with some other commands in-between i.e.:
Load the staging table from source A
Use do_stuff.sql to process it
Move the results somewhere.
Repeat Steps 1-3 for source B.
The brute force approach would be to just copy & paste dostuff.sql as needed. While this would technically work, is there a better way?
I'm hoping there's a command like RunThisSQL 'C:\do_stuff.sql' that I haven't discovered yet.
Update
Well, it's been about 5 years and I just re-discovered this old question. I did this recently and made a cursor to loop thru a master table. For each record in that master table, the script runs through an inner script using variables set by the master table.
https://www.mssqltips.com/sqlservertip/1599/sql-server-cursor-example/
If you use visual studio you can create "Sql Server Database" project. Withing the project you can create script that let you execute your *.sql files in a manner
/*
Post-Deployment Script Template
--------------------------------------------------------------------------------------
This file contains SQL statements that will be appended to the build script.
Use SQLCMD syntax to include a file in the post-deployment script.
Example: :r .\myfile.sql
Use SQLCMD syntax to reference a variable in the post-deployment script.
Example: :setvar TableName MyTable
SELECT * FROM [$(TableName)]
--------------------------------------------------------------------------------------
*/
see also. http://candordeveloper.com/2013/01/08/creating-a-sql-server-database-project-in-visual-studio-2012/
Try using xp_cmdshell.
EXEC xp_cmdshell 'sqlcmd -S ' + #ServerName + ' -d ' + #DBName + ' -i ' +#FileName
xp_cmdshell and concatenation do not play together nicely, often resulting in an "Incorrect syntax near '+'" error. So further to Jeotics solution above you will need to make a variable of the entire string you pass to xp_cmdshell (including quotes around anything that may contain a space (eg filepath\filename). This is mentioned in the Microsoft documentation for xp_cmdshell here. Other issues you will have to contend with are the default set up for SQL Server which has xp_cmdshell disabled as outlined here and granting permission to non-system administrators to use xp_cmdshell outlined here. The documentation generally advises against giving xp_cmdshell rights to too many people owing to it being a vehicle for those with malintent but if, like me, you have minimal and trustworthy database users then it seems like a reasonable solution. One last issue that requires correct configuration is the SQL Server Agent as outlined here. Documentation outlines that SQL Agent is responsible for background scheduling (such as back ups) and performance of command line statements, etc..
DECLARE
#Server nvarchar (50)
,#Database nvarchar(50)
,#File nvarchar(100)
,#cmd nvarchar(300);
SET #Server = server_name;
SET #Database = database_name;
SET #File = 'C:\your file path with spaces';
SET #cmd = 'sqlcmd -S ' + #Server + ' -d ' + #Database + ' i "' + #File + '"';
EXEC xp_cmdshell #cmd;
There are some security issues with enabling xp_cmdshell in SQL Server. You can create a CLR Stored procedure, which executes the passed file content. This CLR stored procedure is especially for this purpose, not like xp_cmdshell, which can do anything over the command prompt.
issues with enabling xp_cmdshell
Create CLR stored procedure
Heyy,
I'm trying to use BCP to export a SP result to a text file using this query:
EXEC xp_cmdshell 'bcp "exec asmary..usp_Contract_SelectByEmpId -1,1" queryout "C:\test.txt" -w -C OEM -t$ -T -r ~ -S heba\HEBADREAMNET '
The output of this query is telling this error:
Error = [Microsoft][SQL Server Native Client 10.0][SQL Server]Incorrect syntax near the keyword 'where'.
even thought I'm sure that the stored procedure "usp_Contract_SelectByEmpId" is working correctly.
Anyone faced that kind of error before?
As Lynn suggested, check your stored procedure. It looks like the issue is within that.
Ensure any plain SELECT works (e.g., C: drive is database server's local drive, not necessarily your own local drive).
If the first two items work fine, then add SET FMTONLY OFF as follows:
EXEC xp_cmdshell 'bcp "set fmtonly off exec asmary..usp_Contract_SelectByEmpId -1,1" queryout "C:\test.txt" -w -C OEM -t$ -T -r ~ -S heba\HEBADREAMNET'
I have to admit that when I tried similar on my computer it failed with 'Function sequence error', and I found that it is related to a SQL Server 2008 bug fixed in 2011.
Please note also that even without SET FMTONLY OFF everything works with BCP library (odbcbcp.dll/odbcbcp.lib). So, you can have much more generic ODBC-wide bcp solution if you write your own wrapper executable (for instance, in C or C++).
I also found the following at http://msdn.microsoft.com/en-us/library/ms162802.aspx
The query can reference a stored procedure as long as all tables referenced inside the stored procedure exist prior to executing the bcp statement. For example, if the stored procedure generates a temp table, the bcp statement fails because the temp table is available only at run time and not at statement execution time. In this case, consider inserting the results of the stored procedure into a table and then use bcp to copy the data from the table into a data file.
Please see also my later separate reply - I think the whole concept of using stored procedure for BCP/queryout is wrong.
Try this.
DECLARE #strbcpcmd NVARCHAR(max)
SET #strbcpcmd = 'bcp "EXEC asmary..usp_Contract_SelectByEmpId -1,1" queryout "C:\test.txt" -w -C OEM -t"$" -T -S'+##servername
EXEC master..xp_cmdshell #strbcpcmd
Sorry for flooding your question with multiple answers, but I wanted to find out how much heavier (performance-wise) the use of stored procedure is compared to plain SELECT. And I got a very important information from
http://social.msdn.microsoft.com/Forums/en-US/transactsql/thread/b8340289-7d7e-4a8d-b570-bec7a0d73ead/
This forced me to create another (separate) answer. The post I refer to invalidates the whole concept.
In a few words: stored procedure might be called several (3) times in order to figure out the structure of the resultset, then the actual data.
Therefore (and especially if calling from SQL Server connection rather than client), I think it makes a lot more sense to have a stored procedure or function, which will return SELECT statement. Then you can have another generic stored procedure or function to create and execute full BCP command with that statement embedded. I am pretty sure that in this case BCP might use a lot better execution plan. Unfortunately, I cannot verify that in practice, because of BCP bug in SQL Server 2008 R2 I mentioned in my previous post.
N.B. Please be careful creating dynamic queries and escape all explicit literal strings (i.e. repeat all single quotes twice) in order to avoid notorious SQL injection. Unfortunately, there is another pitfall: you should ensure you are not escaping your queries twice or more times.
Can we delete the physical file from server when I delete the corresponding entry from database?
i.e., C:\Test\Test.txt -> when deleting this record from database, i need to delete the corresponding Test.txt file from mentioned location.
Is there a way, I m using SQL 2008.
Any help would be highly appreciable..
Thanks
The ways are:
use of xp_cmdshell proc (exec master..xp_cmdshell 'del C:\Test\Test.txt')
use the .NET CLR unsafe proc (need to write in any .NET language and deploy to sql server. Its a long story)
Both ways are ugly
And once again - it is the worst practice. Server should not delete user files, or any files, is they are not integral part of its database.
You could use CREATE TRIGGER FOR DELETE to create a trigger that runs when rows are deleted. The SQL statement that is run upon deletion can walk through the table deleted to get the deleted rows. For each row it can exec xp_cmdshell. xp_cmdshell is disabled by default, so you must enable it first using exec sp_configure.
I didnt tested this but i think it should work.
Try writing a stored procedure which has the filename as a parameter and delete it using the:
exec master.dbo.xp_cmdshell 'del <Filename>'
then create a trigger for after delete on the table containing the Filenames which calls the stored procedure and provides the Filename from table deleted, or maby you can run directly the command exec master.dbo.xp_cmdshell 'del ' from the trigger.
The better way would be to save the files as an Object in the Database instead of the file path, and when deleting you just delete the file Object.