I'm new to T-SQL; all my experience is in a completely different database environment (Openedge). I've learned enough to write the procedure below -- but also enough to know that I don't know enough!
This routine will have to go into a live environment soon, and it works, but I'm quite certain there are a number of c**k-ups and gotchas in it that I know nothing about.
The routine copies data from table A to table B, replacing the data in table B. The tables could be in any database. I plan to call this routine multiple times from another stored procedure. Permissions aren't a problem: the routine will be run by the dba as a timed job.
Could I have your suggestions as to how to make it fit best-practice? To bullet-proof it?
ALTER PROCEDURE [dbo].[copyTable2Table]
#sdb varchar(30),
#stable varchar(30),
#tdb varchar(30),
#ttable varchar(30),
#raiseerror bit = 1,
#debug bit = 0
as
begin
set nocount on
declare #source varchar(65)
declare #target varchar(65)
declare #dropstmt varchar(100)
declare #insstmt varchar(100)
declare #ErrMsg nvarchar(4000)
declare #ErrSeverity int
set #source = '[' + #sdb + '].[dbo].[' + #stable + ']'
set #target = '[' + #tdb + '].[dbo].[' + #ttable + ']'
set #dropStmt = 'drop table ' + #target
set #insStmt = 'select * into ' + #target + ' from ' + #source
set #errMsg = ''
set #errSeverity = 0
if #debug = 1
print('Drop:' + #dropStmt + ' Insert:' + #insStmt)
-- drop the target table, copy the source table to the target
begin try
begin transaction
exec(#dropStmt)
exec(#insStmt)
commit
end try
begin catch
if ##trancount > 0
rollback
select #errMsg = error_message(),
#errSeverity = error_severity()
end catch
-- update the log table
insert into HHG_system.dbo.copyaudit
(copytime, copyuser, source, target, errmsg, errseverity)
values( getdate(), user_name(user_id()), #source, #target, #errMsg, #errSeverity)
if #debug = 1
print ( 'Message:' + #errMsg + ' Severity:' + convert(Char, #errSeverity) )
-- handle errors, return value
if #errMsg <> ''
begin
if #raiseError = 1
raiserror(#errMsg, #errSeverity, 1)
return 1
end
return 0
END
Thanks!
I'm speaking from a Sybase perspective here (I'm not sure if you're using SQLServer or Sybase) but I suspect you'll find the same issues in either environment, so here goes...
Firstly, I'd echo the comments made in earlier answers about the assumed dbo ownership of the tables.
Then I'd check with your DBAs that this stored proc will be granted permissions to drop tables in any database other than tempdb. In my experience, DBAs hate this and rarely provide it as an option due to the potential for disaster.
DDL operations like drop table are only allowed in a transaction if the database has been configured with the option sp_dboption my_database, "ddl in tran", true. Generally speaking, things done inside transactions involving DDL should be very short since they will lock up the frequently referenced system tables like sysobjects and in doing so, block the progress of other dataserver processes. Given that we've no way of knowing how much data needs to be copied, it could end up being a very long transaction which locks things up for everyone for a while. What's more, the DBAs will need to run that command on every database which contains tables that might contain a '#Target' table of this stored proc. If you were to use a transaction for the drop table it'd be a good idea to make it separate from any transaction handling the data insertion.
While you can do drop table commands in a transaction if the ddl in tran option is set, it's not possible to do select * into inside a transaction. Since select * into is a combination of table creation with insert, it would implicitly lock up the database (possibly for a while if there's a lot of data) if it were executed in a transaction.
If there are foreign key constraints on your #target table, you won't be able to just drop it without getting rid of the foreign key constraints first.
If you've got an 'id' column which relies upon a numeric identity type (often used as an autonumber feature to generate values for surrogate primary keys), be aware that you won't be able to copy the values from the '#Source' table's 'id' column across to the '#Target' table's id column.
I'd also check the size of your transaction log in any possible database which might hold a '#Target' table in relation to the size of any possible '#Source' table. Given that all the copying is being done in a single transaction, you may well find yourself copying a table so large that it blows out the transaction log in your prod dataserver, bringing all processes to a crashing halt. I've seen people using chunking to achieve this over particularly large tables, but then you end up needing to put your own checks into the code to make sure that you've actually captured a consistent snapshot of the table.
Just a thought - if this is being used to get snapshots, how about BCP? That could be used to dump out the contents of the table giving you the snapshot you're looking for. If you use the -c option you'd even get it in a human readable form.
All the best,
Stuart
This line seems a bit dangerous:
set #dropStmt = 'drop table ' + #target
What if the target table doesn't exist?
I'd try to safeguard that somehow - something like:
set #dropStmt =
'if object_id(' + #target + ') IS NOT NULL DROP TABLE ' + #target
That way, only if the call to OBJECT_ID(tablename) doesn't return NULL (that means: table doesn't exist) and the table is guaranteed to exist, issue the DROP TABLE statement.
Firstly, replace all the code like
set #source = '[' + #sdb + '].[dbo].[' + #stable + ']'
with code like
set #source = QuoteName(#sdb) + '.[dbo].' + QuoteName(#stable)
Secondly, your procedure assumes all objects are owned by dbo - this may not be the case.
Thirdly, your variable names are too short at 30 characters - 128 is the length of sysname.
I find the whole process you wrote to be terribly dangerous. Even if this is running from the database and not by the user, dynamic SQL is a poor practice! In databases using this to be able to do this to any table anytime is dangerous and would out and out be forbidden in the databases I work with. It is way too easy to accidentally drop the wrong tables! Nor is it possible to correctly test all possible values that the sp could run with, so this could be buggy code as well and you won't know until it has been in production.
Further, in dropping and recreating with select into, you must not have indexes or feoriegn key constraints or the things you need to havefor performance and data integrity. BAD BAD IDEA in general (OK if these are just staging tables of some type but not for anything else).
This is a task for SSIS. We save our SSIS packages and commit them to Subversion just like everything else. We can do a diff on them (they are just XML files) and we can tell what is running on prod and what configuration we are using.
You should not drop and recreate tables unless they are relatively small. You should update existing records, delete records no longer needed, and only add new ones. If you havea million records and 27000 have changed, 10 have been deleted, and 3000 are new, why drop and insert all 1,000,000 records. It is wasteful of server resources, could cause locking and blocking issues, and could create issues if the users are looking at the tables at the time you run this and the data suddenly disappears and takes some minutes to come back. Users get cranky about that.
Related
I need to create non-temporary tables in a MariaDB 10.3 database using Node. I therefore need a way of generating a table name that is guaranteed to be unique.
The Node function cannot access information regarding any unique feature about what or when the tables are made, so I cannot build the name from a timestamp or connection ID. I can only verify the name's uniqueness using the current database.
This question had a PostgreSQL answer suggesting the following:
SET #name = GetBigRandomNumber();
WHILE TableExists(#name)
BEGIN
SET #name = GetBigRandomNumber();
END
I attempted a MariaDB implementation using #name = CONCAT(MD5(RAND()),MD5(RAND())) to generate a random 64 character string, and (COUNT(*) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE #name) >0 to check if it was a unique name:
SET #name = CONCAT(MD5(RAND()),MD5(RAND()));
WHILE ((COUNT(*) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE #name) >0) DO
SET #name = CONCAT(MD5(RAND()),MD5(RAND()));
END WHILE;
CREATE TABLE #name ( ... );
However I get a syntax error when I try to run the above query. My SQL knowledge isn't that great so I'm at a loss as to what the problem might be.
Furthermore, is this approach efficient? The randomly generated name is long enough that it is very unlikely to have any clashes with any current table in the database, so the WHILE loop will very rarely need to run, but is there some sort of built in function to auto increment table names, or something similar?
SET #name := UUID();
If the dashes in that cause trouble, then
SET #name := REPLACE(UUID(), '-', '');
It will be safer (toward uniqueness) than RAND(). And, in theory, there will be no need to verify its uniqueness. After all, that's the purpose of UUIDs.
I have database on Sql Server 2008 R2.
On that database a delete query on 400 Million records, has been running for 4 days , but I need to reboot the machine. How can I force it to commit whatever is deleted so far? I want to reject that data which is deleted by running query so far.
But problem is that query is still running and will not complete before the server reboot.
Note : I have not set any isolation / begin/end transaction for the query. The query is running in SSMS studio.
If machine reboot or I cancelled the query, then database will go in recovery mode and it will recovering for next 2 days, then I need to re-run the delete and it will cost me another 4 days.
I really appreciate any suggestion / help or guidance in this.
I am novice user of sql server.
Thanks in Advance
Regards
There is no way to stop SQL Server from trying to bring the database into a transactionally consistent state. Every single statement is implicitly a transaction itself (if not part of an outer transaction) and is executing either all or nothing. So if you either cancel the query or disconnect or reboot the server, SQL Server will from transaction log write the original values back to the updated data pages.
Next time when you delete so many rows at once, don't do it at once. Divide the job in smaller chunks (I always use 5.000 as a magic number, meaning I delete 5000 rows at the time in the loop) to minimize transaction log use and locking.
set rowcount 5000
delete table
while ##rowcount = 5000
delete table
set rowcount 0
If you are deleting that many rows you may have a better time with truncate. Truncate deletes all rows from the table very efficiently. However, I'm assuming that you would like to keep some of the records in the table. The stored procedure below backs up the data you would like to keep into a temp table then truncates then re-inserts the records that were saved. This can clean a huge table very quickly.
Note that truncate doesn't play well with Foreign Key constraints so you may need to drop those then recreate them after cleaned.
CREATE PROCEDURE [dbo].[deleteTableFast] (
#TableName VARCHAR(100),
#WhereClause varchar(1000))
AS
BEGIN
-- input:
-- table name: is the table to use
-- where clause: is the where clause of the records to KEEP
declare #tempTableName varchar(100);
set #tempTableName = #tableName+'_temp_to_truncate';
-- error checking
if exists (SELECT [Table_Name] FROM Information_Schema.COLUMNS WHERE [TABLE_NAME] =(#tempTableName)) begin
print 'ERROR: already temp table ... exiting'
return
end
if not exists (SELECT [Table_Name] FROM Information_Schema.COLUMNS WHERE [TABLE_NAME] =(#TableName)) begin
print 'ERROR: table does not exist ... exiting'
return
end
-- save wanted records via a temp table to be able to truncate
exec ('select * into '+#tempTableName+' from '+#TableName+' WHERE '+#WhereClause);
exec ('truncate table '+#TableName);
exec ('insert into '+#TableName+' select * from '+#tempTableName);
exec ('drop table '+#tempTableName);
end
GO
You must know D(Durability) in ACID first before you understand why database goes to Recovery mode.
Generally speaking, you should avoid long running SQL if possible. Long running SQL means more lock time on resource, larger transaction log and huge rollback time when it fails.
Consider divided your task some id or time. For example, you want to insert large volume data from TableSrc to TableTarget, you can write query like
DECLARE #BATCHCOUNT INT = 1000;
DECLARE #Id INT = 0;
DECLARE #Max = ...;
WHILE Id < #Max
BEGIN
INSERT INTO TableTarget
FROM TableSrc
WHERE PrimaryKey >= #Id AND #PrimaryKey < #Id + #BatchCount;
SET #Id = #Id + #BatchCount;
END
It's ugly more code and more error prone. But it's the only way I know to deal with huge data volume.
We have a lot of data and to save time on making backups I have been tasked with creating a copy of the current database which only contains specified company codes/companies.
I have been told to research into TSQL scripts, information schemas and bulk copies.Im just wondering what the best route is to go down and how to make a start I do have the latest database copy locally.
The script would work by
Allow us at the top to specify name of new database and a list of company codes or ids from company table.
And create a new database and copy over only the data that is relevant
Probably not even close to the best way... but 1 of 100 ways you can do it.
declare #databasename nvarchar(100) = 'roflcopter'
, #destination = 'myschema.mytable'
declare #companylist nvarchar(max) = (select stuff( select ',' +convert(nvarchar(5),companyid) from companytable where database = #databasename),1,1,'')
declare #query nvarchar(max)
= N'use ['+#databasename+']
insert into '+#destination+' /* maybe add dynamic columns here */
select /*same dynamic columns */
from mytable /*or another variable*/
where companyid in('''+#companylist+''')
'
exec (#query)
add your other variables and such to declare destination, future database name (if creating) and anything you want... dynamic sql can be slow, cumbersome and does reconnect. Might want to check the sp_executesql for differences between exec () and executesql
there are a ton of system procedures... ones that find FKs, column names, data types, pks... you can complicate it by building dynamically creating your current schema using (no help from me here) those system procedures, C# to easily 'generate scripts' and execute them.
May also be in the wrong direction you were thinking of.
I am getting
Statement 'SELECT INTO' is not supported in this version of SQL Server
in SQL Server
for the below query inside stored procedure
DECLARE #sql NVARCHAR(MAX)
,#sqlSelect NVARCHAR(MAX) = ''
,#sqlFrom NVARCHAR(MAX) = ''
,#sqlTempTable NVARCHAR(MAX) = '#itemSearch'
,#sqlInto NVARCHAR(MAX) = ''
,#params NVARCHAR(MAX)
SET #sqlSelect ='SELECT
,IT.ITEMNR
,IT.USERNR
,IT.ShopNR
,IT.ITEMID'
SET #sqlFrom =' FROM dbo.ITEM AS IT'
SET #sqlInto = ' INTO ' + #sqlTempTable + ' ';
IF (#cityId > 0)
BEGIN
SET #sqlFrom = #sqlFrom +
' INNER JOIN dbo.CITY AS CI2
ON CI2.CITYID = #cityId'
SET #sqlSelect = #sqlSelect +
'CI2.LATITUDE AS CITYLATITUDE
,CI2.LONGITUDE AS CITYLONGITUDE'
END
SELECT #params =N'#cityId int '
SET #sql = #sqlSelect +#sqlInto +#sqlFrom
EXEC sp_executesql #sql,#params
I have around 50,000 records, so decided to use Temp Table. But surprised to see this error.
How can i achieve the same in SQL Azure?
Edit: Reading this blog http://blogs.msdn.com/b/sqlazure/archive/2010/05/04/10007212.aspx suggesting us to CREATE a Table inside Stored procedure for storing data instead of Temp table. Is it safe under concurrency? Will it hit performance?
Adding some points taken from http://blog.sqlauthority.com/2011/05/28/sql-server-a-quick-notes-on-sql-azure/
Each Table must have clustered index. Tables without a clustered index are not supported.
Each connection can use single database. Multiple database in single transaction is not supported.
‘USE DATABASE’ cannot be used in Azure.
Global Temp Tables (or Temp Objects) are not supported.
As there is no concept of cross database connection, linked server is not the concept in Azure at this moment.
SQL Azure is shared environment and because of the same there is no concept of Windows Login.
Always drop TempDB objects after their need as they create pressure on TempDB.
During buck insert use batchsize option to limit the number of rows to be inserted. This will limit the usage of Transaction log space.
Avoid unnecessary usage of grouping or blocking ORDER by operations as they leads to high end memory usage.
SELECT INTO is one of the many things that you can unfortunately not perform in SQL Azure.
What you'd have to do is first create the temporary table, then perform the insert. Something like:
CREATE TABLE #itemSearch (ITEMNR INT, USERNR INT, IT.ShopNR INT, IT.ITEMID INT)
INSERT INTO #itemSearch
SELECT IT.ITEMNR, IT.USERNR, IT.ShopNR ,IT.ITEMID
FROM dbo.ITEM AS IT
The new Azure DB Update preview has this problem resolved:
The V12 preview enables you to create a table that has no clustered
index. This feature is especially helpful for its support of the T-SQL
SELECT...INTO statement which creates a table from a query result.
http://azure.microsoft.com/en-us/documentation/articles/sql-database-preview-whats-new/
Create the table using # prefix, e.g. create table #itemsearch then use insert into. The scope of the temp table is limited to the session so there will no concurrency problems.
Well, As we all know SQL Azure table must have a clustered index, that is why SELECT INTO failure copy data from one table in to another table.
If you want to migrate, you must create a table first with same structure and then execute INSERT INTO statement.
For temporary table which followed by # you don't need to create Index.
how to create index and how to execute insert into for temp table?
I was wondering how / if it's possible to have a query which drops all temporary tables?
I've been trying to work something out using the tempdb.sys.tables, but am struggling to format the name column to make it something that can then be dropped - another factor making things a bit trickier is that often the temp table names contain a '_' which means doing a replace becomes a bit more fiddly (for me at least!)
Is there anything I can use that will drop all temp tables (local or global) without having to drop them all individually on a named basis?
Thanks!
The point of temporary tables is that they are.. temporary. As soon as they go out of scope
#temp create in stored proc : stored proc exits
#temp created in session : session disconnects
##temp : session that created it disconnects
The query disappears. If you find that you need to remove temporary tables manually, you need to revisit how you are using them.
For the global ones, this will generate and execute the statement to drop them all.
declare #sql nvarchar(max)
select #sql = isnull(#sql+';', '') + 'drop table ' + quotename(name)
from tempdb..sysobjects
where name like '##%'
exec (#sql)
It is a bad idea to drop other sessions' [global] temp tables though.
For the local (to this session) temp tables, just disconnect and reconnect again.
The version below avoids all of the hassles of dealing with the '_'s. I just wanted to get rid of non-global temp tables, hence the '#[^#]%' in my WHERE clause, drop the [^#] if you want to drop global temp tables as well, or use a '##%' if you only want to drop global temp tables.
The DROP statement seems happy to take the full name with the '_', etc., so we don't need to manipulate and edit these. The OBJECT_ID(...) NOT NULL allows me to avoid tables that were not created by my session, presumably since these tables should not be 'visible' to me, they come back with NULL from this call. The QUOTENAME is needed to make sure the name is correctly quoted / escaped. If you have no temp tables, #d_sql will be the empty string still, so we check for that before printing / executing.
DECLARE #d_sql NVARCHAR(MAX)
SET #d_sql = ''
SELECT #d_sql = #d_sql + 'DROP TABLE ' + QUOTENAME(name) + ';
'
FROM tempdb..sysobjects
WHERE name like '#[^#]%'
AND OBJECT_ID('tempdb..'+QUOTENAME(name)) IS NOT NULL
IF #d_sql <> ''
BEGIN
PRINT #d_sql
-- EXEC( #d_sql )
END
In a stored procedure they are dropped automatically when the execution of the proc completes.
I normally come across the desire for this when I copy code out of a stored procedure to debug part of it and the stored proc does not contain the drop table commands.
Closing and reopening the connection works as stated in the accepted answer. Rather than doing this manually after each execution you can enable SQLCMD mode on the Query menu in SSMS
And then use the :connect command (adjust to your server/instance name)
:connect (local)\SQL2014
create table #foo(x int)
create table #bar(x int)
select *
from #foo
Can be run multiple times without problems. The messages tab shows
Connecting to (local)\SQL2014...
(0 row(s) affected)
Disconnecting connection from (local)\SQL2014...