How to execute commands/functions in order - sql

I'm writing a long SQL query that I will be using to automate the process of ingesting large-ish flat files (using python to flatten heavily nested JSON files) and normalizing them for scalability and ease of use with PowerBI reports and dashboards.
Currently, I've got a long process that slices the table into multiple tables, generates mapping tables between them and the primary table, remaps a PK/FK link back to the primary table and drops the old unneeded columns from the primary table.
I'm still building and debugging the script, and I'm getting really frustrated with something that I think I'm doing wrong as I'm not very proficient in SQL.
Currently, if I try to run all of my code at once it will fail saying I'm using invalid column names. The column names are invalid with the tables in their current state, but if it would simply execute from top to bottom, they would be valid by the time it got to them. I've got to highlight and execute my drop tables statement by itself every time I want to rerun the entire script even though I've got the same drop tables statement at the top.
Any advice on how to make the script simply execute from top to bottom or how to make it step through and ignore the "current" state of the tables (prior to execution) would be greatly helpful.
Some example pseudo of what I've got:
CREATE OR ALTER PROCEDURE DropTables
AS
BEGIN
DROP TABLE IF EXISTS
t1,
t2,
t3
END
GO
CREATE OR ALTER PROCEDURE GenerateTable1
AS
BEGIN
~make table~
END
GO
CREATE OR ALTER PROCEDURE GenerateTable2
BEGIN
~make table~
END
GO
CREATE OR ALTER PROCEDURE GenerateTable3
BEGIN
~make table~
ALTER TABLE t1 ADD ~fk from t3~
UPDATE t1
SET ~keys to match~
FROM t3 WHERE t1.old_col = t3.new_col
ALTER TABLE t1
DROP COLUMN old_col
END
GO
EXEC DropTables
GO
EXEC GenerateTable1
GO
EXEC GenerateTable2
GO
EXEC GenerateTable3
Upon executing this I get "Invalid column name old_col" because old_col currently doesnt exist, however, if it would just execute from top to bottom, old_col would exist when it got to it.
Current workaround is highlighting droptables and executing it by itself first, then I can execute everything at once

GO breaks the script into batches. You just need to scope the batches so each one compiles. Or use dynamic SQL which is just a different way to issue separate batches.

It may not be the most elegant solution, but simply wrapping everything in an exec block with single quotes seems to work. "EXEC(' stuff ');" SQL doesnt try to get ahead of itself and is forced to execute in order from top to bottom, example:
CREATE OR ALTER PROCEDURE DropTables
AS
BEGIN
EXEC('
DROP TABLE IF EXISTS
t1,
t2,
t3
')
;
END
GO
CREATE OR ALTER PROCEDURE GenerateTable1
AS
BEGIN
EXEC('
~make table~
')
;
END
GO
CREATE OR ALTER PROCEDURE GenerateTable2
BEGIN
EXEC('
~make table~
')
;
END
GO
CREATE OR ALTER PROCEDURE GenerateTable3
BEGIN
EXEC('
~make table~
ALTER TABLE t1 ADD ~fk from t3~
UPDATE t1
SET ~keys to match~
FROM t3 WHERE t1.old_col = t3.new_col
ALTER TABLE t1
DROP COLUMN old_col
')
;
END
GO
EXEC DropTables
GO
EXEC GenerateTable1
GO
EXEC GenerateTable2
GO
EXEC GenerateTable3

Related

What is the process during re-naming and re-creating a MS-SQL table using stored procedure?

I have a table called myTable where continuous insertion is happening. I will rename that table by myTable_Date and create a new table, myTable through a Store Procedure.
I want to know what will happen during re-naming and re-creating the table, will it drop any packet?
SQL Server has sp_rename built in if you just want to change the name of a table.
sp_rename myTable, myTable_Date
Would change the name from myTable to myTable_Date
But it only changes the name reference in sys.Objects so make sure any references are altered and read the documentation about it :)
The Microsoft doc for it is HERE
When you rename the myTable to myTableDate, myTable won't exist anymore so if someone tries to insert something inside myTable it will fail.
When you create new myTable with the same name and columns everything will be fine and the insertion process will continue.
I suggest you to make a little script renaming the table and creating new one. Something like this:
sp_rename myTable, myTable_Date
GO
CREATE TABLE myTable(
-- Table definition
)
When you rename the table you will get warning like this: "Caution: Changing any part of an object name could break scripts and stored procedures." so you better create the new table fast.
Other option is you create a table exact like myTable and insert all data from myTable there and then can delete them from myTable. No renaming, no dropping and insertion process will not be interrupted.
I want to know what will happen during re-naming and re-creating the
table, will it drop any packet?
Inserts attempted after the table is renamed will err until the table is recreated. You can avoid that by executing the tasks in a transaction. Short term blocking will happen if an insert is attempted before the transaction is committed but no rows will be lost. For example:
CREATE PROC dbo.ReanmeMytableWithDate
AS
DECLARE #NewName sysname = 'mytable_' + CONVERT(nchar(8), SYSDATETIME(), 112);
SET XACT_ABORT ON;
BEGIN TRY;
BEGIN TRAN;
EXEC sp_rename N'dbo.mytable', #NewName;
CREATE TABLE dbo.mytable(
col1 int
);
COMMIT;
END TRY
BEGIN CATCH
THROW;
END CATCH;
GO
I don't know your use case for renaming tables like this but it seems table partitioning might be a better approach as #Damien_The_Unbeliever suggested. Although table partitioning previously required Enterprise Edition, the feature is available in Standard Edition beginning with SQL Server 2016 SP1 as well as Azure SQL Database.

How do I update triggers across multiple databases?

I have a query that I can select the databases from the sys.databases with the triggers that I wish to update. From there I can create a cursor. However when I go into the cursor to update my triggers using a dynamic db name #DatabaseExecuteName that is set to MyDatabaseName.dbo I receive the error ''CREATE/ALTER TRIGGER' does not allow specifying the database name as a prefix to the object name.' Because I am in a cursor I am not able to execute a USE MyDatabaseName ... GO, the GO statement is not allowed inside the CURSOR. I have tried SQLCMD MODE :setvar DatabaseName "MyDatabaseName" with USE [$(DatabaseName)]; to try to set the use database. I feel I am very close however my strength is not SQL queries. I could use some assistance on what I am missing.
You can nest EXEC calls so that you can use a USE and then execute a further statement and you don't need to use GO to seperate the batches. This is a complete script to demonstrate the technique:
create database DB1
go
create database DB2
go
use DB2
go
create table T1 (ID int not null)
go
create table T2 (ID int not null)
go
use DB1
go
exec('use DB2; exec(''create trigger T_T on T1 after insert as
insert into T2(ID) select i.ID from inserted i'')');
select DB_NAME()
insert into DB2..T1(ID) values (1),(2);
select * from DB2..T2
Which then shows that this connection is still in the DB1 database, but the trigger was successfully created on the T1 table within the DB2 database.
What you have to watch for is getting your quote-escaping correct.

StoredProc manipulating Temporary table throws 'Invalid column name' on execution

I have a a number of sp's that create a temporary table #TempData with various fields. Within these sp's I call some processing sp that operates on #TempData. Temp data processing depends on sp input parameters. SP code is:
CREATE PROCEDURE [dbo].[tempdata_proc]
#ID int,
#NeedAvg tinyint = 0
AS
BEGIN
SET NOCOUNT ON;
if #NeedAvg = 1
Update #TempData set AvgValue = 1
Update #TempData set Value = -1;
END
Then, this sp is called in outer sp with the following code:
USE [BN]
--GO
--DBCC FREEPROCCACHE;
GO
Create table #TempData
(
tele_time datetime
, Value float
--, AvgValue float
)
Create clustered index IXTemp on #TempData(tele_time);
insert into #TempData(tele_time, Value ) values( GETDATE(), 50 ); --sample data
declare
#ID int,
#UpdAvg int;
select
#ID = 1000,
#UpdAvg = 1
;
Exec dbo.tempdata_proc #ID, #UpdAvg ;
select * from #TempData;
drop table #TempData
This code throws an error: Msg 207, Level 16, State 1, Procedure tempdata_proc, Line 8: Invalid column name "AvgValue".
But if only I uncomment declaration AvgValue float - everything works OK.
The question: is there any workaround letting the stored proc code remain the same and providing a tip to the optimizer - skip this because AvgValue column will not be used by the sp due to params passed.
Dynamic SQL is not a welcomed solution BTW. Using alternative to #TempData tablename is undesireable solution according to existing tsql code (huge modifications necessary for that).
Tried SET FMTONLY, tempdb.tempdb.sys.columns, try-catch wrapping without any success.
The way that stored procedures are processed is split into two parts - one part, checking for syntactical correctness, is performed at the time that the stored procedure is created or altered. The remaining part of compilation is deferred until the point in time at which the store procedure is executed. This is referred to as Deferred Name Resolution and allows a stored procedure to include references to tables (not just limited to temp tables) that do not exist at the point in time that the procedure is created.
Unfortunately, when it comes to the point in time that the procedure is executed, it needs to be able to compile all of the individual statements, and it's at this time that it will discover that the table exists but that the column doesn't - and so at this time, it will generate an error and refuse to run the procedure.
The T-SQL language is unfortunately a very simplistic compiler, and doesn't take runtime control flow into account when attempting to perform the compilation. It doesn't analyse the control flow or attempt to defer the compilation in conditional paths - it just fails the compilation because the column doesn't (at this time) exist.
Unfortunately, there aren't any mechanisms built in to SQL Server to control this behaviour - this is the behaviour you get, and anything that addresses it is going to be perceived as a workaround - as evidenced already by the (valid) suggestions in the comments - the two main ways to deal with it are to use dynamic SQL or to ensure that the temp table always contains all columns required.
One way to workaround your concerns about maintenance if you go down the "all uses of the temp table should have all columns" is to move the column definitions into a separate stored procedure, that can then augment the temporary table with all of the required columns - something like:
create procedure S_TT_Init
as
alter table #TT add Column1 int not null
alter table #TT add Column2 varchar(9) null
go
create procedure S_TT_Consumer
as
insert into #TT(Column1,Column2) values (9,'abc')
go
create procedure S_TT_User
as
create table #TT (tmp int null)
exec S_TT_Init
insert into #TT(Column1) values (8)
exec S_TT_Consumer
select Column1 from #TT
go
exec S_TT_User
Which produces the output 8 and 9. You'd put your temp table definition in S_TT_Init, S_TT_Consumer is the inner query that multiple stored procedures call, and S_TT_User is an example of one such stored procedure.
Create the table with the column initially. If you're populating the TEMP table with SPROC output just make it an IDENTITY INT (1,1) so the columns line up with your output.
Then drop the column and re-add it as the appropriate data type later on in the SPROC.
The only (or maybe best) way i can thing off beyond dynamic SQL is using checks for database structure.
if exists (Select 1 From tempdb.sys.columns Where object_id=OBJECT_ID('tempdb.dbo.#TTT') and name = 'AvgValue')
begin
--do something AvgValue related
end
maybe create a simple function that takes table name and column or only column if its always #TempTable and retursn 1/0 if the column exists, would be useful in the long run i think
if dbo.TempTableHasField('AvgValue')=1
begin
-- do something AvgValue related
end
EDIT1: Dang, you are right, sorry about that, i was sure i had ... this.... :( let me thing a bit more

SQL Server Import and Export wizard gives invalid object error on stored procedure with temp table

I created a stored proc that creates a temp table, inserts, selects then drops. Executing the stored proc within SQL Server Management Studio works fine and gives the expected result.
CREATE PROCEDURE usp_TempTableTest
-- Add the parameters for the stored procedure here
#color VARCHAR(10)
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
CREATE TABLE #tmptable (
color VARCHAR(10)
)
INSERT INTO #tmptable (color) VALUES (#color)
SELECT color FROM #tmptable
DROP TABLE #tmptable
END
GO
However, when creating in the Import/Export tool and using that stored proc as the data source, it gives me the error:
Invalid object name '#tmptable'.
Any idea why this would happen? If I change it to a table variable it seems work fine with Import/Export, but I don't understand why it is not working with a temp table.
When I run a mimicked stored procedure, like yours above, in SSMS, I can get the data returned like you mentioned in the procedure. However, if I try the #tmptable, like you did, I also get the same error because the DROP TABLE removes it. From what I can tell, the import/export is basically a final INSERT process. The reason it works with the table variable is because the data still exist on the final insert; in the case of the DROP TABLE, it does not. For instance, when I remove the DROP TABLE, it works.
I might be wrong here, but it seems the logic when it's an import or export in the case of the above procedure is
INSERT data
SELECT data
DROP data
INSERT (import/export): this generates the "Invalid object name tmptable'"
With the variable (or no DROP), it's
INSERT data
SELECT data
INSERT (import/export)
In the second case, the data still exist. In the first case, they're gone. One way around it if you want to use the #tmptable, start your code with:
IF OBJECT_ID('tempdb..#tmptable') IS NOT NULL DROP TABLE #tmptable
Put "SET FMTONLY OFF;" right above "SET NOCOUNT ON"
http://msdn.microsoft.com/en-us/library/ms173839.aspx

Create SQL Server tables and stored procedures in one script?

I have a SQL script that is setting up two database tables with their keys and constraints without any problem. I won't include the whole code but the 'skeleton' of it looks like this:
BEGIN
CREATE TABLE [table] (
)
CREATE TABLE [table2] (
)
ALTER TABLE table...
ALTER TABLE table2....
END
I am stuck trying to add stored procedures to this script though, ideally I would like to include this all within the same script. Could someone tell me how to include the following stored procedure into the above script?
CREATE PROCEDURE Test
#x int
AS
BEGIN
SELECT COUNT(*)
FROM table
END
GO
I have tried putting it towards the end of the script and have also tried with and without the BEGIN, END and GO tags but I keep getting an error that says 'incorrect syntax near PROCEDURE'.
Try it like this:
USE BDNAME
GO
BEGIN
CREATE TABLE [table] (
)
CREATE TABLE [table2] (
)
ALTER TABLE table...
ALTER TABLE table2....
END
USE BDNAME
GO
CREATE PROCEDURE Test
#x int
AS
BEGIN
SELECT COUNT(*)
FROM table
END
GO
Instead of using BEGIN END, put GO between all your Statements like Create, Alter. Also I would like to inform you that putting GO will create blocks in your script, so if you create some local variable in one block, it is not accessible in another.
CREATE Table Table1(
--Your Code
)
GO
CREATE PROCEDURE Test
#x int
AS
BEGIN
SELECT COUNT(*)
FROM Table1
END
GO
--Continue your script
Hope this helps.