We have a horribly designed database that has procedures in it that generate new tables every week, and has been doing so for a couple of years. It's got a SQL script that gets 200+ columns of data via queries on various tables, dumps it all into a single table called Selections, and then after it's done you have to manually rename that table (Selections_Week001, Selections_Week002, etc) every week. Now we have around 100+ tables. What I want to do is put all that data into one table, with a column indicating its original source table (001, 002, etc), so I can then get rid of all those individual tables. Is there a way to do this using some SQL script, or am I going to have to manually edit and run 100 append queries?
Try this, for me worked well at test data:
select *
into Selections_Total
from Selections_Week001
where 1=0
alter table Selections_Total
add Original_Table varchar(64)
--select * From Selections_Total
declare #sql varchar(max)
select #sql = COALESCE(#sql + ' UNION ALL select *, ''' + name + ''' from ', 'select *, ''' + name + ''' from ') + name
from sys.tables where name like 'Selections_Week%'
--select #sql
insert into Selections_Total
exec(#sql)
UPD: missed with a column indicating its original source table at first. Now fixed.
Related
I have a linked server which contains many tables. Some of those tables contain a certain column called exampleColumn. I would like to select all of the records in the linked server which have a non-null value for the exampleColumn. (ignoring all tables which do not have this column and all records which have a null value for this column) I would like to select the entire row including all of the other columns if the exampleColumn has a value.
Additionally, I would like to create the tables and columns in my local server to house the selected data with the same names and datatypes as they are in the linked server (without creating tables which will have no records brought over)
I have found many resources online which describe how to select into from different databases based on a specific value, but I have not found any which checks all tables instead of just one. I also am unable to find anything on dynamically creating tables/columns based on the source of the select.
Not totally sure what you want here as there is not much in the way of details but pretty sure you can do something like this. This will generate all the insert statements you need to run. You could either stick this into a variable and execute it via dynamic sql or just copy and paste to run this. Not really sure where the linked server is here in relation to what you are running your query either.
declare #ColumnName sysname = 'exampleColumn'
select 'select * into ' + quotename(object_name(object_id)) + ' from [LinkServer].[Database].[Schema].' + quotename(object_name(object_id)) + ' where ' + QUOTENAME(#ColumnName) + ' is not null;'
from sys.columns
where name = #ColumnName
--EDIT--
To only create local tables for remote tables that have at least one row where your column is not null simply add an EXISTS before the select. Like this.
select 'IF EXISTS(select * from [LinkServer].[Database].[Schema].' + quotename(object_name(object_id)) + ' where ' + QUOTENAME(#ColumnName) + ' is not null) select * into ' + quotename(object_name(object_id)) + ' from [LinkServer].[Database].[Schema].' + quotename(object_name(object_id)) + ' where ' + QUOTENAME(#ColumnName) + ' is not null;'
from sys.columns
where name = #ColumnName
if exists (select 1 from [linkedserver].[db].[schema].[tableX]) begin
select *
into [schema].[tableX]
from [linkedserver].[db].[schema].[tableX]
end;
I have a requirement where I need to create multiple tables with the help of some existing tables but the new tables would have some different names.
For example:
Existing Table Names:
INT_Employees,
INT_HumanResource,
INT_Payroll,
INT_Contacts etc..
New Tables should have prefix STG so it should look like:
STG_Employees,
STG_HumanResource,
STG_Payroll,
STG_Contacts etc..
Note: There are around 127 tables and we need to create new 127 tables with prefix STG_.
Thanks in Advance.
If you are just trying to copy the table structure you can use select into with a where predicate that will prevent any rows from being returned.
select * into STG_Employees from INT_Employees where 1 = 0
Just repeat this for each table.
--EDIT--
Here is an example of you can leverage dynamic sql to do this task instead of resorting to loops. We simply use sys.tables to help us generate our sql statement.
declare #sql nvarchar(max) = ''
select #sql = #sql + 'select * into ' + stuff(name, 1, 4, 'STG_') + ' from ' + name + ' where 1 = 0;'
from sys.tables
where name like 'INT[_]%'
select #sql --once your comfortable the sql being generated is correct simply uncomment the next line
--exec sp_executesql #sql
Please note that if you want primary keys, indexes, foreign keys, computed columns etc you will need to generate those scripts another way. This will only generate the table structures.
It looks like #temptables created using dynamic SQL via the EXECUTE string method have a different scope and can't be referenced by "fixed" SQLs in the same stored procedure.
However, I can reference a temp table created by a dynamic SQL statement in a subsequence dynamic SQL but it seems that a stored procedure does not return a query result to a calling client unless the SQL is fixed.
A simple 2 table scenario:
I have 2 tables. Let's call them Orders and Items. Order has a Primary key of OrderId and Items has a Primary Key of ItemId. Items.OrderId is the foreign key to identify the parent Order. An Order can have 1 to n Items.
I want to be able to provide a very flexible "query builder" type interface to the user to allow the user to select what Items he want to see. The filter criteria can be based on fields from the Items table and/or from the parent Order table. If an Item meets the filter condition including and condition on the parent Order if one exists, the Item should be return in the query as well as the parent Order.
Usually, I suppose, most people would construct a join between the Item table and the parent Order tables. I would like to perform 2 separate queries instead. One to return all of the qualifying Items and the other to return all of the distinct parent Orders. The reason is two fold and you may or may not agree.
The first reason is that I need to query all of the columns in the parent Order table and if I did a single query to join the Orders table to the Items table, I would be repoeating the Order information multiple times. Since there are typically a large number of items per Order, I'd like to avoid this because it would result in much more data being transfered to a fat client. Instead, as mentioned, I would like to return the two tables individually in a dataset and use the two tables within to populate a custom Order and child Items client objects. (I don't know enough about LINQ or Entity Framework yet. I build my objects by hand). The second reason I would like to return two tables instead of one is because I already have another procedure that returns all of the Items for a given OrderId along with the parent Order and I would like to use the same 2-table approach so that I could reuse the client code to populate my custom Order and Client objects from the 2 datatables returned.
What I was hoping to do was this:
Construct a dynamic SQL string on the Client which joins the orders table to the Items table and filters appropriate on each table as specified by the custom filter created on the Winform fat-client app. The SQL build on the client would have looked something like this:
TempSQL = "
INSERT INTO #ItemsToQuery
OrderId, ItemsId
FROM
Orders, Items
WHERE
Orders.OrderID = Items.OrderId AND
/* Some unpredictable Order filters go here */
AND
/* Some unpredictable Items filters go here */
"
Then, I would call a stored procedure,
CREATE PROCEDURE GetItemsAndOrders(#tempSql as text)
Execute (#tempSQL) --to create the #ItemsToQuery table
SELECT * FROM Items WHERE Items.ItemId IN (SELECT ItemId FROM #ItemsToQuery)
SELECT * FROM Orders WHERE Orders.OrderId IN (SELECT DISTINCT OrderId FROM #ItemsToQuery)
The problem with this approach is that #ItemsToQuery table, since it was created by dynamic SQL, is inaccessible from the following 2 static SQLs and if I change the static SQLs to dynamic, no results are passed back to the fat client.
3 around come to mind but I'm look for a better one:
1) The first SQL could be performed by executing the dynamically constructed SQL from the client. The results could then be passed as a table to a modified version of the above stored procedure. I am familiar with passing table data as XML. If I did this, the stored proc could then insert the data into a temporary table using a static SQL that, because it was created by dynamic SQL, could then be queried without issue. (I could also investigate into passing the new Table type param instead of XML.) However, I would like to avoid passing up potentially large lists to a stored procedure.
2) I could perform all the queries from the client.
The first would be something like this:
SELECT Items.* FROM Orders, Items WHERE Order.OrderId = Items.OrderId AND (dynamic filter)
SELECT Orders.* FROM Orders, Items WHERE Order.OrderId = Items.OrderId AND (dynamic filter)
This still provides me with the ability to reuse my client sided object-population code because the Orders and Items continue to be returned in two different tables.
I have a feeling to, that I might have some options using a Table data type within my stored proc, but that is also new to me and I would appreciate a little bit of spoon feeding on that one.
If you even scanned this far in what I wrote, I am surprised, but if so, I woul dappreciate any of your thoughts on how to accomplish this best.
You first need to create your table first then it will be available in the dynamic SQL.
This works:
CREATE TABLE #temp3 (id INT)
EXEC ('insert #temp3 values(1)')
SELECT *
FROM #temp3
This will not work:
EXEC (
'create table #temp2 (id int)
insert #temp2 values(1)'
)
SELECT *
FROM #temp2
In other words:
Create temp table
Execute proc
Select from temp table
Here is complete example:
CREATE PROC prTest2 #var VARCHAR(100)
AS
EXEC (#var)
GO
CREATE TABLE #temp (id INT)
EXEC prTest2 'insert #temp values(1)'
SELECT *
FROM #temp
1st Method - Enclose multiple statements in the same Dynamic SQL Call:
DECLARE #DynamicQuery NVARCHAR(MAX)
SET #DynamicQuery = 'Select * into #temp from (select * from tablename) alias
select * from #temp
drop table #temp'
EXEC sp_executesql #DynamicQuery
2nd Method - Use Global Temp Table:
(Careful, you need to take extra care of global variable.)
IF OBJECT_ID('tempdb..##temp2') IS NULL
BEGIN
EXEC (
'create table ##temp2 (id int)
insert ##temp2 values(1)'
)
SELECT *
FROM ##temp2
END
Don't forget to delete ##temp2 object manually once your done with it:
IF (OBJECT_ID('tempdb..##temp2') IS NOT NULL)
BEGIN
DROP Table ##temp2
END
Note: Don't use this method 2 if you don't know the full structure on database.
I had the same issue that #Muflix mentioned. When you don't know the columns being returned, or they are being generated dynamically, what I've done is create a global table with a unique id, then delete it when I'm done with it, this looks something like what's shown below:
DECLARE #DynamicSQL NVARCHAR(MAX)
DECLARE #DynamicTable VARCHAR(255) = 'DynamicTempTable_' + CONVERT(VARCHAR(36), NEWID())
DECLARE #DynamicColumns NVARCHAR(MAX)
--Get "#DynamicColumns", example: SET #DynamicColumns = '[Column1], [Column2]'
SET #DynamicSQL = 'SELECT ' + #DynamicColumns + ' INTO [##' + #DynamicTable + ']' +
' FROM [dbo].[TableXYZ]'
EXEC sp_executesql #DynamicSQL
SET #DynamicSQL = 'IF OBJECT_ID(''tempdb..##' + #DynamicTable + ''' , ''U'') IS NOT NULL ' +
' BEGIN DROP TABLE [##' + #DynamicTable + '] END'
EXEC sp_executesql #DynamicSQL
Certainly not the best solution, but this seems to work for me.
I would strongly suggest you have a read through http://www.sommarskog.se/arrays-in-sql-2005.html
Personally I like the approach of passing a comma delimited text list, then parsing it with text to table function and joining to it. The temp table approach can work if you create it first in the connection. But it feel a bit messier.
Result sets from dynamic SQL are returned to the client. I have done this quite a lot.
You're right about issues with sharing data through temp tables and variables and things like that between the SQL and the dynamic SQL it generates.
I think in trying to get your temp table working, you have probably got some things confused, because you can definitely get data from a SP which executes dynamic SQL:
USE SandBox
GO
CREATE PROCEDURE usp_DynTest(#table_type AS VARCHAR(255))
AS
BEGIN
DECLARE #sql AS VARCHAR(MAX) = 'SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = ''' + #table_type + ''''
EXEC (#sql)
END
GO
EXEC usp_DynTest 'BASE TABLE'
GO
EXEC usp_DynTest 'VIEW'
GO
DROP PROCEDURE usp_DynTest
GO
Also:
USE SandBox
GO
CREATE PROCEDURE usp_DynTest(#table_type AS VARCHAR(255))
AS
BEGIN
DECLARE #sql AS VARCHAR(MAX) = 'SELECT * INTO #temp FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = ''' + #table_type + '''; SELECT * FROM #temp;'
EXEC (#sql)
END
GO
EXEC usp_DynTest 'BASE TABLE'
GO
EXEC usp_DynTest 'VIEW'
GO
DROP PROCEDURE usp_DynTest
GO
I need som help with a problem, our company have a vendor that deliver a database to us. inside that database, the vendor has a table with alot of t sql scripts. What i want to do is the following, i want to make a select to find the script and then execut the script and store the result in a variable or temp tabel. I can not alter the script from the vedor, so I need the result into something i can manupilate. Another problem is that i dont know how many columns ther result will have. So it has to be flexible. Like one script have 5 columns and and the next script has 8 and so on.
exsample:
DECLARE #SQL nvarchar(MAX) = ( Select distinct script_details
from scripttable where .......)
This will give me the script I want to use, then I use
EXEC(#SQL)
to execute the script.
Then my problem is, the result from this I want into a variable or a table.
I have tryed to make a temp table like this:
create table #TmpTblSP (col1 varchar(MAX),col2 varchar(MAX),col3 varchar(MAX),col4 varchar(MAX),col5 varchar(MAX),col6 varchar(MAX),col7 varchar(MAX),col8 varchar(MAX),col9 varchar(MAX),col10 varchar(MAX),col11 varchar(MAX),col12 varchar(MAX))
then
insert into #TmpTblSP
EXEC(#SQL)
This gives me the following error:
Msg 213, Level 16, State 7, Line 1
Column name or number of supplied values does not match table definition.
But if i know how many columns there are and specify that into the insert it works.
insert into #TmpTblSP(Col1,Col2,Col3)
EXEC(#SQL)
But here you se my problem, I dont know how many columns there are in every script. I could make one script for every script the vendor has, but that will be alot, it's like 3000 scripts in that table and they change them often.
You could try something like:
DECLARE #SQL nvarchar(MAX) = (
Select distinct script_details
into #temptbl
from scripttable where .......
);
EXEC(#SQL);
If you don't know how many columns yous #sql gives then the only solution is use SELECT INTO. I use it in this way:
DECLARE #QRY nvarchar(MAX) = ( Select distinct script_details
from scripttable where .......)
SET #sql = 'SELECT * into ' + #temptablename + ' FROM (' + #qry + ') A '
It gives some flexibility
Remember that it is easy to check structure of the table created in this way in sys so you can build another #SQL from this info if needed.
I this as well recommended to split "SELECT INTO" to 2 parts
One is
SELECT INTO ......... WHERE 1=2
Second
INSERT INTO SELECT ......
Creation of table locks all DB. So it is good to create it as fast as possible and then insert into it.
This is my first time posting on SO, so please go easy!
I'm attempting to write a SQL script that queries the same table for a list of databases in a single SQL Server instance.
I have successfully queried the list of databases that I required using the following, and inserting this data into a temp table.
Select name Into #Versions
From sys.databases
Where name Like 'Master%'
Master is suffixed with numerical values to identify different environments.
Select * From #Versions
Drop Table #Versions
The table name I am trying to query, is the same in each of the databases, and I want to extract the newest value from this table and insert it into the temp table for each of the database names returned.
I have tried researching this but to no avail. I am fairly comfy with SQL but I fear I could be out of my depth here.
You can do the following. Once you have the list of your databases, you can build up the query (you need to edit it for your purpose).
Select name Into #Versions
From sys.databases
Where name Like 'test%'
declare #sql as varchar(max) = ''
select #sql = #sql + 'INSERT INTO sometable SELECT TOP 1 * FROM ' + name + '..sourcetable ORDER BY somedate DESC; '
FROM #Versions
exec (#sql)
Drop Table #Versions
Look at The undocumented sp_MSforeachdb procedure and here