I have a table with Table name, Id, Column Name and Value
I need to run these into a Azure SQL Database as updates.
The tables are all different with different columns for the update.
The only way I could think of is RBAR, but there has to be an easier way to process this, either SQL or maybe even data factory?
Example of the data is:
Table Name Id Column Name Value
Table 1 1234 column1 1
Table 1 1234 column2 2022-01-02
Table 2 4321 column6 2144
Table 2 4321 column12 2022-01-02
The column name could be any column in the table with a update value.
As mentioned I have tried Row By Agonising Row but, as you would expect, WAY too painful as I have approx 161K rows that need processing.
You'll need to build an SQL string for each row and run that string as it's own command.
One thing I would do as part of this is pull the table/column information from Information_Schema.Columns. This will give you the same text you already have, but it ensures (in a safe way) that the data in your source table really is a valid object in your database, and not an SQL injection attempt. I would also use sp_executesql for the final query, to include the new value in a safe way.
So the main query would like looks this:
SELECT d.ID, d.Value,
'UPDATE ' + QUOTENAME(c.Table_Name) + ' SET ' + QUOTENAME(c.Column_Name) + ' = #new_value WHERE ID= #RowID' as SQL
FROM [MyData] d
INNER JOIN Information_Schema.Columns c ON c.[Table Name] = d.Table_Name
AND c.[Column Name] = d.Column_Name
And the loop would then take each row and do something like this:
-- #SQL, #ID, and #Value from the row
Execute sp_executeSql #SQL, '#new_value varchar(50), #id int', #new_value = #Value, #RowID = #ID
I skipped over the part where you loop through the rows because you have many options here, and which is best depends on your specific situation: do you want a cursor? Client code? Hard to know.
The trick here is we don't know the data type of the new value... and it certainly looks like you're mixing data types in there. You know that, for example, storing dates in varchar columns is really bad, right?
Related
Is it possible to test for a column before selecting it within a select statement?
This may be rough for me to explain, I have actually had to teach myself dynamic SQL over the past 4 months. I am using a dynamically generated parameter (#TableName) to store individual tables within a loop (apologize for the vagueness, but the details aren't relevant).
I then want to be able to be able to conditionally select a column from the table (I will not know if each table has certain columns). I have figured out how to check for a column outside of a select statement...
SET #SQLQuery2 = 'Select #OPFolderIDColumnCheck = Column_Name From INFORMATION_SCHEMA.COLUMNS Where Table_Name = #TABLENAME And Column_Name = ''OP__FolderID'''
SET #ParameterDefinition2 = N'#TABLENAME VARCHAR(100), #OPFolderIDColumnCheck VARCHAR(100) OUTPUT'
EXECUTE SP_EXECUTESQL #SQLQuery2, #ParameterDefinition2, #TABLENAME, #OPFolderIDColumnCheck OUTPUT
IF #OPFolderIDColumnCheck IS NULL
BEGIN
SET #OP__FOLDERID = NULL
END
ELSE
IF #OPFolderIDColumnCheck IS NOT NULL
BEGIN
...etc
but id like to be able to do it inside of a select statement. Is there a way to check and see if OP__FOLDERID exists in the table?
Id like to be able to do something like this:
SELECT IF 'OP__FOLDERID' EXISTS IN [TABLE] THEN 'OP__FOLDERID' FROM [TABLE]
Thank you for any help or direction you can offer.
I'm afraid there isn't any direct way to do this within a SELECT statement at all. You can determine if a column exists in a table, however, and construct your dynamic SQL accordingly. To do this, use something like this:
IF COL_LENGTH('schemaName.tableName', 'columnName') IS NOT NULL
BEGIN
-- Column Exists
END
You could then set a variable as a flag, and the code to construct the dynamic SQL would construct the expression with/without the column, as desired. Another approach would be to use a string value, and set it to the column name if it is present (perhaps with a prefix or suffix comma, as appropriate to the expression). This would allow you to save writing conditionals in the expression building, and would be particularly helpful where you have more than one or two of these maybe-columns in a dynamic expression.
It looks like #temptables created using dynamic SQL via the EXECUTE string method have a different scope and can't be referenced by "fixed" SQLs in the same stored procedure.
However, I can reference a temp table created by a dynamic SQL statement in a subsequence dynamic SQL but it seems that a stored procedure does not return a query result to a calling client unless the SQL is fixed.
A simple 2 table scenario:
I have 2 tables. Let's call them Orders and Items. Order has a Primary key of OrderId and Items has a Primary Key of ItemId. Items.OrderId is the foreign key to identify the parent Order. An Order can have 1 to n Items.
I want to be able to provide a very flexible "query builder" type interface to the user to allow the user to select what Items he want to see. The filter criteria can be based on fields from the Items table and/or from the parent Order table. If an Item meets the filter condition including and condition on the parent Order if one exists, the Item should be return in the query as well as the parent Order.
Usually, I suppose, most people would construct a join between the Item table and the parent Order tables. I would like to perform 2 separate queries instead. One to return all of the qualifying Items and the other to return all of the distinct parent Orders. The reason is two fold and you may or may not agree.
The first reason is that I need to query all of the columns in the parent Order table and if I did a single query to join the Orders table to the Items table, I would be repoeating the Order information multiple times. Since there are typically a large number of items per Order, I'd like to avoid this because it would result in much more data being transfered to a fat client. Instead, as mentioned, I would like to return the two tables individually in a dataset and use the two tables within to populate a custom Order and child Items client objects. (I don't know enough about LINQ or Entity Framework yet. I build my objects by hand). The second reason I would like to return two tables instead of one is because I already have another procedure that returns all of the Items for a given OrderId along with the parent Order and I would like to use the same 2-table approach so that I could reuse the client code to populate my custom Order and Client objects from the 2 datatables returned.
What I was hoping to do was this:
Construct a dynamic SQL string on the Client which joins the orders table to the Items table and filters appropriate on each table as specified by the custom filter created on the Winform fat-client app. The SQL build on the client would have looked something like this:
TempSQL = "
INSERT INTO #ItemsToQuery
OrderId, ItemsId
FROM
Orders, Items
WHERE
Orders.OrderID = Items.OrderId AND
/* Some unpredictable Order filters go here */
AND
/* Some unpredictable Items filters go here */
"
Then, I would call a stored procedure,
CREATE PROCEDURE GetItemsAndOrders(#tempSql as text)
Execute (#tempSQL) --to create the #ItemsToQuery table
SELECT * FROM Items WHERE Items.ItemId IN (SELECT ItemId FROM #ItemsToQuery)
SELECT * FROM Orders WHERE Orders.OrderId IN (SELECT DISTINCT OrderId FROM #ItemsToQuery)
The problem with this approach is that #ItemsToQuery table, since it was created by dynamic SQL, is inaccessible from the following 2 static SQLs and if I change the static SQLs to dynamic, no results are passed back to the fat client.
3 around come to mind but I'm look for a better one:
1) The first SQL could be performed by executing the dynamically constructed SQL from the client. The results could then be passed as a table to a modified version of the above stored procedure. I am familiar with passing table data as XML. If I did this, the stored proc could then insert the data into a temporary table using a static SQL that, because it was created by dynamic SQL, could then be queried without issue. (I could also investigate into passing the new Table type param instead of XML.) However, I would like to avoid passing up potentially large lists to a stored procedure.
2) I could perform all the queries from the client.
The first would be something like this:
SELECT Items.* FROM Orders, Items WHERE Order.OrderId = Items.OrderId AND (dynamic filter)
SELECT Orders.* FROM Orders, Items WHERE Order.OrderId = Items.OrderId AND (dynamic filter)
This still provides me with the ability to reuse my client sided object-population code because the Orders and Items continue to be returned in two different tables.
I have a feeling to, that I might have some options using a Table data type within my stored proc, but that is also new to me and I would appreciate a little bit of spoon feeding on that one.
If you even scanned this far in what I wrote, I am surprised, but if so, I woul dappreciate any of your thoughts on how to accomplish this best.
You first need to create your table first then it will be available in the dynamic SQL.
This works:
CREATE TABLE #temp3 (id INT)
EXEC ('insert #temp3 values(1)')
SELECT *
FROM #temp3
This will not work:
EXEC (
'create table #temp2 (id int)
insert #temp2 values(1)'
)
SELECT *
FROM #temp2
In other words:
Create temp table
Execute proc
Select from temp table
Here is complete example:
CREATE PROC prTest2 #var VARCHAR(100)
AS
EXEC (#var)
GO
CREATE TABLE #temp (id INT)
EXEC prTest2 'insert #temp values(1)'
SELECT *
FROM #temp
1st Method - Enclose multiple statements in the same Dynamic SQL Call:
DECLARE #DynamicQuery NVARCHAR(MAX)
SET #DynamicQuery = 'Select * into #temp from (select * from tablename) alias
select * from #temp
drop table #temp'
EXEC sp_executesql #DynamicQuery
2nd Method - Use Global Temp Table:
(Careful, you need to take extra care of global variable.)
IF OBJECT_ID('tempdb..##temp2') IS NULL
BEGIN
EXEC (
'create table ##temp2 (id int)
insert ##temp2 values(1)'
)
SELECT *
FROM ##temp2
END
Don't forget to delete ##temp2 object manually once your done with it:
IF (OBJECT_ID('tempdb..##temp2') IS NOT NULL)
BEGIN
DROP Table ##temp2
END
Note: Don't use this method 2 if you don't know the full structure on database.
I had the same issue that #Muflix mentioned. When you don't know the columns being returned, or they are being generated dynamically, what I've done is create a global table with a unique id, then delete it when I'm done with it, this looks something like what's shown below:
DECLARE #DynamicSQL NVARCHAR(MAX)
DECLARE #DynamicTable VARCHAR(255) = 'DynamicTempTable_' + CONVERT(VARCHAR(36), NEWID())
DECLARE #DynamicColumns NVARCHAR(MAX)
--Get "#DynamicColumns", example: SET #DynamicColumns = '[Column1], [Column2]'
SET #DynamicSQL = 'SELECT ' + #DynamicColumns + ' INTO [##' + #DynamicTable + ']' +
' FROM [dbo].[TableXYZ]'
EXEC sp_executesql #DynamicSQL
SET #DynamicSQL = 'IF OBJECT_ID(''tempdb..##' + #DynamicTable + ''' , ''U'') IS NOT NULL ' +
' BEGIN DROP TABLE [##' + #DynamicTable + '] END'
EXEC sp_executesql #DynamicSQL
Certainly not the best solution, but this seems to work for me.
I would strongly suggest you have a read through http://www.sommarskog.se/arrays-in-sql-2005.html
Personally I like the approach of passing a comma delimited text list, then parsing it with text to table function and joining to it. The temp table approach can work if you create it first in the connection. But it feel a bit messier.
Result sets from dynamic SQL are returned to the client. I have done this quite a lot.
You're right about issues with sharing data through temp tables and variables and things like that between the SQL and the dynamic SQL it generates.
I think in trying to get your temp table working, you have probably got some things confused, because you can definitely get data from a SP which executes dynamic SQL:
USE SandBox
GO
CREATE PROCEDURE usp_DynTest(#table_type AS VARCHAR(255))
AS
BEGIN
DECLARE #sql AS VARCHAR(MAX) = 'SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = ''' + #table_type + ''''
EXEC (#sql)
END
GO
EXEC usp_DynTest 'BASE TABLE'
GO
EXEC usp_DynTest 'VIEW'
GO
DROP PROCEDURE usp_DynTest
GO
Also:
USE SandBox
GO
CREATE PROCEDURE usp_DynTest(#table_type AS VARCHAR(255))
AS
BEGIN
DECLARE #sql AS VARCHAR(MAX) = 'SELECT * INTO #temp FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = ''' + #table_type + '''; SELECT * FROM #temp;'
EXEC (#sql)
END
GO
EXEC usp_DynTest 'BASE TABLE'
GO
EXEC usp_DynTest 'VIEW'
GO
DROP PROCEDURE usp_DynTest
GO
I have a stored procedure that is called from an SSIS package several times with a different value as an argument. The stored procedure contains an open query to an SSRS cube, and returns a list of people. The resultset is stored in a temporary table.
CREATE TABLE #tmp (Person varchar(50), Cat1 bit, Cat2 bit, Cat3 bit, Cat4 bit, Cat5 bit)
INSERT INTO #tmp EXEC sys.sp_executesql #query
SET #sqlCommand = 'SELECT Person FROM #tmp WHERE ' + #Category + '= 1'
EXEC (#sqlCommand);
For each person in the resultset, there should be an insert to another table with some information. The information is the name of the person, some other static content, and a value which I can get from running another stored procedure and count the rows returned.
An example of an insert could be
Name Static info ##rowcount
+-------------++--------------++--------------+
|Homer Simpson||Something here||ValueFromCount|
+-------------++--------------++--------------+
As of now, I have a prewritten stored procedure that inserts a row to a the desired table, but it has a parameter which is the person name. I get the ##rowcount value by executing the other SP from the current SP (a bit nested but works fine).
I could easily use an SQL WHILE loop or a CURSOR for each person in the resultset, and call the insert SP, but I can't help to think that there must be a more efficient way of doing this!
The flow right now is
SSIS package executes the stored procedure that returns a list of persons
For each person in the list, execute the SP_Insert_To_Table procedure.
2.1. From SP_Insert_To_Table procedure, execute SP_Get_All_Actions and select ##rowcount as the dynamic value to be inserted.
I have the idea that I might be able to join the result from the #tmp table with the count of actions the person has done, so that I would get a new resultset that could be inserted all in one, but I can't figure it out since the count is specific for each person in the list.
Is there a way to to this in a set based fashion instead of using a procedural approach?
With no details to work with this is a bit challenging but it certainly sounds like removing the loop should be pretty straight forward. All I can do is rough pseudocode.
Insert into SomeTable (ColumnsListedHere)
from SomeOtherTable
join Something on some conditions
where SomeCriteria = GetListOfPerson
Brief history:
I'm writing a stored procedure to support a legacy reporting system (using SQL Server Reporting Services 2000) on a legacy web application.
In keeping with the original implementation style, each report has a dedicated stored procedure in the database that performs all the querying necessary to return a "final" dataset that can be rendered simply by the report server.
Due to the business requirements of this report, the returned dataset has an unknown number of columns (it depends on the user who executes the report, but may have 4-30 columns).
Throughout the stored procedure, I keep a column UserID to track the user's ID to perform additional querying. At the end, however, I do something like this:
UPDATE #result
SET Name = ppl.LastName + ', ' + ppl.FirstName
FROM #result r
LEFT JOIN Users u ON u.id = r.userID
LEFT JOIN People ppl ON ppl.id = u.PersonID
ALTER TABLE #result
DROP COLUMN [UserID]
SELECT * FROM #result r ORDER BY Name
Effectively I set the Name varchar column (that was previously left NULL while I was performing some pivot logic) to the desired name format in plain text.
When finished, I want to drop the UserID column as the report user shouldn't see this.
Finally, the data set returned has one column for the username, and an arbitrary number of INT columns with performance totals. For this reason, I can't simply exclude the UserID column since SQL doesn't support "SELECT * EXCEPT [UserID]" or the like.
With this known (any style pointers are appreciated but not central to this problem), here's the problem:
When I execute this stored procedure, I get an execution error:
Invalid column name 'userID'.
However, if I comment out my DROP COLUMN statement and retain the UserID, the stored procedure performs correctly.
What's going on? It certainly looks like the statements are executing out of order and it's dropping the column before I can use it to set the name strings!
[Edit 1]
I defined UserID previously (the whole stored procedure is about 200 lies of mostly irrelevant logic, so I'll paste snippets:
CREATE TABLE #result ([Name] NVARCHAR(256), [UserID] INT);
Case sensitivity isn't the problem but did point me to the right line - there was one place in which I had userID instead of UserID. Now that I fixed the case, the error message complains about UserID.
My "broken" stored procedure also works properly in SQL Server 2008 - this is either a 2000 bug or I'm severely misunderstanding how SQL Server used to work.
Thanks everyone for chiming in!
For anyone searching this in the future, I've added an extremely crude workaround to be 2000-compatible until we update our production version:
DECLARE #workaroundTableName NVARCHAR(256), #workaroundQuery NVARCHAR(2000)
SET #workaroundQuery = 'SELECT [Name]';
DECLARE cur_workaround CURSOR FOR
SELECT COLUMN_NAME FROM [tempdb].INFORMATION_SCHEMA.Columns WHERE TABLE_NAME LIKE '#result%' AND COLUMN_NAME <> 'UserID'
OPEN cur_workaround;
FETCH NEXT FROM cur_workaround INTO #workaroundTableName
WHILE ##FETCH_STATUS = 0
BEGIN
SET #workaroundQuery = #workaroundQuery + ',[' + #workaroundTableName + ']'
FETCH NEXT FROM cur_workaround INTO #workaroundTableName
END
CLOSE cur_workaround;
DEALLOCATE cur_workaround;
SET #workaroundQuery = #workaroundQuery + ' FROM #result ORDER BY Name ASC'
EXEC(#workaroundQuery);
Thanks everyone!
A much easier solution would be to not drop the column, but don't return it in the final select.
There are all sorts of reasons why you shouldn't be returning select * from your procedure anyway.
EDIT: I see now that you have to do it this way because of an unknown number of columns.
Based on the error message, is the database case sensitive, and so there's a difference between userID and UserID?
This works for me:
CREATE TABLE #temp_t
(
myInt int,
myUser varchar(100)
)
INSERT INTO #temp_t(myInt, myUser) VALUES(1, 'Jon1')
INSERT INTO #temp_t(myInt, myUser) VALUES(2, 'Jon2')
INSERT INTO #temp_t(myInt, myUser) VALUES(3, 'Jon3')
INSERT INTO #temp_t(myInt, myUser) VALUES(4, 'Jon4')
ALTER TABLE #temp_t
DROP Column myUser
SELECT * FROM #temp_t
DROP TABLE #temp_t
It says invalid column for you. Did you check the spelling and ensure there even exists that column in your temp table.
You might try wrapping everything preceding the DROP COLUMN in a BEGIN...COMMIT transaction.
At compile time, SQL Server is probably expanding the * into the full list of columns. Thus, at run time, SQL Server executes "SELECT UserID, Name, LastName, FirstName, ..." instead of "SELECT *". Dynamically assembling the final SELECT into a string and then EXECing it at the end of the stored procedure may be the way to go.
I have a table with a view_name field (varchar(256)) and I would like to use that field in an sql query.
Example :
TABLE university_members
id | type | view_name | count
1 | professors | view_professors | 0
2 | students | view_students2 | 0
3 | staff | view_staff4 | 0
And I would like to update all rows with some aggregate calculated on the corresponding view (for instance ..SET count = SELECT count(*) FROM view_professors).
This is probably a newbie question, I'm guessing it's either obviously impossible or trivial. Comments on the design, i.e. the way one handle meta-data here (explicity storing DB object names as strings) would be appreciated. Although I have no control over that design (so I'll have to find out the answer anyway), I'm guessing it's not so clean although some external constraints dictated it so I would really appreciate the community's view on this for my personal benefit.
I use SQL Server 2005 but cross-platform answers are welcome.
To do this you would have to do it as a bit of dynamic SQL, something like this might work, obviously you would need to edit to actually match what you are trying to do.
DECLARE #ViewName VARCHAR(500)
SELECT #ViewName = view_name
FROM University_Members
WHERE Id = 1
DECLARE #SQL VARCHAR(MAX)
SET #SQL = '
UPDATE YOURTABLE
SET YOURVALUE = SELECT COUNT(*) FROM ' + #ViewName + '
WHERE yourCriteria = YourValue'
EXEC(#SQL)
The way I see it, you could generate SQL code in a VARCHAR(MAX) variable and then execute it using EXEC keyword. I don't know of any way to do it directly, as you tried.
Example:
DECLARE #SQL VARCHAR(MAX)
SET #SQL = ''
SELECT #SQL = #SQL + 'UPDATE university_members SET count = (SELECT COUNT(*) FROM ' + view_name + ') WHERE id = ' + id + CHAR(10) + CHAR(13) FROM university_members
EXEC #SQL
Warning! This code is not tested. It's just a hint...
Dynamic SQl is the only way to do this which is why this is a bad design choice. Please read the following article if you must be using dynamic SQl in order to protect your data.
http://www.sommarskog.se/dynamic_sql.html
As HLGEM wrote, the fact that you're being forced to use dynamic SQL is a sign that there is a problem with the design itself. I'll also point out that storing an aggregate in a table like that is most likely another bad design choice.
If you need to determine a value at some point, then do that when you need it. Trying to keep a calculated value like that synchronized with your data is almost always fraught with problems - inaccuracy, extra overhead, etc.
There are very rarely situations where storing a value like that is necessary or gives an advantage and those are typically in very large data warehouses or systems with EXTREMELY high throughput. It's nothing that a school or university is likely to encounter.