I'm trying to do an update query in which a single row the table is updated and, if nothing has matched and updated, a new row is inserted. In each case, I need the query to return the ID of the inserted row.
The issue I'm having is this query is returning 2 separate results when the insert case is reached, one for each output (the first empty, the second containing the ID). I'm running this query using SQL Alchemy on python and I'm only able to see the first result, which is empty.
UPDATE [Rights]
SET accessLevel = :access_level
OUTPUT inserted.rightsID
WHERE principal = :principal and [function] = :function
IF ##ROWCOUNT = 0
INSERT INTO Rights(principal, [function], accessLevel)
OUTPUT inserted.rightsID
VALUES(:principal, :function, :access_level)
And I'm calling it like so:
inserted_right_id = session.execute(sql_rights_update, right).fetchall()
Can anyone recommend a way of changing the query so that I can still use the UPSERT method, but only receive one of the IDs? I was considering storing the OUTPUT values into a table and returning that, or wrapping the whole thing in a select but hopefully there's something more elegant out there.
Thanks a million.
Feeling quite dumb. I simply added a
IF EXISTS(SELECT * FROM Rights WHERE principal = :principal and [function] = :function)
UPDATE ...
ELSE
INSERT ...
Even after the below configuration , im not able to assign a query result to a variable
In the first form you have specified the ResultSet type as single row, and in the SQL Command you have wrote a SELECT query that may returns multiple rows.
You have to change the SQL Command, and add a TOP 1 keyword
SELECT TOP 1 ... FROM ... WHERE ...
Also if it still not working, in the Result Set Yab, try replacing the columns names with relevant indexes.
MailAddressFrom >> 0
MailAddressTo >> 1
Subject >> 2
If you are looking to store all rows returned by a SELECT Query, you should select FullResultSet option and store the result inside an Object variable:
SSIS Basics: Using the Execute SQL Task to Generate Result Sets
I am using SSIS in VS 2013.
I need to get a list of IDs from 1 database, and with that list of IDs, I want to query another database, ie SELECT ... from MySecondDB WHERE ID IN ({list of IDs from MyFirstDB}).
There is 3 Methods to achieve this:
1st method - Using Lookup Transformation
First you have to add a Lookup Transformation like #TheEsisia answered but there are more requirements:
In the Lookup you Have to write the query that contains the ID list (ex: SELECT ID From MyFirstDB WHERE ...)
At least you have to select one column from the lookup table
These will not filter rows , but this will add values from the second table
To filter rows WHERE ID IN ({list of IDs from MyFirstDB}) you have to do some work in the look up error output Error case there are 2 ways:
set Error handling to Ignore Row so the added columns (from lookup) values will be null , so you have to add a Conditional split that filter rows having values equal NULL.
Assuming that you have chosen col1 as lookup column so you have to use a similar expression
ISNULL([col1]) == False
Or you can set Error handling to Redirect Row, so all rows will be sent to the error output row, which may not be used, so data will be filtered
The disadvantage of this method is that all data is loaded and filtered during execution.
Also if working on network filtering is done on local machine (2nd method on server) after all data is loaded is memory.
2nd method - Using Script Task
To avoid loading all data, you can do a workaround, You can achieve this using a Script Task: (answer writen in VB.NET)
Assuming that the connection manager name is TestAdo and "Select [ID] FROM dbo.MyTable" is the query to get the list of id's , and User::MyVariableList is the variable you want to store the list of id's
Note: This code will read the connection from the connection manager
Public Sub Main()
Dim lst As New Collections.Generic.List(Of String)
Dim myADONETConnection As SqlClient.SqlConnection
myADONETConnection = _
DirectCast(Dts.Connections("TestAdo").AcquireConnection(Dts.Transaction), _
SqlClient.SqlConnection)
If myADONETConnection.State = ConnectionState.Closed Then
myADONETConnection.Open()
End If
Dim myADONETCommand As New SqlClient.SqlCommand("Select [ID] FROM dbo.MyTable", myADONETConnection)
Dim dr As SqlClient.SqlDataReader
dr = myADONETCommand.ExecuteReader
While dr.Read
lst.Add(dr(0).ToString)
End While
Dts.Variables.Item("User::MyVariableList").Value = "SELECT ... FROM ... WHERE ID IN(" & String.Join(",", lst) & ")"
Dts.TaskResult = ScriptResults.Success
End Sub
And the User::MyVariableList should be used as source (Sql command in a variable)
3rd method - Using Execute Sql Task
Similar to the second method but this will build the IN clause using an Execute SQL Task then using the whole query as OLEDB Source,
Just add an Execute SQL Task before the DataFlow Task
Set ResultSet property to single
Select User::MyVariableList as Result Set
Use the following SQL command
DECLARE #str AS VARCHAR(4000)
SET #str = ''
SELECT #str = #str + CAST([ID] AS VARCHAR(255)) + ','
FROM dbo.MyTable
SET #str = 'SELECT * FROM MySecondDB WHERE ID IN (' + SUBSTRING(#str,1,LEN(#str) - 1) + ')'
SELECT #str
If the column has string data type you should add quotation before and after values as below:
SELECT #str = #str + '''' + CAST([ID] AS VARCHAR(255)) + ''','
FROM dbo.MyTable
Make sure that you have set the DataFlow Task Delay Validation property to True
This is a classic case for using LookUp Transformation. First, use a OLE DB Source to get data from the first database. Then, use a LookUp Transformation to filter this data-set based on the ID values from the second data-set. Here is the steps for using a LookUp Transformation:
In the General tab, select Full Cash, OLE DB Connection Manager and Redirect rows to no match output as shown in the following picture. Notice that using Full Cash provides great performance for your package.
General Setting
In the Connection tab, use OLE DB Connection Manager to connect to your second server. Then, you can either directly select the data-set with ID values or (as is shown in the picture below) you can use SQL code to select the IDs from the filtering data-set.
Connection:
Go to Columns tab and select ID columns from the both datasets. For each record from your first data-set, it will check to see if its ID is in the Available LookUp Column. If it is, it will go to the Matching output, else to No Matching output.
Match ID columns:
Click on OK to close the LookUp. Then you need to select the LookUp Match Output.
Match Output:
The "best" answer depends on data volumes and source systems involved.
Many of the other answers propose building out a list of values based on clever concatenation within SQL Server. That doesn't work so well if the referenced system is Oracle, MySQL, DB2, Informix, PostGres, etc. There may be an equivalent concept but there might not be.
For best performance, you need to filter against the second db before any of those rows ever hit the data flow. That means adding a filtering condition, as the others have suggested, to your source query. The challenge with this approach is that your query is going to be limited by some practical bounds that I don't remember. Ten, one hundred, a thousand values in your where clause is probably fine. A lakh, a million - probably not so much.
In the cases where you have large volumes of values to filter against the source table, it can make sense to create a table on that server and truncate and reload that table (execute sql task + data flow). This allows you to have all of the data local and then you can index the filter table and let the database engine do what it's really good at.
But, you say the source database is some custom solution that you can't make tables in. You can look at the above approach with temporary tables and within SSIS you just need to mark the connection as singleton/persisted (TODO: look this up). I don't much care for temporary tables with SSIS as debugging them is a nightmare I'd not wish upon my mortal enemy.
If you're still reading, we've identified why filtering in the source system might not be "doable", even if it will provide the best performance.
Now we're stuck with purely SSIS solutions. To get the best performance, do not select the table name in the drop down - unless you absolutely need every column. Also, pay attention to your data types. Pulling LOB (XML, text, image (n)varchar(max), varbinary(max)) into the dataflow is a recipe for bad performance.
The default suggestion is to use a Lookup Component to filter the data within the data flow. As long as your source system supports and OLE DB provider (or you can coerce the data into a Cache Connection Manager)
If you can't use a Lookup component for some reason, then you can explicitly sort your data in your source systems, mark your source components as such, and then use a Merge Join of type Inner Join in the data flow to only bring in matched data.
However, be aware that sorts in source systems are going to be sorted according to native rules. I ran into a situation where SQL Server was sorting based on the default ASCII sort and my DB2 instance, running on zOS, provided an EBCDIC sort. Which was great when my domain was only integers but went to hell in a handbasket when the keys became alphanumeric (AAA, A2B, and AZZ will sort differently based on this).
Finally, excluding the final paragraph, the above assumes you have integers. If you're performing string matching, you get an extra level of ugliness because different components may or may not perform a case sensitive match (sorting with case sensitive systems can also be a factor).
I would first create a String variable e.g. SQL_Select, at the Scope of the Package. Then I would assign that a value using an Execute SQL Task against the 1st database. The ResultSet property on the General page should be set to Single row. Add an entry to the Result Set tab to assign it to your Variable.
The SQL Statement used needs to be designed to return the required SELECT statement for your 2nd database, in a single row of text. An example is shown below:
SELECT
'SELECT * from MySecondDB WHERE ID IN ( '
+ STUFF ( (
SELECT TOP 5
' , ''' + [name] + ''''
FROM dbo.spt_values
FOR XML PATH(''), TYPE).value('(./text())[1]', 'VARCHAR(4000)'
) , 1 , 3, '' )
+ ' ) '
AS SQL_Select
Remove the TOP 5 and replace [name] and dbo.spt_values with your column and table names.
Then you can use the variable SQL_Select in a downstream task e.g. an OLE DB Source against database 2. OLE DB Sources and OLE DB Command Tasks both let you specify a Variable as the SQL Statement source.
You could add a LinkedServer between the two servers. The SQL command would be something like this:
EXEC sp_addlinkedserver #server='SRV' --or any name you want
EXEC sp_addlinkedsrvlogin 'SRV', 'false', null, 'username', 'password'
SELECT * FROM SRV.CatalogNameInSecondDB.dbo.SecondDBTableName s
INNER JOIN FirstDBTableName f on s.ID = f.ID
WHERE f.ID IN (list of values)
EXEC sp_dropserver 'SRV', 'droplogins'
I want to create an interface between 2 databases on SQL Server 2008+ to copy several similar named tables into one.
I have n tables that all have the same naming convention, for example:
SalesInvoicePlanning2014ver1
SalesInvoicePlanning2015ver1
SalesInvoicePlanning2015ver2
etc.
The numbers can vary and do not have a set start (or end), but are always of the "int"-Datatype.
I also have a table "tabledir" that contains all table names as list. (one field) There are a total of 30-40 entries in that list with (for me) undesired entries. In the above example I would need 3 of the 30 tables.
The plan is to use a loop container to
select Top 1([name]) from [tabledir] where name like 'SalesinvoicePlanning%'
and then use the result as variable in the following SSIS Data transfer task:
Select * from [variable]
However, I'm stuck with the SQL statement to give me the desired tablename on each iteration.
Performance is not really an issue. Any advice? Am I wrong trying to use a loop-container?
You can follow below steps -
Step 1 - You can first create SQL task to get all table names into one variable lets say, TableNames of type Object(recordset) using you query.
e.g. select ([name]) as TableName from [tabledir] where name like 'SalesinvoicePlanning%'
Step 2 - Add foreach loop container to iterate over this variable TableNames to take single table name into new variable current_table and add data flow into the container to import data to destination table. Your source query will be expression like -
Select column_names from current_table
I have table called Table1 with columns, col1 and col2 with col1 having weblinks for the report and col2 the name of the report. Now, i have a package with a variables var1 and var2 which should get the col1 and col2 values respectively from table1 and send it through an email. if the weblink gets updated in the table, package should send the updated link. i know the reverse way of it but trying to do somethig like this.
Appreciate any help from you guys.
Thanks
There are a few ways you can do this. You can try one of the following based on whether you are processing one row or many rows.
One-Row Solution
If you are running the SSIS package so that it will process one and only one row from the table, you can use the Execute SQL Task to get the values from your table. Begin by adding your two variables
In the task, specify the connection to your SQL Server and enter a SQL statement like this.
SELECT col1, col2
FROM Table1
WHERE <your condition to retrieve one row>
Also, on the General tab, set the ResultSet value to Single row.
On the Result Set tab, add two elements. The first row should be Result Name of 0, Variable Name of Var1. The second row will be Result Name 1, Variable Name Var2.
When the task runs, the results of the SQL statement will be stored to the variables.
Multi-Row Solution
If the SSIS package must return multiple rows, then you need a different approach. First, you will need to create another variable that is an Object data type. The SQL results will be saved to this variable.
You will have an Execute SQL Task that is similar to the one above. However, your SQL statement should return all rows to be processed. The ResultSet property should be set to Full result set. On the Result Set tab, there should be one result set, with 0 as Result Name and your new object variable as the Variable Name. Close the Execute SQL Task editor.
Add a Foreach Loop Container. Edit the container and select the Collection tab. Set the Enumerator property to Foreach ADO Enumerator. Under the Enumerator configuration, set your object variable as the ADO object source variable.
On the Variable Mappings tab, add to rows. The first row will have a Variable Var1 and Index 0, and the second row will have Variable Var2 and Index 1. The Foreach Loop Container will process each for in the result set and assign the column values to your variable. You can add any tasks in the container and they will run once for each row in the result set.