How to get all triggering environments from SQL Server database - sql

I am trying to query bamboo's database to get the following information. I want to find out the trigger information, which is what environment is triggered by which branch on which build plan. The problem is that some environments can be triggered by other environments.
So far I have looked up the Bamboo database and I seem to have most if not all the information that I need. What I have done is there is a table in the Bamboo database called dbo.Deployment_Environment which has the fields, EnvironmentID and Triggers_XML_Data which is exactly what I need from it. Then there is another table called the dbo.Deployment_Result which has the EnvironmentID, Deployment_State and the Trigger_Reason.
From these two tables what I think I need to do is this. select all these columns but if the trigger_reason contains environment and also if the triggers_xml_data contains environment, then get that environment from the triggers_xml_Data (this is under the xml node /item/value). Once you get that ID select all the same columns until and repeat the process if environment is the reason for deployment again until it is not. The triggers_xml_data data type is ntext and the trigger_reason is nvarchar.
I am not exactly sure how I can do this in SQL this is what I am trying so far but I can't get the right information:
alter proc dbo.myStoredProc3
as
Declare #EnvironmentID int
select
#EnvironmentID as ENVIRONMENT_ID,
de.ENVIRONMENT_ID, dr.DEPLOYMENT_STATE,
de.TRIGGERS_XML_DATA, dr.TRIGGER_REASON, de.NAME
from
dbo.BUILDRESULTSUMMARY as br,
dbo.DEPLOYMENT_ENVIRONMENT as de,
dbo.DEPLOYMENT_RESULT as dr
where
dr.TRIGGER_REASON like '%environment%'
(
select de2.ENVIRONMENT_ID as 'test'
from dbo.DEPLOYMENT_ENVIRONMENT as de2
)
I know this is not going to get the right information but every time I tried to do case's or if statements in the SQL it would create an error too: How can I do this query correctly? I think that I all the information I need but if not I can add more.
I have also tried to do the following:
While (#Counter <= 5)
Begin
Select de.NAME As 'Deployment Name', dr.TRIGGER_REASON as 'Trigger Reason', dr.DEPLOYMENT_STATE as 'Status'
from dbo.DEPLOYMENT_ENVIRONMENT as de, dbo.DEPLOYMENT_RESULT as dr
Where de.TRIGGERS_XML_DATA like '%environment%' and dr.TRIGGER_REASON like '%environment%'
SET #Counter = #Counter + 1
END
But this will not link the columns that I am getting. If the Trigger_Reason is to do with the environment then in the ntext the it will have a node that has the environmentID, I want to get this and set it as a variable, which I can use to get further columns that link to together e.g. Select name from dbo.Environment where #NewEnvironmentID = de.EnvironmentID.
Edit
Using the following SQL query:
alter proc getEnvironmentTriggers
#EnvironmentID int
as
Select a.NAME, a.TRIGGERS_XML_DATA, a.TRIGGER_REASON, a.ENVIRONMENT_ID
From (Select de.NAME, de.TRIGGERS_XML_DATA, de.ENVIRONMENT_ID, dr.TRIGGER_REASON From dbo.DEPLOYMENT_ENVIRONMENT as de
Inner join dbo.DEPLOYMENT_RESULT as dr
on dr.ENVIRONMENT_ID = de.ENVIRONMENT_ID
Where #EnvironmentID = de.ENVIRONMENT_ID and de.TRIGGERS_XML_DATA like '%Environment%' and dr.TRIGGER_REASON like '%Environment%' ) a
I am getting close to what I need. Doing this and the executing the Stored procedure like this: exec dbo.getEnvironmentTriggers 15892483, this will return in the query inside the ntext l of the TRIGGERS_XML_DATA, will have a value of 18317322, and then using that ID I can get the value of 19234819, that is then the last environment that triggers all of my environments from that ID.
So what I am trying to do is this:
In an automated way get all of these ID's possibly by doing the following.
Exec the stored procedure using all of the ID's (maybe using some sort of for each way).
Search the ntext column in the SQL to get the environment ID (this is in the following node /item/value.
Store that as a variable, then use it to get all the columns from the other stored procedure.
If there is returned columns using the stored procedure, run the procedure again, if not export all of the TRIGGER_XML_DATA columns that were got from stored procedures.
Is there a way that I can do this?

See my answer here - https://answers.atlassian.com/questions/44908050/how-to-get-the-trigger-information-from-the-bamboo-database
Not sure it's possible to do with plain SQL

What about using Bamboo REST API
http://bamboo_host/rest/api/latest/plan.json?expand=plans.plan.actions

Related

INSERT FROM EXISTING SELECT without amending

With GDPR in the UK on the looming horizon and already have a team of 15 users creating spurious SELECT statements (in excess of 2,000) across 15 differing databases I need to be able to create a method to capture an already created SELECT statement and be able to assign surrogate keys/data WITHOUT rewriting every procedure we already have.
There will be a need to run the original team members script as normal and there will be requirements to pseudo the values.
My current thinking is to create a stored procedure along the lines of:
CREATE PROC Pseudo (#query NVARCHAR(MAX))
INSERT INTO #TEMP FROM #query
Do something with the data via a mapping table of real and surrogate/pseudo data.
UPDATE #TEMP
SET FNAME = (SELECT Pseudo_FNAME FROM PseudoTable PT WHERE #TEMP.FNAME = PT.FNAME)
SELECT * FROM #TEMP
So that team members can run their normal SELECT statements and get pseudo data simply by using:
EXEC Pseudo (SELECT FNAME FROM CUSTOMERS)
The problem I'm having is you can't use:
INSERT INTO #TEMP FROM #query
So I tried via CTE:
WITH TEMP AS (#query)
..but I can't use that either.
Surely there's a way of capturing the recordset from an existing select that I can pull into a table to amend it or capture the SELECT statement; without having to amend the original script. Please bear in mind that each SELECT statement will be unique so I can't write COLUMN or VALUES etc.
Does any anyone have any ideas or a working example(s) on how to best tackle this?
There are other lengthy methods I could externally do to carry this out but I'm trying to resolve this within SQL if possible.
So after a bit of deliberation I resolved it.
I passed the Original SELECT SQL to SP that used some SQL Injection, which when executed INSERTed data. I then Updated from that dataset.
The end result was "EXEC Pseudo(' Orginal SQL ;')
I will have to set some basic rules around certain columns for now as a short term fix..but at least users can create NonPseudo and Pseudo data as required without masses of reworking :)

Query a database based on result of query from another database

I am using SSIS in VS 2013.
I need to get a list of IDs from 1 database, and with that list of IDs, I want to query another database, ie SELECT ... from MySecondDB WHERE ID IN ({list of IDs from MyFirstDB}).
There is 3 Methods to achieve this:
1st method - Using Lookup Transformation
First you have to add a Lookup Transformation like #TheEsisia answered but there are more requirements:
In the Lookup you Have to write the query that contains the ID list (ex: SELECT ID From MyFirstDB WHERE ...)
At least you have to select one column from the lookup table
These will not filter rows , but this will add values from the second table
To filter rows WHERE ID IN ({list of IDs from MyFirstDB}) you have to do some work in the look up error output Error case there are 2 ways:
set Error handling to Ignore Row so the added columns (from lookup) values will be null , so you have to add a Conditional split that filter rows having values equal NULL.
Assuming that you have chosen col1 as lookup column so you have to use a similar expression
ISNULL([col1]) == False
Or you can set Error handling to Redirect Row, so all rows will be sent to the error output row, which may not be used, so data will be filtered
The disadvantage of this method is that all data is loaded and filtered during execution.
Also if working on network filtering is done on local machine (2nd method on server) after all data is loaded is memory.
2nd method - Using Script Task
To avoid loading all data, you can do a workaround, You can achieve this using a Script Task: (answer writen in VB.NET)
Assuming that the connection manager name is TestAdo and "Select [ID] FROM dbo.MyTable" is the query to get the list of id's , and User::MyVariableList is the variable you want to store the list of id's
Note: This code will read the connection from the connection manager
Public Sub Main()
Dim lst As New Collections.Generic.List(Of String)
Dim myADONETConnection As SqlClient.SqlConnection
myADONETConnection = _
DirectCast(Dts.Connections("TestAdo").AcquireConnection(Dts.Transaction), _
SqlClient.SqlConnection)
If myADONETConnection.State = ConnectionState.Closed Then
myADONETConnection.Open()
End If
Dim myADONETCommand As New SqlClient.SqlCommand("Select [ID] FROM dbo.MyTable", myADONETConnection)
Dim dr As SqlClient.SqlDataReader
dr = myADONETCommand.ExecuteReader
While dr.Read
lst.Add(dr(0).ToString)
End While
Dts.Variables.Item("User::MyVariableList").Value = "SELECT ... FROM ... WHERE ID IN(" & String.Join(",", lst) & ")"
Dts.TaskResult = ScriptResults.Success
End Sub
And the User::MyVariableList should be used as source (Sql command in a variable)
3rd method - Using Execute Sql Task
Similar to the second method but this will build the IN clause using an Execute SQL Task then using the whole query as OLEDB Source,
Just add an Execute SQL Task before the DataFlow Task
Set ResultSet property to single
Select User::MyVariableList as Result Set
Use the following SQL command
DECLARE #str AS VARCHAR(4000)
SET #str = ''
SELECT #str = #str + CAST([ID] AS VARCHAR(255)) + ','
FROM dbo.MyTable
SET #str = 'SELECT * FROM MySecondDB WHERE ID IN (' + SUBSTRING(#str,1,LEN(#str) - 1) + ')'
SELECT #str
If the column has string data type you should add quotation before and after values as below:
SELECT #str = #str + '''' + CAST([ID] AS VARCHAR(255)) + ''','
FROM dbo.MyTable
Make sure that you have set the DataFlow Task Delay Validation property to True
This is a classic case for using LookUp Transformation. First, use a OLE DB Source to get data from the first database. Then, use a LookUp Transformation to filter this data-set based on the ID values from the second data-set. Here is the steps for using a LookUp Transformation:
In the General tab, select Full Cash, OLE DB Connection Manager and Redirect rows to no match output as shown in the following picture. Notice that using Full Cash provides great performance for your package.
General Setting
In the Connection tab, use OLE DB Connection Manager to connect to your second server. Then, you can either directly select the data-set with ID values or (as is shown in the picture below) you can use SQL code to select the IDs from the filtering data-set.
Connection:
Go to Columns tab and select ID columns from the both datasets. For each record from your first data-set, it will check to see if its ID is in the Available LookUp Column. If it is, it will go to the Matching output, else to No Matching output.
Match ID columns:
Click on OK to close the LookUp. Then you need to select the LookUp Match Output.
Match Output:
The "best" answer depends on data volumes and source systems involved.
Many of the other answers propose building out a list of values based on clever concatenation within SQL Server. That doesn't work so well if the referenced system is Oracle, MySQL, DB2, Informix, PostGres, etc. There may be an equivalent concept but there might not be.
For best performance, you need to filter against the second db before any of those rows ever hit the data flow. That means adding a filtering condition, as the others have suggested, to your source query. The challenge with this approach is that your query is going to be limited by some practical bounds that I don't remember. Ten, one hundred, a thousand values in your where clause is probably fine. A lakh, a million - probably not so much.
In the cases where you have large volumes of values to filter against the source table, it can make sense to create a table on that server and truncate and reload that table (execute sql task + data flow). This allows you to have all of the data local and then you can index the filter table and let the database engine do what it's really good at.
But, you say the source database is some custom solution that you can't make tables in. You can look at the above approach with temporary tables and within SSIS you just need to mark the connection as singleton/persisted (TODO: look this up). I don't much care for temporary tables with SSIS as debugging them is a nightmare I'd not wish upon my mortal enemy.
If you're still reading, we've identified why filtering in the source system might not be "doable", even if it will provide the best performance.
Now we're stuck with purely SSIS solutions. To get the best performance, do not select the table name in the drop down - unless you absolutely need every column. Also, pay attention to your data types. Pulling LOB (XML, text, image (n)varchar(max), varbinary(max)) into the dataflow is a recipe for bad performance.
The default suggestion is to use a Lookup Component to filter the data within the data flow. As long as your source system supports and OLE DB provider (or you can coerce the data into a Cache Connection Manager)
If you can't use a Lookup component for some reason, then you can explicitly sort your data in your source systems, mark your source components as such, and then use a Merge Join of type Inner Join in the data flow to only bring in matched data.
However, be aware that sorts in source systems are going to be sorted according to native rules. I ran into a situation where SQL Server was sorting based on the default ASCII sort and my DB2 instance, running on zOS, provided an EBCDIC sort. Which was great when my domain was only integers but went to hell in a handbasket when the keys became alphanumeric (AAA, A2B, and AZZ will sort differently based on this).
Finally, excluding the final paragraph, the above assumes you have integers. If you're performing string matching, you get an extra level of ugliness because different components may or may not perform a case sensitive match (sorting with case sensitive systems can also be a factor).
I would first create a String variable e.g. SQL_Select, at the Scope of the Package. Then I would assign that a value using an Execute SQL Task against the 1st database. The ResultSet property on the General page should be set to Single row. Add an entry to the Result Set tab to assign it to your Variable.
The SQL Statement used needs to be designed to return the required SELECT statement for your 2nd database, in a single row of text. An example is shown below:
SELECT
'SELECT * from MySecondDB WHERE ID IN ( '
+ STUFF ( (
SELECT TOP 5
' , ''' + [name] + ''''
FROM dbo.spt_values
FOR XML PATH(''), TYPE).value('(./text())[1]', 'VARCHAR(4000)'
) , 1 , 3, '' )
+ ' ) '
AS SQL_Select
Remove the TOP 5 and replace [name] and dbo.spt_values with your column and table names.
Then you can use the variable SQL_Select in a downstream task e.g. an OLE DB Source against database 2. OLE DB Sources and OLE DB Command Tasks both let you specify a Variable as the SQL Statement source.
You could add a LinkedServer between the two servers. The SQL command would be something like this:
EXEC sp_addlinkedserver #server='SRV' --or any name you want
EXEC sp_addlinkedsrvlogin 'SRV', 'false', null, 'username', 'password'
SELECT * FROM SRV.CatalogNameInSecondDB.dbo.SecondDBTableName s
INNER JOIN FirstDBTableName f on s.ID = f.ID
WHERE f.ID IN (list of values)
EXEC sp_dropserver 'SRV', 'droplogins'

Find out all useful columns in a table in sql server

I have a table which has 50+ columns but only few columns are getting used. that means when any stored procedure uses that table it only refers 4-5 columns in select/where statements . rest of columns are not getting used . i just want to list down those columns that are actually getting used. one way is finding out the dependencies of a table and then go through every SP and find out which columns are getting used . but in that case i have around 30+ Sp. is there any efficient way to do it.
To use multiple columns in a procedure, you can use a code like below
create procedure sp_sample
#column_names varchar(200)
as
if #column_names='' or #column_nams is null
set #column_names='*'
exec ('select '+#column_name +' from table')
Here are some examples :
exec sp_sample #columnname='id,name'
or
exec sp_sample #columnname='id,name,telphone'
Try this:
select name from syscomments c
join sysobjects o on c.id = o.id
where TEXT like '%table_name%' and TEXT like '%column_name%'
In table_name give you table name, in column_name give the column for which you want to chck the procedure dependencies.You will get the stored procedure names as output
If you import your database as a database project using the SQL Server Data Tools, you will be able to find all references to a table or column using the "Find All References" context command. What makes this particularly useful is the accuracy: it will even find instances of SELECT * that don't mention the column explicitly, but implicitly refer to it anyway. It will also not be confused by tables or columns with similar names (finding particular instances of ID is otherwise rather problematic).
If all you want to know if a column is referenced at all, you can simply delete it and see if any "unresolved reference" errors appear in the error list -- if yes, then the column is used somewhere.

Selecting from the result set of a DB2 stored procedure

I have a stored procedure which returns multiple records (the SP cannot be changed, I need to work with what I have). I'd like to do a DB2 select statement from Shell script that selects one record based on a combination of column data like the following:
select a.description_column from (call my_stored_proc) a where a.name_column='name_filter' and a.value_column='value_filter';
The columns description_column, name_column and value_column exist in the result set of the SP. I get a SQLCODE=-727, SQLSTATE=56098, SQLERRMC=2 error. As I need to sort it out from a Shell script and I only have read access to the DB, I can't create additional tables for this.
You can't select from SP.
But you can from a Table-Function.
SELECT ... FROM TABLE(<table-function(param1, param2, ..., paramN))) as t WHERE ....
So, easiest way is to ask the DBA to create a table-function based in the source SP.
Good luck

SSIS - Set multiple package variables from query results

I am working on an SSIS 2010 package that makes use of multiple package variables that I want to populate with the results of a SQL query. We have an app settings table that is set up as such:
[ID] [settingName] [settingValue]
1 appName test app
2 scheduled yes
Within the package, I have variables set up for appName and scheduled but I'm having a hard time figuring out the best way to load them from the database. The query is running successfully, just a simple select statement, but I'm not sure what to do next. I've found suggestions saying I should parse it out in a script task but it seems like there should be a much easier way to do this.
Your query needs to return a single row that contains the values for each variable in separate columns:
Then map the result set columns to your variables:
In short, you need to pivot your query.
Edit: Like this. (minus the table variable; use your actual table)
declare #pivot table (settingname varchar(20), settingvalue varchar(20))
insert into #pivot
values
('appname','test app'),
('scheduled','yes')
SELECT *
FROM (
SELECT settingname, settingvalue from #pivot
) as source
pivot
(
max(settingvalue)
for settingname in ([appname],[scheduled])
)
as pvt