I am creating an MSSQL2008 SSIS package to generate and email reports from database tables. It works perfectly on a single database. The client is running 3 different databases used by 3 different divisions. The database structure is exactly the same. All three databases are located on the same server, same security / credentials are used.
I created a "For Each Loop Container" in my SSIS package that loops through the list of 3 items and populates it into a variable. How do I now take that and pass it to the "Execute SQL Task" to run three times (once for each database)?
Thank you for your time!
It was a lot easier than I expected.
I went to Properties of the "Execute SQL Task" and under "Expressions" for "Connection" I specified #varDBName, which was the variable I populated in the outer "for each" loop. I also needed to set "DelayValidation" property to "True" so it's only evaluated during run-time.
I hope this helps somebody else.
Related
I have 11 databases in which I'm having tables contains User Details i.e. all employee details. There I have a column "Status"(which is 1 for Active and 0 for Inactive). I have a regular tasks for updating "Status" column value 0 or 1 for mentioned employees and for that, I have to go through all the databases then User table then I have to update. The same task i have to do for all the database and it consumes a lot of time.
If I will get a short Query or Procedure that I have to run once and will do all updation at once then, it would be a great help.
I see a couple of possible options.
You could build an SSIS package to connect to each database and do the necessary updates provided the criteria of which employees to update and what to update them to could be found within the database or some external source such as a text file.
Alternatively, you could use SQLCMD mode in SQL Server Management Studio and then within your SQL script use CONNECT command to switch to each server and database something like this...
:CONNECT Server1
USE Database1
--put your update SQL script
:CONNECT Server2
USE Database2
--put your update SQL script
...
These links provide some further information on using SQLCMD mode...
Connecting to multiple servers in a Query Window using SQLCMD
SQL Server SQLCMD Basics
Noel
As you mentioned, you have 11 databases.
Problem : First, you are using very bad approach for database design,
What really Happens : When you are using multiple databases and you need to check in every database, then the server needs to connect to different database again and again, which takes very more time compared to switching into the tables, because of connection handling.
Solution : In your case, you have only one option to connect different databases in loops and then run the query in the loop for every DB.
Suggestion : you should keep all the data in the same database, you can use an extra column in tables to keep track your data to different entities.
In a SSIS ETL, I have a query that I need to run on a server/db that does not allow us to create stored procedures.
I would normally use the stored procedure in my variable as the source for my OLE DB source:
However, since we can't put the stored procedure on this server, I was going to store the code for the stored procedure into a variable by executing a SQL statement, retrieving the text from our home database, then use the text stored in this variable as the SQL command for the source:
This way, I can still remotely change the SSIS OLE DB Source object WHERE clause (as long as I don't change the SELECT portion).
I can't imagine that this is very common, so I wanted to get some opinions - is there a better way to do this? I don't want to put all of the code for this SP into the OLE DB Source editor directly because we can't afford to redeploy in case of a WHERE clause update.
You've got the part down that many folks don't do and that's using Variables to drive your package execution. You are further correct in that you can't exactly swap out your columns. To be pedantic, which I am, you can completely change out the query as long as the same metadata is presented.
So, then this question becomes how best to accomplish allowing a package to have a query's filter driven by an external force. Factoring in maintainability, ease of debugging, etc.
My gut reaction is 3 Variables
QueryBase: String. Hardcoded. SELECT * FROM MyTable except of course I'd enumerate my columns
Query: String. EvaluateAsExpression = True Expression: #[User::QueryBase] + #[User::QueryFilter]
QueryFilter: String
So, we use Query in the OLE DB Source much as you have your longer variable name in there. The only downside to this approach, pre SSIS-2012 is the limitation on string length in an expression. It was ... 4k I believe. If you assign a value of 5k characters, it's fine. It's just in the expression language, adding two strings together can't exceed 4k.
I didn't specify what QueryFilter is going to have in it or the magic to get it there. That, I would base on the bigger picture of your environment, usage, etc. but the general concept is that it will eventually turn into WHERE Condition1 IS NOT NULL but maybe in a full reload situation, it becomes an empty string.
So, what are our options for changing the value of QueryFilter
/SET is an optional parameter passed to the invoking process (dtexec.exe) that makes SSIS packages go. If you have a very limited set of choices and aren't interested in building additional infrastructure out to support the parameters, just hard code some examples. Approximately dtexec /file p1.dtsx /set \Package.Variables[User::QueryFilter].Properties[Value];" WHERE Condition1 IS NOT NULL" Save it into .bat files, different sql agent jobs, whatever. Click and run and you're done.
Configuration approach. SSIS offers native ability to use configurations from a SQL Server table, XML, Registry, Parent Package and Environment Variable for 2005 to current edition. The only downside to this approach is that it would not support concurrent execution with different parameters like the first would.
Environment approach. 2012 and 2014, with their new Project Deployment Model, give us the concept of Environments within the SSISDB catalog which is similar to configuration with a SQL Server table but it is done after development is complete and the packages are deployed. It's rather nice as it builds out a history of values used so if someone asks why is the data all wrong, you can write a query to pull back the parameters used and Oh look someone used the initial load filter instead of the daily. Whoopsidaisy. Same concern over concurrent execution and changing values.
Table driven approach. Instead of using the Configuration with SQL Server table backing, you roll your own table and then add into your package an Execute SQL Task to retrieve the filter, Single Row, into our QueryFilter Variable.
Script Task. Use whatever floats your boat to determine what the filter should be.
Message Queue. They have built in a Message Queue Task and might be of use here if you're already doing it. Otherwise, too much effort to manage
I have a question regarding ssis packages. I have an ssis package with OnError, OnPreExecute and OnPostExecute. In these event handlers SQL task that perform different task and update different tables. My question is this: there are some system variables that I make use of like the SourceName and SourceDescriptin (which is the current sql task's name and description). I notice there isn't any variables for the connection (Server name, Database name) of the "Source" i.e step. Is there any way to get the database name and server name that Source/Step used? Any help will be much appreciated, thanking you in advance.
I dont think, we will have a system variable carrying the Server name and Db name. In an SSIS package, any number of connections can be created in the connection manager. So making them all available in a system variable will be not possible. I think such a thing need to be managed by developer itself. May be these steps will hep you.
1. Add a parameter (1 parameter per Server).
This parameter will be available through out the package.
2. When the package is running , if required give value to the parameter from SSIS catalog (through SSIS SPs)
For the server name, you can use the system variable called MachineName. It will give you the name of the machine the package is running on. As for the DB, you will have to capture it using a Execute SQL Task and store it within a variable of your choice.
Say I have a stored procedure, ProcA. It calls ProcB some of the time, ProcC all of the time and ProcD if it hits an error. ProcB calls ProcE every time. ProcC calls ProcF if it hits an error.
How could I automate getting the text of ProcA along with the text of all the procs that it calls and regress all the way down the tree (A down to F)? This would help a lot with finding errors in complex sql procs.
My first thought here is get the text of ProcA, regex through it and find any calls to other procs, wash rinse repeat, at the end spit out a text (file or to UI) that looks like this:
ProcA definition
...
ProcB definition
...
...
ProcF definition
But I'm open to suggestion, perhaps there's an easier way. If there's a tool that does this already lemme know. I have no idea what to put into google on this one.
In SQL Server you have sys.sysdepends and sys.syscomments.
INFORMATION_SCHEMA.ROUTINES can be useful in the general SQL case, but it has a limit to the size of the text returned on SQL Server, so I avoid it.
this is highly database dependent, because the best way (if possible) would be to query the database's internal tables.
If you're using MS SQL in SQL 2000 Enterprise Manager right click on your stored procedure > All Tasks > Display Dependencies. In SQL 2005 Management Studio, right click on your stored procedure > View Dependencies.
It won't show you any code for the other objects that this proc id dependent on, but it will list the objects which you can then "wash, rinse, repeat"
i want to extract specific database tables & stored procedures into one master script. Do you know any software that can help me do this faster? I've tried using the SQL Database publishing tool, but it's not that efficient since its gathering tables that I didn't select.
In SQL Server 2005, right click on the database, then select Tasks, and then select Generate Scripts.
Generating SQL Scripts in SQL Server 2005
As mentioned in that link, I'm fairly sure you have to generate the DROP and CREATE statements separately.
Try DBSourceTools. http://dbsourcetools.codeplex.com
Its open source, and specifically designed to script databases - tables, views, procs to disk.
It also allows you to select which tables, views, db-objects to script.
I use Redgate SQL compare for this (by comparing to an empty DB), as well as for doing upgrades between all my DB versions (I save a copy of the DB for each released version, and then just do a compare between current and previous to get a change script for that version).
I have found the "Generate Scripts" does a bad job in some cases with dependencies - eg, it will try to create a stored procedure that uses a table before the table is created, causing the script to fail. I'll accept I'm possibly using it wrong, but SQL Compare "just works". The scripts it generates are also enclosed in a transaction -- so if something fails, the whole change is rolled back. You don't end up with a half-populated or half-upgraded database.
Downside is that this is a commercial tool, but IMHO worth the money.