I have been scouring the internet for a clear answer to this question and I haven't been able to find one.
We currently have a SQL configuration enabled across packages which stores everything in a database, (dev or prod). It is a tricky process to execute packages, figuring out which configuration the packages are mapped to, and having to change them during testing. My hope is that we can use an environment variable to always use a specific connection string to the specific sql server configuration we want to use based on the machine.
I have seen ways to set up variables with the connection string hard coded as an environment variable with an additional SQL server configuration after to manage this. I have also seen ways to manipulate the connection string using XML and calling that from a variable, or from the package itself.
Both of these solutions work and very well may be viable solutions.
My question that I'm really trying to answer is; Is it possible to develop the entire SQL Server configuration connection through the environment variable? As in combining the connection string with the configuration string so that the one environment variable connects straight to the package configuration? Has anyone tried this? Does it make sense as a solution to the problem?
Yes, we have done this and it works fine. Each package has two configurations: the first is an environment variable and the second is a SQL Server table. The environment variable is the connection string for a database containing a table with all the configurations and it is applied to a connection manager. The SQL Server configuration uses that connection manager to connect to the configuration database and retrieve the individual settings.
Because configurations are applied in order, the connection string for the connection manager is set before the SQL Server configuration tries to use it. In summary:
Create your package
Add a connection manager called "Configuration Database" or whatever
Add a SQL Server configuration to the package, use the "Configuration Database" as the Connection, and choose the settings to be stored in the table
Create an environment variable such as SSIS_PKG_CONF with the connection string to be used for the "Configuration Database" connection manager
Add an environment variable configuration to the package, use the SSIS_PKG_CONF variable, and assign it to the connection string for the "Configuration Database" connection manager
Now when you run the package, the configuration proceeds as follows:
The package processes the environment variable configuration
It assigns the value of the environment variable to the "Configuration Database" connection manager's connection string
The package processes the SQL Server configuration
It uses the "Configuration Database" connection manager to retrieve individual settings from a database table
This has worked well for us because we can change the connection string to point at a different source database for the configuration, without changing anything inside the package or the launcher (i.e. the SQL Agent job or script that runs the packages). There are other ways to do it and some people feel strongly that .dtsConfig files are the better option, but we found it easier to manage environment variables than config files
But in the end managing package configurations is a preference and a lot depends on what is easier in your environment with the tools and practices you already have.
Related
Setup: A pretty standard data export SSIS package (SQL Server 2016 compatible), created in VS2019/Data Tools and deployed using the SSIS Project Deployment model to the Integration Services Catalog of a SQL Server 2016 instance. The package creates files in a network folder before sending the file out via FTP and putting a copy of the file in a Sent folder.
The project requirements include having the package running on a schedule using "default" parameter values, as well as allowing users to manually run the package using "non-default" parameter values from within a stand-alone application.
Current behavior: the package behaves correctly when run from a SQL Server Agent Job that is configured with a SQL proxy and credentials mapped to a domain login with the proper permissions for the network folder.
Problem: the Data Flow task fails to create the file with a "Cannot open the datafile" error when running the package directly using any of the following methods (even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job):
Using SSMS to right-click on the package and selecting Execute
Using the DTEXEC SQL utility
Using the SSISDB.catalog.start_execution SQL Server stored procedure
As far as I'm aware, these are the only methods capable of starting a SSIS package and changing the package's parameter values. I either need to get one of the latter 2 methods to work, find another option that allows for changing the parameter values while launching the package, or use one of 2 techniques I'm aware of (detailed below) that would add yet another failure point to the process as well as other potential issues.
Note: If the process is changed to initially create the file on the SQL Server's local harddrive, then the Data Flow task succeeds, but the later copy to Sent folder task fails with a very similar permissions error.
Alternative #1: this technique requires creating a new table, loading the parameter values to the table, changing the package to check the table and potentially set it's parameters/variables based on what it finds. The package can then be launched using a SQL Server Agent Job (for which there are multiple methods to manually launch them) and if the calling object has correctly populated the table, the package will behave as if it's parameters were changed at runtime otherwise it will run with the default values.
Alternative #2: Change all folders used by the package to point to folders local to the SQL Server instance and then create a separate scheduled task/application/whatever, with the valid credentials, that would synchronize or move the files to their proper network folders.
even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job
This is probably because the account is not logged on locally at the SQL Server, and so it's a Double-Hop Impersonation scenario, and would require Kerberos Constrained Delegation to be configured.
And you are correct in assessing the options. The general solution is to invoke catalog.start_execution from a session running on the SQL Server, and an Agent Job is the simplest built-in way to do this (the others being xp_cmdshell, Service Broker Activation, or SQL CLR).
I created a software using Visual Basic 2010 and SSMS 2012 and I wish to deploy it. The question here is how should I go about doing it? I know I can create an executable file .exe which is already a good thing about VS2010 and I also think that installing SSMS-2012 as well before installing the main setup. Also, the script of the database would be generated and then run on the client's computer which enables the database to be attached. However, the question here is that should I hard-code the directory of the database files (.mdf) in Visual Basic
Currently, my connection string is such:
Server=myServerAddress;Database=myDataBase;Trusted_Connection=True;
When SSMS 2012 gets installed on any computer, the server (which is based on the computer's name) which could differ in all computers. So will this work? Or are there any other options
P.S i'm a beginner, so please go easy on me :)
if your problem only lies on the computer's name on a different pc, then try to concatenate this on your myServerAddress variable. Use
My.Computer.Name
to get the target computer's name.
This is also a good link I guess, it is an Access database though but the same concept is used (the use of DataDirectory).
I'd not suggest to hardcode the connection string as this ties your application to a very specific deployment scenario. What if a customer wants to use his or her existing SQL Server instance? What if the customer creates a named instance for the SQL Server Express installation (e.g. SERVERNAME\SQLEXPRESS). Sooner or later you'd have to change your code to reflect the different situations and you'd do the same thing that you can do already now.
In .NET, you can store a connection string in an application configuration file. It is named as the Exe file but has a .config extension (e.g. MyExeName.exe.config). In Visual Studio, all you need to do is to add a file to your project ("Application Configuration File", app.config) if it doesn't exist yet. Upon build, it will be renamed to the Exe-name plus .config. See this link for detailed information.
For your scenario, you'd have a configuration file like this. It references the local computer by using "." as the server name. So you don't have to change it after deployment if this is your default scenario:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<connectionStrings>
<add name="MyConnection"
connectionString="Data Source=.;Initial Catalog=myDatabase;Integrated Security=SSPI"/>
</connectionStrings>
</configuration>
In your code, you can access the connection string using the ConfigurationManager class (you might need to add a reference to the System.Configuration assembly). You can use the acquired connection string when you create the connection:
Dim connStr = ConfigurationManager.ConnectionStrings("MyConnection").ConnectionString
Using conn = new SqlConnection(connStr)
conn.Open()
' Use the connection
End Using
To summarize: you'd add a default configuration file with a connection string that works for any server name to your deployment. If the customer wants to have the database on a different instance or another server, you simply change the connection string - but you don't have to if the customer is fine with the default configuration. As you see, it is not much more effort than to hard code it, but it will make your deployments more flexible.
I work for a large company delivering enterprise solutions and the way we handle this is very easy. All you need is a small configuration utility (could even be your installer wizard) where you request a user credentials (which presumably have enough access right to the SQL Server to create/alter databases and objects), and for the server name.
All you need to do after that is to establish a connection to the server, create the database and run the table and/or views creation scripts.
I see you mentioned attaching an MDF file but I cannot see a reason why you would like to do that. If you have system data you still can insert it using the same script or an additional one.
We are in the process of completing SSIS packages. However as we move towards the deployment phase surprisingly we realized that deployment of SSIS package is not really straight forward as terms of security.
we need to run a SSIS package using either batch file or .NET program in production server. The caller (batch or .NET) should call the SSIS package by supplying parameters (such as source file, destination database, userid, password etc) and also read return code (Success/Error) from SSIS. The user id and password should not be in clear text format visible to anyone. We did some research and found that deployment can be done using many options such as XML config, SQL server Configuration, environment variables, Registry etc but did not mentioned any about password encryption. How can we achieve security and ensure flexibility deploying same package in multiple environment (DEV,UAT,PROD) and simple changing the environment specific variables at the time of deployment.
I am looking for somthing where MVC/Entity framework the connection strings are automatically encypted when the application is run first time using following code.
Configuration config = WebConfigurationManager.OpenWebConfiguration("~");
ConfigurationSection section = config.GetSection("connectionStrings");
if (!section.SectionInformation.IsProtected)
{
section.SectionInformation.ProtectSection("DataProtectionConfigurationProvider");
config.Save();
}
Your question confuses things.
By default, SSIS is going to encrypt anything that could be of a sensitive nature by using the author's Active Directory, AD, account. SSIS Packages Using Package Protection Level This Package Protection Level is EncryptSensitiveWithUserKey If you're using a SQL Server user and password in your connection string, the at-rest value in your SSIS package is going to be encrypted. Don't believe, me, open the file and find the Connection Manager's ConnectionString property.
Deployment is not done through XML, SQL Server, Registry etc. Those refer to methods of creating and storing configuration values that an SSIS package may access during run-time. Configuration has no concept of decrypting values. That isn't to say you can't encrypt values and have them decrypted for the caller, just that there is not built in mechanism for signifying "decrypt using this key."
An approach we are using is to use SQL Server for the holding of encrypted data. We create roles authorized to use the keys and then have computed columns that automatically decrypt the data based on role membership. A person not in that role querying the column will only ever get NULL back as the decrypted value. It's working well for use. EncryptionPOC
Deployment is done through
file copy
dtutil.exe
dtsinstall.exe
.NET api
I have a package that loads data from a text file into a table but I am having trouble setting the package configurations. I was hoping that I could import my package to any database and that that package would automatically lookup the SSISConfigurations table on the database that package is installed on. The package seems to always look on the original database SSISConfigration table where I first created the package. Is it possible to tell the package to look on the source(where the package is installed) database for the SSISConfigration table? Note. I have created the package configuration to look at a SSISConfigruation table for the connection string.
You have to set the connection string to the connection used for the configuration database to match the machine you wish to use for configuration. You could set up your connection string to look something like this:
Data Source=.;Initial Catalog=ConfigurationDB;Provider=SQLNCLI10.1;Integrated Security=SSPI;Auto Translate=False;
The "." after Data Source= is a shortcut meaning the current server. As long as all of the servers you are running your package on have the same configuration database then this will work for you. Alternatively, you can set the connection string for your configuration database from an environment variable. In this case you can use whatever DB you want for each server in your environment. You will simply need to set the environment variable you use to have the proper connection string for that server.
We start with an environment variable that defines the database to be looking into. This varaiable is then set on every server (As well as the devs machines) to look to the correct database for that server. Then we have a configuration that uses the SSIS config tables for the rest of the configuration.
I have an SQL 2005 SSIS package that takes data from an Oracle DB Table, and transfers it to a SQL Server Table.
I have set up an "Oracle Provider for OLE DB" for the Oracle connection and a "SQL Native Client" for SQL Server Connection.
The Oracle and SQL connections will depend on the development and shipping stage, which are:
Local environment
SYS - For integration and System testing
UAT - For user acceptance testing
PRE - Mimics the LIVE system for confidence testing
Live - The live system
In the Connection Manager for Oracle, it expects the following:
Server Name (which, for example can be DEVSERVER)
User Name (which, for example, can be devserver_user)
Password (which, for example, can be devserver_pass)
So, I was wondering how I could parameterise these such that the settings are picked up depending on the server. Ideally this would be a connection string that is stored in the registry (to have commonality with the architecture of other systems in our company).
I have attempted to specify the above settings through Package Configurations. I have also tried specifying the connection string, which would look something like this:
Provider=OraOLEDB.Oracle;Data Source=DEVSERVER;User ID=devserver_user;Password=devserver_pass;PLSQLRSet=1;OLE DB Services = -2;
I have tried this through a registry setting, environment settings, and XML config file. I am mapping these item to the properties on the connection object, but the settings do not seem to hold. I.e. when I open the connection object these settings are not there.
What happens is that when I open the OLE DB source and specify the connection, it fails, because the connection object is not picking up the items in the Package Configurations.
Is there something I am missing, some setting that I have to configure. I guess I'm not sure as to what I'm not seeing anything!
Any help would be appreciated.
Just worked this out this myself.
This really was a case of RTFM! The first paragraph on the MSDN Package Configurations page says it all:
Typically, you create a package set
properties on the package objects
during package development, and then
add the configuration to the package.
Still, I hope this is still of help to other RTFMers!