I have a deployment package that needs to run against about 3 different enviroments.
I want to specify a sql script to run (source) with the enviroments database (destiniation).
I don't want to specify the connection string in the deploy script because it contains sql login info.
I would like to be able to read a setting from the destination for the connection string.
Can I mark this a parameter to be specified when unpackaging the deployment package on the server? If so, how so I use the parameter in the dest:sql="connection string"?
Any suggestions would be great.
Scott Guthrie has a pretty good write up on this sort of thing here. He specifically mentions the changing of parameters both in prompts for the admin and in an automated fashion via the command line within deployment and/or automation scripts.
Related
Setup: A pretty standard data export SSIS package (SQL Server 2016 compatible), created in VS2019/Data Tools and deployed using the SSIS Project Deployment model to the Integration Services Catalog of a SQL Server 2016 instance. The package creates files in a network folder before sending the file out via FTP and putting a copy of the file in a Sent folder.
The project requirements include having the package running on a schedule using "default" parameter values, as well as allowing users to manually run the package using "non-default" parameter values from within a stand-alone application.
Current behavior: the package behaves correctly when run from a SQL Server Agent Job that is configured with a SQL proxy and credentials mapped to a domain login with the proper permissions for the network folder.
Problem: the Data Flow task fails to create the file with a "Cannot open the datafile" error when running the package directly using any of the following methods (even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job):
Using SSMS to right-click on the package and selecting Execute
Using the DTEXEC SQL utility
Using the SSISDB.catalog.start_execution SQL Server stored procedure
As far as I'm aware, these are the only methods capable of starting a SSIS package and changing the package's parameter values. I either need to get one of the latter 2 methods to work, find another option that allows for changing the parameter values while launching the package, or use one of 2 techniques I'm aware of (detailed below) that would add yet another failure point to the process as well as other potential issues.
Note: If the process is changed to initially create the file on the SQL Server's local harddrive, then the Data Flow task succeeds, but the later copy to Sent folder task fails with a very similar permissions error.
Alternative #1: this technique requires creating a new table, loading the parameter values to the table, changing the package to check the table and potentially set it's parameters/variables based on what it finds. The package can then be launched using a SQL Server Agent Job (for which there are multiple methods to manually launch them) and if the calling object has correctly populated the table, the package will behave as if it's parameters were changed at runtime otherwise it will run with the default values.
Alternative #2: Change all folders used by the package to point to folders local to the SQL Server instance and then create a separate scheduled task/application/whatever, with the valid credentials, that would synchronize or move the files to their proper network folders.
even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job
This is probably because the account is not logged on locally at the SQL Server, and so it's a Double-Hop Impersonation scenario, and would require Kerberos Constrained Delegation to be configured.
And you are correct in assessing the options. The general solution is to invoke catalog.start_execution from a session running on the SQL Server, and an Agent Job is the simplest built-in way to do this (the others being xp_cmdshell, Service Broker Activation, or SQL CLR).
We are in the process of completing SSIS packages. However as we move towards the deployment phase surprisingly we realized that deployment of SSIS package is not really straight forward as terms of security.
we need to run a SSIS package using either batch file or .NET program in production server. The caller (batch or .NET) should call the SSIS package by supplying parameters (such as source file, destination database, userid, password etc) and also read return code (Success/Error) from SSIS. The user id and password should not be in clear text format visible to anyone. We did some research and found that deployment can be done using many options such as XML config, SQL server Configuration, environment variables, Registry etc but did not mentioned any about password encryption. How can we achieve security and ensure flexibility deploying same package in multiple environment (DEV,UAT,PROD) and simple changing the environment specific variables at the time of deployment.
I am looking for somthing where MVC/Entity framework the connection strings are automatically encypted when the application is run first time using following code.
Configuration config = WebConfigurationManager.OpenWebConfiguration("~");
ConfigurationSection section = config.GetSection("connectionStrings");
if (!section.SectionInformation.IsProtected)
{
section.SectionInformation.ProtectSection("DataProtectionConfigurationProvider");
config.Save();
}
Your question confuses things.
By default, SSIS is going to encrypt anything that could be of a sensitive nature by using the author's Active Directory, AD, account. SSIS Packages Using Package Protection Level This Package Protection Level is EncryptSensitiveWithUserKey If you're using a SQL Server user and password in your connection string, the at-rest value in your SSIS package is going to be encrypted. Don't believe, me, open the file and find the Connection Manager's ConnectionString property.
Deployment is not done through XML, SQL Server, Registry etc. Those refer to methods of creating and storing configuration values that an SSIS package may access during run-time. Configuration has no concept of decrypting values. That isn't to say you can't encrypt values and have them decrypted for the caller, just that there is not built in mechanism for signifying "decrypt using this key."
An approach we are using is to use SQL Server for the holding of encrypted data. We create roles authorized to use the keys and then have computed columns that automatically decrypt the data based on role membership. A person not in that role querying the column will only ever get NULL back as the decrypted value. It's working well for use. EncryptionPOC
Deployment is done through
file copy
dtutil.exe
dtsinstall.exe
.NET api
I am trying to move all .zip in a specific folder to another folder. the source folder is located on another server, currently i am using
EXECUTE xp_cmdshell 'copy \\server1\e$\ETL\*.zip \\server2\e$\ETL\'
GO
Which is working if I am logged into both server, but the goal is to automate this process VIA sql server job agent. I have tried
EXECUTE sp_xp_cmdshell_proxy_account 'domain\useracc','pass'
GO
EXECUTE xp_cmdshell 'copy \\server1\e$\ETL\*.zip \\server2\e$\ETL\'
GO
but I am receiving the following error;
An error occurred during the execution of sp_xp_cmdshell_proxy_account. Possible reasons: the provided account was invalid or the '##xp_cmdshell_proxy_account##' credential could not be created. Error code: '0'.
And also not sure if this is my solution. Please help with how I can achieve this. The file names on server1 change name and quantity everyday.
I would strongly advise...Do not use xp_cmdshell. It opens up large security wholes in your surface area and makes you vulnerable to attack. xp_cmdshell should be disabled!
Instead, if you want to automate this with server agent you have 2 options. My preference would be to write a simple SSIS package with a file system task and schedule this package with server agent. SSIS is underutilized for this kind of task but is actually pretty good at it.
Alternatively re-write your script to use Server Agent CmdExec job steps. This does not require xp_cmdshell to be enabled and reduces the attack surface.
I Found that the following worked for me;
In the command prompt, type services.msc, this would open the list of all services on the server.
In the list of services, look for SQL Server Agent, Right Click -> Properties. Go to Logon Tab
Change the logon to a user with access on both servers. then re-write your script to use Server Agent CmdExec job steps(Thank you Pete Carter)
I have been scouring the internet for a clear answer to this question and I haven't been able to find one.
We currently have a SQL configuration enabled across packages which stores everything in a database, (dev or prod). It is a tricky process to execute packages, figuring out which configuration the packages are mapped to, and having to change them during testing. My hope is that we can use an environment variable to always use a specific connection string to the specific sql server configuration we want to use based on the machine.
I have seen ways to set up variables with the connection string hard coded as an environment variable with an additional SQL server configuration after to manage this. I have also seen ways to manipulate the connection string using XML and calling that from a variable, or from the package itself.
Both of these solutions work and very well may be viable solutions.
My question that I'm really trying to answer is; Is it possible to develop the entire SQL Server configuration connection through the environment variable? As in combining the connection string with the configuration string so that the one environment variable connects straight to the package configuration? Has anyone tried this? Does it make sense as a solution to the problem?
Yes, we have done this and it works fine. Each package has two configurations: the first is an environment variable and the second is a SQL Server table. The environment variable is the connection string for a database containing a table with all the configurations and it is applied to a connection manager. The SQL Server configuration uses that connection manager to connect to the configuration database and retrieve the individual settings.
Because configurations are applied in order, the connection string for the connection manager is set before the SQL Server configuration tries to use it. In summary:
Create your package
Add a connection manager called "Configuration Database" or whatever
Add a SQL Server configuration to the package, use the "Configuration Database" as the Connection, and choose the settings to be stored in the table
Create an environment variable such as SSIS_PKG_CONF with the connection string to be used for the "Configuration Database" connection manager
Add an environment variable configuration to the package, use the SSIS_PKG_CONF variable, and assign it to the connection string for the "Configuration Database" connection manager
Now when you run the package, the configuration proceeds as follows:
The package processes the environment variable configuration
It assigns the value of the environment variable to the "Configuration Database" connection manager's connection string
The package processes the SQL Server configuration
It uses the "Configuration Database" connection manager to retrieve individual settings from a database table
This has worked well for us because we can change the connection string to point at a different source database for the configuration, without changing anything inside the package or the launcher (i.e. the SQL Agent job or script that runs the packages). There are other ways to do it and some people feel strongly that .dtsConfig files are the better option, but we found it easier to manage environment variables than config files
But in the end managing package configurations is a preference and a lot depends on what is easier in your environment with the tools and practices you already have.
I have an SQL 2005 SSIS package that takes data from an Oracle DB Table, and transfers it to a SQL Server Table.
I have set up an "Oracle Provider for OLE DB" for the Oracle connection and a "SQL Native Client" for SQL Server Connection.
The Oracle and SQL connections will depend on the development and shipping stage, which are:
Local environment
SYS - For integration and System testing
UAT - For user acceptance testing
PRE - Mimics the LIVE system for confidence testing
Live - The live system
In the Connection Manager for Oracle, it expects the following:
Server Name (which, for example can be DEVSERVER)
User Name (which, for example, can be devserver_user)
Password (which, for example, can be devserver_pass)
So, I was wondering how I could parameterise these such that the settings are picked up depending on the server. Ideally this would be a connection string that is stored in the registry (to have commonality with the architecture of other systems in our company).
I have attempted to specify the above settings through Package Configurations. I have also tried specifying the connection string, which would look something like this:
Provider=OraOLEDB.Oracle;Data Source=DEVSERVER;User ID=devserver_user;Password=devserver_pass;PLSQLRSet=1;OLE DB Services = -2;
I have tried this through a registry setting, environment settings, and XML config file. I am mapping these item to the properties on the connection object, but the settings do not seem to hold. I.e. when I open the connection object these settings are not there.
What happens is that when I open the OLE DB source and specify the connection, it fails, because the connection object is not picking up the items in the Package Configurations.
Is there something I am missing, some setting that I have to configure. I guess I'm not sure as to what I'm not seeing anything!
Any help would be appreciated.
Just worked this out this myself.
This really was a case of RTFM! The first paragraph on the MSDN Package Configurations page says it all:
Typically, you create a package set
properties on the package objects
during package development, and then
add the configuration to the package.
Still, I hope this is still of help to other RTFMers!