Setup: A pretty standard data export SSIS package (SQL Server 2016 compatible), created in VS2019/Data Tools and deployed using the SSIS Project Deployment model to the Integration Services Catalog of a SQL Server 2016 instance. The package creates files in a network folder before sending the file out via FTP and putting a copy of the file in a Sent folder.
The project requirements include having the package running on a schedule using "default" parameter values, as well as allowing users to manually run the package using "non-default" parameter values from within a stand-alone application.
Current behavior: the package behaves correctly when run from a SQL Server Agent Job that is configured with a SQL proxy and credentials mapped to a domain login with the proper permissions for the network folder.
Problem: the Data Flow task fails to create the file with a "Cannot open the datafile" error when running the package directly using any of the following methods (even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job):
Using SSMS to right-click on the package and selecting Execute
Using the DTEXEC SQL utility
Using the SSISDB.catalog.start_execution SQL Server stored procedure
As far as I'm aware, these are the only methods capable of starting a SSIS package and changing the package's parameter values. I either need to get one of the latter 2 methods to work, find another option that allows for changing the parameter values while launching the package, or use one of 2 techniques I'm aware of (detailed below) that would add yet another failure point to the process as well as other potential issues.
Note: If the process is changed to initially create the file on the SQL Server's local harddrive, then the Data Flow task succeeds, but the later copy to Sent folder task fails with a very similar permissions error.
Alternative #1: this technique requires creating a new table, loading the parameter values to the table, changing the package to check the table and potentially set it's parameters/variables based on what it finds. The package can then be launched using a SQL Server Agent Job (for which there are multiple methods to manually launch them) and if the calling object has correctly populated the table, the package will behave as if it's parameters were changed at runtime otherwise it will run with the default values.
Alternative #2: Change all folders used by the package to point to folders local to the SQL Server instance and then create a separate scheduled task/application/whatever, with the valid credentials, that would synchronize or move the files to their proper network folders.
even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job
This is probably because the account is not logged on locally at the SQL Server, and so it's a Double-Hop Impersonation scenario, and would require Kerberos Constrained Delegation to be configured.
And you are correct in assessing the options. The general solution is to invoke catalog.start_execution from a session running on the SQL Server, and an Agent Job is the simplest built-in way to do this (the others being xp_cmdshell, Service Broker Activation, or SQL CLR).
I have a very small test application in which I'm trying to install a Windows Service and create a LocalDB database during the install process, then connect to that LocalDB database when the Windows Service runs.
I am running into huge problems connecting to a LocalDB instance from my Windows Service.
My installation process is exactly like this:
Execute an installer .msi file which runs the msiexec process as the NT AUTHORITY\SYSTEM account.
Run a custom action to execute SqlLocalDB.exe with the following commands:
sqllocaldb.exe create MYINSTANCE
sqllocaldb.exe share MYINSTANCE MYINSTANCESHARE
sqllocaldb.exe start MYINSTANCE
Run a custom C# action using ADO.NET (System.Data.SqlConnection) to perform the following actions:
Connect to the following connection string, Data Source=(localdb)\MYINSTANCE; Integrated Security=true
CREATE DATABASE TestDB
USE TestDB
CREATE TABLE ...
Start the Windows Service before the installer finishes.
The Windows Service is installed to the LocalSystem account and so also runs as the NT AUTHORITY\SYSTEM user account.
The service attempts to connect using the same connection string used above.
I am consistently getting the following error when trying to open the connection to the above connection string from within the Windows Service:
System.Data.SqlClient.SqlException (0x80131904): A network-related or
instance-specific error occurred while establishing a connection to
SQL Server. The server was not found or was not accessible. Verify
that the instance name is correct and that SQL Server is configured to
allow remote connections. (provider: SQL Network Interfaces, error: 50
- Local Database Runtime error occurred. The specified LocalDB instance does not exist.
This is frustrating because both the msi installer custom action and the Windows Service are running under the same Windows user account (I checked, they're both NT AUTHORITY\System). So why the first works and the second does not is beyond me.
I have tried changing the connection strings used in the custom action and the Windows Service to use the share name (localdb)\.\MYINSTANCESHARE and I get the exact same error from the Windows Service.
I have tried changing the user account that the Windows Service logs on as to my Windows user account, which does work as long as I first run a command to add it to the SQL server logins for that instance.
I've also tried running a console application and connecting to the share name connection string and that works as well.
I've also tried connecting to the share name from SQL Server Management Studio and that works as well.
However none of these methods really solve my problem. I need a Windows Service because it starts up as soon as the computer starts up (even if no user logs on) and starts up no matter which user account is logged in.
How does a Windows Service connect to a LocalDB private instance?
I am using SQL Server 2014 Express LocalDB.
Picking up from the comments on the question, here are some areas to look at. Some of these have already been answered in those comments, but I am documenting here for others in case the info might be helpful.
Check here for a great source of info on SQL Server Express LocalDB:
SQL Server 2014 Express LocalDB
SqlClient Support for LocalDB
SqlLocalDB Utlity
Introducing LocalDB, an improved SQL Express (also look at the Q&A section at the end of the main post, just before the comments, as someone asked if LocalDB can be launched from a service, and the answer is:
LocalDB can be launched from a service, as long as the profile is loaded for the service account.
What version of .Net is being used? Here it is 4.5.1 (good) but earlier versions could not handle the preferred connection string (i.e. #"(localdb)\InstanceName"). The following quote is taken from the link noted above:
If your application uses a version of .NET before 4.0.2 you must connect directly to the named pipe of the LocalDB.
And according to the MSDN page for SqlConnection.ConnectionString:
Beginning in .NET Framework 4.5, you can also connect to a LocalDB database as follows:
server=(localdb)\\myInstance
Paths:
Instances: C:\Users{Windows Login}\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances
Databases:
Created via SSMS or direct connection: C:\Users{Windows Login}\Documents or C:\Users{Windows Login}
Created via Visual Studio: C:\Users{Windows Login}\AppData\Local\Microsoft\VisualStudio\SSDT
Initial Problem
Symptoms:
Database files (.mdf and .ldf) created in the expected location:
C:\Windows\System32\config\systemprofile
Instance files created in an unexpected location:
C:\Users\{current user}\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances
Cause (note taken from "SqlLocalDB Utility" MSDN page that is linked above; emphasis mine):
Operations other than start can only be performed on an instance belonging to currently logged in user.
Things to try:
Connection string that specifies the database (though maybe a long-shot if the error is regarding not being able to connect to the instance):
"Server=(LocalDB)\MYINSTANCE; Integrated Security=true ;AttachDbFileName=C:\Windows\System32\config\systemprofile\TestDB.mdf"
"Server=(LocalDB)\.\MYINSTANCESHARE; Integrated Security=true ;AttachDbFileName=C:\Windows\System32\config\systemprofile\TestDB.mdf"
Is the service running? Run the following from a Command Prompt:
TASKLIST /FI "IMAGENAME eq sqlservr.exe"
It should probably be listed under "Console" for the "Session Name" column
Run the following from a Command Prompt:
sqllocaldb.exe info MYINSTANCE
And verify that the value for "Owner" is correct. Is the value for "Shared name" what it should be? If not, the documentation states:
Only an administrator on the computer can create a shared instance of LocalDB
As part of the setup, add the NT AUTHORITY\System account as a Login to the system, which is required if this account is not showing as the "Owner" of the instance:
CREATE LOGIN [NT AUTHORITY\System] FROM WINDOWS;
ALTER SERVER ROLE [sysadmin] ADD MEMBER [NT AUTHORITY\System];
Check the following file for clues / details:
C:\Users{Windows Login}\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\MYINSTANCE\error.log
In the end you might need to create an actual account to create and own the Instance and Database, as well as run your service. LocalDB really is meant to be user-mode, and is there any downside to having your service have its own login? And you probably wouldn't need to share the instance at that point.
And in fact, as noted by Microsoft on the SQL Server YYYY Express LocalDB MSDN page:
An instance of LocalDB owned by the built-in accounts such as NT AUTHORITY\SYSTEM can have manageability issues due to windows file system redirection; Instead use a normal windows account as the owner.
UPDATE (2015-08-21)
Based on feedback from the O.P. that using a regular User account can be problematic in certain environments, AND keeping in mind the original issue of the LocalDB instance being created in the %LOCALAPPDATA% folder for the user running the installer (and not the %LOCALAPPDATA% folder for NT AUTHORITY\System ), I found a solution that seems to keep with the intent of easy installation (no user to create) and should not require needing extra code to load the SYSTEM profile.
Try using one of the two built-in accounts that is not the LocalSystem account (which does not maintain its own registry info. Use either:
NT AUTHORITY\LocalService
NT AUTHORITY\NetworkService
Both have their profile folders in: C:\Windows\ServiceProfiles
While I have not been able to test via an installer, I did test a service logging on as NT AUTHORITY\NetworkService by setting my SQL Server Express 2014 instance to log on as this account, and restarted the SQL Server service. I then ran the following:
EXEC xp_cmdshell 'sqllocaldb c MyTestInstance -s';
and it created the instance in: C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances
I then ran the following:
EXEC xp_cmdshell N'SQLCMD -S (localdb)\MyTestInstance -E -Q "CREATE DATABASE [MyTestDB];"';
and it had created the database in: C:\Windows\ServiceProfiles\NetworkService
I was able to solve similar issue in our WiX installer recently. We have a Windows service, running under SYSTEM account, and an installer, where LocalDB-based storage is one of the options for database configuration. For some time (a couple of years actually) product upgrades and service worked quite fine, with no issues related to LocalDB. We are using default v11.0 instance, which is created in SYSTEM profile in C:\Windows\System32\config tree, and a database specified via AttachDbFileName, created in ALLUSERSPROFILE tree. DB provider is configured to use Windows authentication. We also have a custom action in installer, scheduled as deferred/non-impersonate, which runs DB schema updates.
All this worked fine until recently. After another bunch of DB updates, our new release started to fail after having upgraded over the former - service was unable to start, reporting infamous "A network-related or instance-specific error occurred while establishing a connection to SQL Server" (error 50) fault.
When investigating this issue, it became apparent that the problem is in a way WiX runs custom actions. Although non-impersonated CA-s run under SYSTEM account, the registry profile and environment remain that of current user (I suspect WiX loads these voluntary when attaching to user's session). This leads to incorrect path being expanded from the LOCALAPPDATA variable - the service receives SYSTEM profile one, but the schema update CA works with the user's one.
So here are two possible solutions. The first one is simple, but too intrusive to user's system - with cmd.exe started via psexec, recreate broken instance under the SYSTEM account. This was not an option for us as the user may have other databases created in v11.0 instance, which is public. The second option assumed lots of refactoring, but wouldn't hurt anything. Here is what to do to run DB schema updates properly with LocalDB in WiX CA:
Configure your CA as deferred/non-impersonate (should run under SYSTEM account);
Fix environment to point to SYSTEM profile paths:
var systemRoot = Environment.GetEnvironmentVariable("SystemRoot");
Environment.SetEnvironmentVariable("USERPROFILE", String.Format(#"{0}\System32\config\systemprofile", systemRoot));
Environment.SetEnvironmentVariable("APPDATA", String.Format(#"{0}\System32\config\systemprofile\AppData\Roaming", systemRoot));
Environment.SetEnvironmentVariable("LOCALAPPDATA", String.Format(#"{0}\System32\config\systemprofile\AppData\Local", systemRoot));
Environment.SetEnvironmentVariable("HOMEPATH", String.Empty);
Environment.SetEnvironmentVariable("USERNAME", Environment.UserName);
Load SYSTEM account profile. I used LogonUser/LoadUserProfile native API methods, as following:
[DllImport("advapi32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool LogonUser(
string lpszUserName,
string lpszDomain,
string lpszPassword,
int dwLogonType,
int dwLogonProvider,
ref IntPtr phToken);
[StructLayout(LayoutKind.Sequential)]
struct PROFILEINFO
{
public int dwSize;
public int dwFlags;
[MarshalAs(UnmanagedType.LPWStr)]
public String lpUserName;
[MarshalAs(UnmanagedType.LPWStr)]
public String lpProfilePath;
[MarshalAs(UnmanagedType.LPWStr)]
public String lpDefaultPath;
[MarshalAs(UnmanagedType.LPWStr)]
public String lpServerName;
[MarshalAs(UnmanagedType.LPWStr)]
public String lpPolicyPath;
public IntPtr hProfile;
}
[DllImport("userenv.dll", SetLastError = true, CharSet = CharSet.Unicode)]
[return: MarshalAs(UnmanagedType.Bool)]
static extern bool LoadUserProfile(IntPtr hToken, ref PROFILEINFO lpProfileInfo);
var hToken = IntPtr.Zero;
var hProfile = IntPtr.Zero;
bool result = LogonUser("SYSTEM", "NT AUTHORITY", String.Empty, 3 /* LOGON32_LOGON_SERVICE */, 0 /* LOGON32_PROVIDER_DEFAULT */, ref token);
if (result)
{
var profileInfo = new PROFILEINFO();
profileInfo.dwSize = Marshal.SizeOf(profileInfo);
profileInfo.lpUserName = #"NT AUTHORITY\SYSTEM";
if (LoadUserProfile(token, ref profileInfo))
hProfile = profileInfo.hProfile;
}
Wrap this in an IDisposable class, and use with a using statement to build a context.
The most important - refactor your code to perform necessary DB updates in a child process. This could be a simple exe-wrapper over your installer DLL, or stand-alone utility, if your already have one.
P.S. All these difficulties could be avoided, if only Microsoft let uses choose where to create LocalDB instances, via command line option. Like Postgres' initdb/pg_ctl utilities have, for example.
I suggest using a different user account, and not using the System account, by doing the following:-
create a new account on the machine, and set that to be the account
under which the Windows Service runs. It's not good practice to use
the system account just to run an application, anyway, as the
permissions are excessive.
ensure that the permissions on the LocalDB files are set to allow the said user account to access the database (and thus continue to
use Integrated Security)
make sure it works by trying to connect to the DB (once installed) under the same user account by running sqlcmd or Management Studio
under the context of the said user, then connecting with Integrated
Security to ensure it works.
Some other things to try/consider:
have you checked the Windows Event log for any events that might be useful for diagnostic purposes?
Make sure that if you have any other versions of SQL Server (especially prior to 2012) that for the command-line tools, the %PATH% isn't set to find an older tools version first. Older tools don't support LocalDB.
It is possible also (as an alternative) to set up LocalDB to be shared with other users. This involves sharing the instance, and then granting access to other users. See the "Sharing Issues" section in this article: Troubleshoot SQL Server 2012 Express LocalDB.
There's also another SO article that may contain some more useful information there in the links within (change the language in the URL from Polish to English by changing pl-pl to en-us). His work-around is using SQL Server accounts, which might not be OK in your case.
This might also be useful, as it relates to security permissions being denied, and possible resolutions: https://dba.stackexchange.com/questions/30383/cannot-start-sqllocaldb-instance-with-my-windows-account
Trevor, the problem you have is with the MSI custom actions. You must configure them with "Impersonate=false" otherwise the custom actions will be executed under the current user context.
BTW what tool are you using to create the installer?
Depending on the tool you use, could you please provide screenshots or code snippets of your custom actions configuration?
The accepted answer from this post will give you some additional information about the different custom action execution alternatives:
Run ExeCommand in customAction as Administrator mode in Wix Installer
You will find additional information about impersonation here:
http://blogs.msdn.com/b/rflaming/archive/2006/09/23/768248.aspx
I wouldn't create the database under the system's localdb instance. I'd create it under the current user installing the product. This will make life much easier if you need to delete or manage the database. They can do this through sql management studio. Otherwise, you'll have to use psexc or something else to launch a process under the SYSTEM account to manage it.
Once the db is created, then use the share option you mentioned. The SYSTEM account can then access the database through the share name.
sqllocaldb share MSSqlLocalDb LOCAL_DB
When sharing, I've noticed you'll have to restart the the local db instance to actually access the db through the share name:
sqllocaldb stop MSSQLLocalDB
sqllocaldb start MSSQLLocalDB
Also, You may need to add the SYSTEM account as a db reader and writer to the database ...
EXEC sp_addrolemember db_datareader, 'NT AUTHORITY\SYSTEM'
We are in the process of completing SSIS packages. However as we move towards the deployment phase surprisingly we realized that deployment of SSIS package is not really straight forward as terms of security.
we need to run a SSIS package using either batch file or .NET program in production server. The caller (batch or .NET) should call the SSIS package by supplying parameters (such as source file, destination database, userid, password etc) and also read return code (Success/Error) from SSIS. The user id and password should not be in clear text format visible to anyone. We did some research and found that deployment can be done using many options such as XML config, SQL server Configuration, environment variables, Registry etc but did not mentioned any about password encryption. How can we achieve security and ensure flexibility deploying same package in multiple environment (DEV,UAT,PROD) and simple changing the environment specific variables at the time of deployment.
I am looking for somthing where MVC/Entity framework the connection strings are automatically encypted when the application is run first time using following code.
Configuration config = WebConfigurationManager.OpenWebConfiguration("~");
ConfigurationSection section = config.GetSection("connectionStrings");
if (!section.SectionInformation.IsProtected)
{
section.SectionInformation.ProtectSection("DataProtectionConfigurationProvider");
config.Save();
}
Your question confuses things.
By default, SSIS is going to encrypt anything that could be of a sensitive nature by using the author's Active Directory, AD, account. SSIS Packages Using Package Protection Level This Package Protection Level is EncryptSensitiveWithUserKey If you're using a SQL Server user and password in your connection string, the at-rest value in your SSIS package is going to be encrypted. Don't believe, me, open the file and find the Connection Manager's ConnectionString property.
Deployment is not done through XML, SQL Server, Registry etc. Those refer to methods of creating and storing configuration values that an SSIS package may access during run-time. Configuration has no concept of decrypting values. That isn't to say you can't encrypt values and have them decrypted for the caller, just that there is not built in mechanism for signifying "decrypt using this key."
An approach we are using is to use SQL Server for the holding of encrypted data. We create roles authorized to use the keys and then have computed columns that automatically decrypt the data based on role membership. A person not in that role querying the column will only ever get NULL back as the decrypted value. It's working well for use. EncryptionPOC
Deployment is done through
file copy
dtutil.exe
dtsinstall.exe
.NET api
I have been scouring the internet for a clear answer to this question and I haven't been able to find one.
We currently have a SQL configuration enabled across packages which stores everything in a database, (dev or prod). It is a tricky process to execute packages, figuring out which configuration the packages are mapped to, and having to change them during testing. My hope is that we can use an environment variable to always use a specific connection string to the specific sql server configuration we want to use based on the machine.
I have seen ways to set up variables with the connection string hard coded as an environment variable with an additional SQL server configuration after to manage this. I have also seen ways to manipulate the connection string using XML and calling that from a variable, or from the package itself.
Both of these solutions work and very well may be viable solutions.
My question that I'm really trying to answer is; Is it possible to develop the entire SQL Server configuration connection through the environment variable? As in combining the connection string with the configuration string so that the one environment variable connects straight to the package configuration? Has anyone tried this? Does it make sense as a solution to the problem?
Yes, we have done this and it works fine. Each package has two configurations: the first is an environment variable and the second is a SQL Server table. The environment variable is the connection string for a database containing a table with all the configurations and it is applied to a connection manager. The SQL Server configuration uses that connection manager to connect to the configuration database and retrieve the individual settings.
Because configurations are applied in order, the connection string for the connection manager is set before the SQL Server configuration tries to use it. In summary:
Create your package
Add a connection manager called "Configuration Database" or whatever
Add a SQL Server configuration to the package, use the "Configuration Database" as the Connection, and choose the settings to be stored in the table
Create an environment variable such as SSIS_PKG_CONF with the connection string to be used for the "Configuration Database" connection manager
Add an environment variable configuration to the package, use the SSIS_PKG_CONF variable, and assign it to the connection string for the "Configuration Database" connection manager
Now when you run the package, the configuration proceeds as follows:
The package processes the environment variable configuration
It assigns the value of the environment variable to the "Configuration Database" connection manager's connection string
The package processes the SQL Server configuration
It uses the "Configuration Database" connection manager to retrieve individual settings from a database table
This has worked well for us because we can change the connection string to point at a different source database for the configuration, without changing anything inside the package or the launcher (i.e. the SQL Agent job or script that runs the packages). There are other ways to do it and some people feel strongly that .dtsConfig files are the better option, but we found it easier to manage environment variables than config files
But in the end managing package configurations is a preference and a lot depends on what is easier in your environment with the tools and practices you already have.
I have a package that loads data from a text file into a table but I am having trouble setting the package configurations. I was hoping that I could import my package to any database and that that package would automatically lookup the SSISConfigurations table on the database that package is installed on. The package seems to always look on the original database SSISConfigration table where I first created the package. Is it possible to tell the package to look on the source(where the package is installed) database for the SSISConfigration table? Note. I have created the package configuration to look at a SSISConfigruation table for the connection string.
You have to set the connection string to the connection used for the configuration database to match the machine you wish to use for configuration. You could set up your connection string to look something like this:
Data Source=.;Initial Catalog=ConfigurationDB;Provider=SQLNCLI10.1;Integrated Security=SSPI;Auto Translate=False;
The "." after Data Source= is a shortcut meaning the current server. As long as all of the servers you are running your package on have the same configuration database then this will work for you. Alternatively, you can set the connection string for your configuration database from an environment variable. In this case you can use whatever DB you want for each server in your environment. You will simply need to set the environment variable you use to have the proper connection string for that server.
We start with an environment variable that defines the database to be looking into. This varaiable is then set on every server (As well as the devs machines) to look to the correct database for that server. Then we have a configuration that uses the SSIS config tables for the rest of the configuration.