Setup: A pretty standard data export SSIS package (SQL Server 2016 compatible), created in VS2019/Data Tools and deployed using the SSIS Project Deployment model to the Integration Services Catalog of a SQL Server 2016 instance. The package creates files in a network folder before sending the file out via FTP and putting a copy of the file in a Sent folder.
The project requirements include having the package running on a schedule using "default" parameter values, as well as allowing users to manually run the package using "non-default" parameter values from within a stand-alone application.
Current behavior: the package behaves correctly when run from a SQL Server Agent Job that is configured with a SQL proxy and credentials mapped to a domain login with the proper permissions for the network folder.
Problem: the Data Flow task fails to create the file with a "Cannot open the datafile" error when running the package directly using any of the following methods (even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job):
Using SSMS to right-click on the package and selecting Execute
Using the DTEXEC SQL utility
Using the SSISDB.catalog.start_execution SQL Server stored procedure
As far as I'm aware, these are the only methods capable of starting a SSIS package and changing the package's parameter values. I either need to get one of the latter 2 methods to work, find another option that allows for changing the parameter values while launching the package, or use one of 2 techniques I'm aware of (detailed below) that would add yet another failure point to the process as well as other potential issues.
Note: If the process is changed to initially create the file on the SQL Server's local harddrive, then the Data Flow task succeeds, but the later copy to Sent folder task fails with a very similar permissions error.
Alternative #1: this technique requires creating a new table, loading the parameter values to the table, changing the package to check the table and potentially set it's parameters/variables based on what it finds. The package can then be launched using a SQL Server Agent Job (for which there are multiple methods to manually launch them) and if the calling object has correctly populated the table, the package will behave as if it's parameters were changed at runtime otherwise it will run with the default values.
Alternative #2: Change all folders used by the package to point to folders local to the SQL Server instance and then create a separate scheduled task/application/whatever, with the valid credentials, that would synchronize or move the files to their proper network folders.
even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job
This is probably because the account is not logged on locally at the SQL Server, and so it's a Double-Hop Impersonation scenario, and would require Kerberos Constrained Delegation to be configured.
And you are correct in assessing the options. The general solution is to invoke catalog.start_execution from a session running on the SQL Server, and an Agent Job is the simplest built-in way to do this (the others being xp_cmdshell, Service Broker Activation, or SQL CLR).
Related
I have a OLEDB task that connects to ms-access database on local file system and dumps data into sql database.
Access db connection string is: Data Source=\\my-share\accessdbfile.accdb;Provider=Microsoft.ACE.OLEDB.16.0;
The SSIS package runs OK in visual studio. Access 2016 32-bit runtime is installed on this computer.
After deployment to SSIS catalog, right click and execute works OK. Access 2016 64-bit runtime is installed on this computer.
When this package is scheduled to run via SQL Agent job, using proxy/credential [domain user - example: domainname\user1] (having read permission on the access db file), then the job fails. The SSIS execution report shows error as: Unexpected Termination. There is no further information.
Suppose if I add the proxy credential (domainname\user1) to the local administrators group (in computer management) then the SQL Job runs OK. What could be the solution to this so I can give only the minimum required permissions?
The error is very difficult to debug as it doesn't occur when running via visual studio. Even when deployed to the SQL server and run from the SSIS catalog, the execution works OK. The error occurs only when the package is run via SQL server agent job (via credential/proxy) - the SSIS execution report doesn't show any helpful messages; and surprisingly the error gets resolved when the credential is added as local administrator on the computer - which is a bad practice.
The solution is analyse what kind of ms access file you are using and appropriately use the correct access runtime version (2013 vs 2016) and the correct provider value in the connection string.
Access runtime:
In my case, I uninstalled Access 2016 64-bit runtime and installed Access 2013 64-bit runtime.
Provider version:
I changed the provider version from 16.0 to 15.0, as shown below-
Before: Data Source=\\my-share\accessdbfile.accdb;Provider=Microsoft.ACE.OLEDB.16.0;
After: Data Source=\\my-share\accessdbfile.accdb;Provider=Microsoft.ACE.OLEDB.15.0;
Note:
There is no need to add the user to the local administrator group.
There is no need to login to the computer using that account.
Only read permission (under windows folder security) is sufficient assuming the SSIS packge will only read the ms access database.
It's hard to debug this only with the information provided, but in general:
The user running the job requires at least write and delete access on the underlying folder in which the Access database resides. This is required to generate a lockfile for the database.
If this is undesirable because the job may be modified by a malicious user, I've seen deployments where a copy is made in a temporary folder with write access and a size quota, and that temporary folder is removed as soon as the job finishes, regardless of if the job succeeded.
I have a developer pc "A". And I have a SQL Server "B".
My SQL Server is a Microsoft SQL Server 2019.
On server B, I have a database with a stored procedure, which bulk loads data from a text file (using the BULK INSERT command).
I have now created an SSIS project on server A, which calls the stored procedure on server B using the "Execute SQL Task". The connection on the task is pointing at the database on server B. I have tried using the OLE DB connection and ADO.
When I place my text file on server B and reference the file like D:\myFolder\myFile.txt, everything works fine.
When I place my text file on server A and references the file like \\\A\myShare\myFile.txt it fails. The error I get is:
[Execute SQL Task] Error: Executing the query "exec BulkInsert
'\\A\myShare\myFile.txt'" failed with the following error: "Cannot
bulk load because the file "\\A\myShare\myFile.txt" could not be
opened. Operating system error code 5(Access is denied.).". Possible
failure reasons: Problems with the query, "ResultSet" property not set
correctly, parameters not set correctly, or connection not established
correctly.
I have tried logging on to server B and via a File Explorer open the file using the above path. It works fine. And yes, I am logged in as the same user on both servers A and B.
I have also tried giving "Everyone" read/write access to the share, but still the same poor result.
I am only interested in knowing how to fix this problem, using the stored procedure call - I do not wish to rebuild its functionality in SSIS (the stored procedure is maintained by an external company and they may change it at their will, but we agree on how I can call it).
According to what I read from Microsoft, it should not be a problem, call BULK INSERT using a UNC path.
According to what I read from Microsoft, it should not be a problem, call BULK INSERT using a UNC path.
This is true, UNC paths are supported by the BULK INSERT command.
Based on the Microsoft documentation, BULK INSERT has three main requirements:
The server must have permission to access both the file and the destination database.
The server runs the Bulk Insert task. Therefore, any format file that the task uses must be located on the server.
The source file that the Bulk Insert task loads can be on the same server as the SQL Server database into which data is inserted, or on a remote server. If the file is on a remote server, you must specify the file name using the Universal Naming Convention (UNC) name in the path.
The first requirement means that you should grant the SQL Server Service account to access the UNC path, not the Windows account you are logging in with.
You should refer to the following articles to find the SQL Server service account name:
Configure File System Permissions for Database Engine Access
How to Find Service Account for SQL Server and SQL Server Agent?
Besides, you can learn more about SQL Server service accounts and permissions in the following documentation:
Configure Windows Service Accounts and Permissions
Alternative - Mapping network drive
As an alternative, you can try mapping the network drive within SQL Server. You can check the following articles for more information:
Make Network Path Visible For SQL Server Backup and Restore in SSMS
How to Map Network Drive as Fixed Drive?
I have a SSIS package which i tried to schedule as a job in SQL Server Agent.
In order to achieve this i have created a Credentials with User having SysAdmin role to the server. Also i have created a proxy account to run the job where the principals added for
SQLServerAgent,SQLServerOperator,SQLServerReader.
While configuring job selected package source as File system and given the package path to execute the same.
Package is running good and updating results to table stand alone but not working when executing via SQL Job.
Error:
The process could not be created for step 1 of job 0x7D722F8836388B46956B164E33DE8955 (reason: A required privilege is not held by the client). The step failed.,00:00:00,0,0,,,,0
Fix Tried: Modified the package security with Encryptallwithpassword and no luck still.
Check your SQL Server Agent service account to make sure it has permissions to the database and the folder where the packages are being called from.
The proxy account is the user that the package will run as, so make sure that is selected in the job step.
For package encryption, unless you are saving credentials in the package, you can set the protection level to "DontSaveSensitive". It sounds like the proxy account will have the permissions you need already assigned.
We are in the process of completing SSIS packages. However as we move towards the deployment phase surprisingly we realized that deployment of SSIS package is not really straight forward as terms of security.
we need to run a SSIS package using either batch file or .NET program in production server. The caller (batch or .NET) should call the SSIS package by supplying parameters (such as source file, destination database, userid, password etc) and also read return code (Success/Error) from SSIS. The user id and password should not be in clear text format visible to anyone. We did some research and found that deployment can be done using many options such as XML config, SQL server Configuration, environment variables, Registry etc but did not mentioned any about password encryption. How can we achieve security and ensure flexibility deploying same package in multiple environment (DEV,UAT,PROD) and simple changing the environment specific variables at the time of deployment.
I am looking for somthing where MVC/Entity framework the connection strings are automatically encypted when the application is run first time using following code.
Configuration config = WebConfigurationManager.OpenWebConfiguration("~");
ConfigurationSection section = config.GetSection("connectionStrings");
if (!section.SectionInformation.IsProtected)
{
section.SectionInformation.ProtectSection("DataProtectionConfigurationProvider");
config.Save();
}
Your question confuses things.
By default, SSIS is going to encrypt anything that could be of a sensitive nature by using the author's Active Directory, AD, account. SSIS Packages Using Package Protection Level This Package Protection Level is EncryptSensitiveWithUserKey If you're using a SQL Server user and password in your connection string, the at-rest value in your SSIS package is going to be encrypted. Don't believe, me, open the file and find the Connection Manager's ConnectionString property.
Deployment is not done through XML, SQL Server, Registry etc. Those refer to methods of creating and storing configuration values that an SSIS package may access during run-time. Configuration has no concept of decrypting values. That isn't to say you can't encrypt values and have them decrypted for the caller, just that there is not built in mechanism for signifying "decrypt using this key."
An approach we are using is to use SQL Server for the holding of encrypted data. We create roles authorized to use the keys and then have computed columns that automatically decrypt the data based on role membership. A person not in that role querying the column will only ever get NULL back as the decrypted value. It's working well for use. EncryptionPOC
Deployment is done through
file copy
dtutil.exe
dtsinstall.exe
.NET api
I have an SQL 2005 SSIS package that takes data from an Oracle DB Table, and transfers it to a SQL Server Table.
I have set up an "Oracle Provider for OLE DB" for the Oracle connection and a "SQL Native Client" for SQL Server Connection.
The Oracle and SQL connections will depend on the development and shipping stage, which are:
Local environment
SYS - For integration and System testing
UAT - For user acceptance testing
PRE - Mimics the LIVE system for confidence testing
Live - The live system
In the Connection Manager for Oracle, it expects the following:
Server Name (which, for example can be DEVSERVER)
User Name (which, for example, can be devserver_user)
Password (which, for example, can be devserver_pass)
So, I was wondering how I could parameterise these such that the settings are picked up depending on the server. Ideally this would be a connection string that is stored in the registry (to have commonality with the architecture of other systems in our company).
I have attempted to specify the above settings through Package Configurations. I have also tried specifying the connection string, which would look something like this:
Provider=OraOLEDB.Oracle;Data Source=DEVSERVER;User ID=devserver_user;Password=devserver_pass;PLSQLRSet=1;OLE DB Services = -2;
I have tried this through a registry setting, environment settings, and XML config file. I am mapping these item to the properties on the connection object, but the settings do not seem to hold. I.e. when I open the connection object these settings are not there.
What happens is that when I open the OLE DB source and specify the connection, it fails, because the connection object is not picking up the items in the Package Configurations.
Is there something I am missing, some setting that I have to configure. I guess I'm not sure as to what I'm not seeing anything!
Any help would be appreciated.
Just worked this out this myself.
This really was a case of RTFM! The first paragraph on the MSDN Package Configurations page says it all:
Typically, you create a package set
properties on the package objects
during package development, and then
add the configuration to the package.
Still, I hope this is still of help to other RTFMers!