I have a SSIS package that talks to a remote server over HTTP. I execute the SSIS package using a stored procedure in my database (SQL Server 2012), which is called from a web server. The web server connects to the database using Windows Authentication. I now have a need to run the stored procedure (and therefore, the SSIS package) from a client which does not support Windows Authentication. The SSIS package is complicated enough that migrating to a different solution is not feasible.
The SSIS package has complex variables that are passed. The stored procedure that runs the package looks something like:
CREATE PROCEDURE [dbo].[SSISPackage]
#Parameter1 XML
AS
BEGIN
SET NOCOUNT ON;
DECLARE #execution_id BIGINT
EXEC [SSISDB].[catalog].[create_execution]
#package_name=N'Package.dtsx',
#execution_id=#execution_id OUTPUT,
#folder_name=N'API',
#project_name=N'APIProject',
#use32bitruntime=False,
#reference_id=Null
EXEC [SSISDB].[catalog].[set_execution_parameter_value]
#execution_id,
#object_type=30,
#parameter_name=N'Parameter1',
#parameter_value=#Parameter1
EXEC [SSISDB].[catalog].[start_execution] #execution_id
END
From what I've been reading, it is not possible to run SSIS packages with users authenticated using SQL Server Authentication.
My questions are:
Is it possible to somehow elevate the SQL user to a Windows-auth user, and then execute the stored procedure.
Are there typical approaches in dealing with this problem (e.g. CLR, a queue table, command line call to package)?
This is what I did, maybe it can help someone.
First step,
I created an SQL Authentication login as sysadmin and I allowed this login to impersonate other logins.
And then, I inpersonate a Windows Authentication login that is sysadmin too before calling SSIS packages. To execute scripts as another login, I used TSQL EXECUTE AS. (check the example below)
For instance: server\winuser with Windows Authentication as sysadmin, and a SQL Authentication login sqlauthuser as sysadmin too.
Enable the "Inpersonate Any Login" for sqlauthuser. Login (or use a connection string to connect) as sqlauthuser then impersonate server\winuser to be able to call SSIS packages or procedures:
EXECUTE AS LOGIN = 'server\winuser';
-- call ssis packages
-- call procedures that uses ssis packages
Reference: https://learn.microsoft.com/en-us/sql/t-sql/statements/execute-as-transact-sql?view=sql-server-ver15
I don't think you can execute this using an SQL Server authentication, you will receive the following exception:
The operation cannot be started by an account that uses SQL Server Authentication. Start the operation with an account that uses Windows Authentication
There are many workaround that you can do, check the following links:
The operation cannot be started by an account that uses SQL Server Authentication. SSIS Package
Issues with module signing and SSIS catalog internal procedures
Related
Setup: A pretty standard data export SSIS package (SQL Server 2016 compatible), created in VS2019/Data Tools and deployed using the SSIS Project Deployment model to the Integration Services Catalog of a SQL Server 2016 instance. The package creates files in a network folder before sending the file out via FTP and putting a copy of the file in a Sent folder.
The project requirements include having the package running on a schedule using "default" parameter values, as well as allowing users to manually run the package using "non-default" parameter values from within a stand-alone application.
Current behavior: the package behaves correctly when run from a SQL Server Agent Job that is configured with a SQL proxy and credentials mapped to a domain login with the proper permissions for the network folder.
Problem: the Data Flow task fails to create the file with a "Cannot open the datafile" error when running the package directly using any of the following methods (even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job):
Using SSMS to right-click on the package and selecting Execute
Using the DTEXEC SQL utility
Using the SSISDB.catalog.start_execution SQL Server stored procedure
As far as I'm aware, these are the only methods capable of starting a SSIS package and changing the package's parameter values. I either need to get one of the latter 2 methods to work, find another option that allows for changing the parameter values while launching the package, or use one of 2 techniques I'm aware of (detailed below) that would add yet another failure point to the process as well as other potential issues.
Note: If the process is changed to initially create the file on the SQL Server's local harddrive, then the Data Flow task succeeds, but the later copy to Sent folder task fails with a very similar permissions error.
Alternative #1: this technique requires creating a new table, loading the parameter values to the table, changing the package to check the table and potentially set it's parameters/variables based on what it finds. The package can then be launched using a SQL Server Agent Job (for which there are multiple methods to manually launch them) and if the calling object has correctly populated the table, the package will behave as if it's parameters were changed at runtime otherwise it will run with the default values.
Alternative #2: Change all folders used by the package to point to folders local to the SQL Server instance and then create a separate scheduled task/application/whatever, with the valid credentials, that would synchronize or move the files to their proper network folders.
even when the "current" session is using the same credentials as the SQL Server Credentials/Proxy used by the SQL Server Agent Job
This is probably because the account is not logged on locally at the SQL Server, and so it's a Double-Hop Impersonation scenario, and would require Kerberos Constrained Delegation to be configured.
And you are correct in assessing the options. The general solution is to invoke catalog.start_execution from a session running on the SQL Server, and an Agent Job is the simplest built-in way to do this (the others being xp_cmdshell, Service Broker Activation, or SQL CLR).
I am in trouble with running SSIS packages in my etl process.
There is a table that includes etl steps with different types. i.e. Stored Procedure steps, cmd (copy-move files) steps, sending mail, SSIS steps.
A main Stored procedures runs these steps according to type of steps and logs begin-end times, errors etc. This main stored procedure is
triggered by an sql server job.
When I scheduled this job, SSIS steps takes error which is below;
Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has
occurred. Error code: 0x80040E4D. An OLE DB record is available.
Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80040E4D
Description: "Login failed for user 'server\user$'.".
Yet, when I run this main procedure manually (with my windows user login) it runs successfully.
I checked this running with proxy account, it didn't work for me. Because inside the sql server job, i run my main stored procedure not SSIS package.
proxy account reference link;
https://www.mssqltips.com/sqlservertip/2163/running-a-ssis-package-from-sql-server-agent-using-a-proxy-account/
It should work so long as your connection to the database is valid. The error message is telling you that server\user$ is not a valid login.
If the SSIS Package is connecting to the DB with a username and password (i.e., a SQL user), then it should be straight forward to set up / configure the login to have access to the server and database where your proc resides, and when you next run the job it should be able to successfully execute the proc.
If instead the SSIS Package is connecting with Windows Authentication, then you'll need to use a Proxy that specifies a Credential that references a Windows / Active Directory account with database access. In SSMS:
Server >> Security >> Credentials
create a new credential
give it a name
specify a windows account that has permissions to execute your proc
Server >> SQL Server Agent >> Proxies >> SSIS Package Execution
create a new proxy
select the credential you just created
Select "SQL Server Integration Services Package" subsystem
Server >> SQL Server Agent >> Jobs >> Your Job >> SSIS Job Step
in the "Run as" field, pick the proxy you just created
Now when you kick off the job, your SSIS package should make a connection to the database using the windows account specified in the credential set up in step 1, which should be able to successfully execute the stored proc.
does your package of SSIS any password? if not,set password for it .then edit your job and write password there too.
If you have a password in your SSIS package, make sure that the ProtectionLevel is set to "EncryptSensitiveWithPassword". If you use "EncryptSensitiveWithUserKey" then you'll be the only user who can run it.
I have an SSIS Package that runs via a SQL job on a SSIS server (Server A) that executes a stored procedure on the database server (server B). This stored procedure does a Bulk insert from a file share that is located on the SSIS Server (Server A). However, every time that the stored procedure runs it fails with the follow error:
Execute Membership Process:Error: Executing the query "exec storedprocname ?, ?" failed with the following error: "Cannot bulk load because the file "\ServerA\TestLoads\Membership\Processing\filename.csv" could not be opened. Operating system error code 5(Access is denied.).". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
I am pretty positive that the issue is related to permissions. If I store the files on the database server (Server B) then it processes. Or if I run the proc manually, it will work.
When I execute the job on Server A it executes as the service account for that server. That account has full access to Server A and Server B (Admin in SQL and on the server). I believe what is happening is the credentials get passed the first time, but they are not continued once the stored proc runs. I ran wireshark on Server A (SSIS Server) so that I could see what was access the share and try to get some more information. What I found was that there was no account information being passed, it was just blank.
I went through a lot of steps just to try to see if could get that work such as granting everyone access to the share, enabling the guest account, allowing anonymous users, etc. Not stuff I would want to do, but trying to narrow down the issue. None of those worked.
I tried modify the stored proc to use WITH EXECUTE AS OWNER. Still did not work, but got a different error. Also tried a variety of other accounts to execute as and got the same error each time.
Execute Membership Process:Error: Executing the query "exec [storedprocname] ?, ?" failed with the following error: "Could not obtain information about Windows NT group/user '', error code 0x5.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
Tried a variety of solutions that I found online to get this to work and nothing so far has done it.
I understand that is not an ideal solution. I was under the impression that the developers where using SSIS to load the file initially and then using SQL for the rest of the process which would have worked. But because SQL has to touch the file system it keeps failing. And at this stage, there is not the option of rewriting this. Additionally, this process will work if we move the files to the database server (Server B), but that eliminates a large need for us in having the SSIS server in the first place which was to get files being processed off of the database server
Any ideas on if there is a way to get the current solution to work? Basically, what I think I am needing is to run the SSIS package and for a way to pass credentials via the stored proc to the file share during that process.
We are using Windows Server 2012 R2 on both servers and SQL Server 2012 sp3 Developer edition.
Thanks for the help!
I've had this issue before, and I still don't fully understand Kerberos authentication, but that fixed it for me. It's something to do with "double-hop" of authentication i.e. creds going from SSIS, through SQL Server, to a network Server.
Try setting up Kerberos Authentication for SQL Server. There are detailed step-by-step instructions with screenshots here => Setup Kerberos Authentication for SQL Server
I understand this is like a "link-only" answer, but I don't want to copy-paste & plagiarize the author's original works i.e. blog post, hence the link.
I am trying to add user permissions using this stored procedure:
exec sp_addrolemember db_datawriter, MYUSER
The database is MS SQL 2005 and within SQL Management Studio the stored procedure works correctly using my authorized windows login.
I created a simple HTML CGI website with Python as the scripting language using pyodbc. I also created a local user 'dataviewer' login that I used when pyodbc connects that has the following server role permissions the same as my windows login:
On the website I echo back the sql command that Python used and copy and paste the exact command in SQL SMS and the stored procedure works correctly. It seems as though there is a security conflict somewhere but not sure what is happening becasue my windows login has the same server roles permission as the dataviewer login.
Either commit the transaction with Connection.commit(), or set autocommit=True in connection string.
I am trying to move all .zip in a specific folder to another folder. the source folder is located on another server, currently i am using
EXECUTE xp_cmdshell 'copy \\server1\e$\ETL\*.zip \\server2\e$\ETL\'
GO
Which is working if I am logged into both server, but the goal is to automate this process VIA sql server job agent. I have tried
EXECUTE sp_xp_cmdshell_proxy_account 'domain\useracc','pass'
GO
EXECUTE xp_cmdshell 'copy \\server1\e$\ETL\*.zip \\server2\e$\ETL\'
GO
but I am receiving the following error;
An error occurred during the execution of sp_xp_cmdshell_proxy_account. Possible reasons: the provided account was invalid or the '##xp_cmdshell_proxy_account##' credential could not be created. Error code: '0'.
And also not sure if this is my solution. Please help with how I can achieve this. The file names on server1 change name and quantity everyday.
I would strongly advise...Do not use xp_cmdshell. It opens up large security wholes in your surface area and makes you vulnerable to attack. xp_cmdshell should be disabled!
Instead, if you want to automate this with server agent you have 2 options. My preference would be to write a simple SSIS package with a file system task and schedule this package with server agent. SSIS is underutilized for this kind of task but is actually pretty good at it.
Alternatively re-write your script to use Server Agent CmdExec job steps. This does not require xp_cmdshell to be enabled and reduces the attack surface.
I Found that the following worked for me;
In the command prompt, type services.msc, this would open the list of all services on the server.
In the list of services, look for SQL Server Agent, Right Click -> Properties. Go to Logon Tab
Change the logon to a user with access on both servers. then re-write your script to use Server Agent CmdExec job steps(Thank you Pete Carter)