Determined having #Exclude_Query_Output = 0 is the failing piece in the SQL code. I have no idea on how to correct this.
I've been working on correcting this issue for quite (Months) a while when sending DB Mail from a SQL job. I created one SQL job executing a stored procedure that queries another servers database & tables. Then the results are stored in a variable and emailed to users daily.
Here is my setup:
Linked server A to linked server B using a SQL user named 'xxxdba'
Server type SQL Server in Linked Server Properties
On both servers I'm using "Be made using this security context"
The user has the database role "DatabaseMailUserRole"
Job owner is NT Service/SQLSERVERAGENT
The job is ran as xxxdba in the job step advanced area
On both servers A & B the user has read/write user mappings to the tables in the query which I have supplied below. Under the database itself, the user has execute to the stored procedure that I created.
Job step properties:
Step name
Type T-SQL
Run As (Blank)
Database (Win)
Command: Execute dbo.procedure_db_mail
Procedure that was created:
CREATE PROCEDURE PROCEDURE_DB_MAIL
AS
Declare #Qry varchar(5000) = 'EXEC XXXXXX.WIN.DBO.[Table_Roll_View]' --VIEW
SET #Qry = N'SELECT [Previous Day Table Roll Information] --Stored Procedure
From MSDB.DBO.sysjobs'
EXEC msdb.dbo.sp_send_dbmail
#recipients = 'Jwendt#xxxxxxx.com;',
#Query = #Qry,
#query_result_header = 1,
#exclude_query_output = 0,
#execute_query_database = 'Win',
#subject = 'Table Roll Data',
#attach_query_result_as_file = 1,
#query_attachment_filename = 'Table Roll Data.txt',
#append_query_error = 1,
#profile_name = 'SQL Mail Alert'
View Information:
CREATE VIEW Table_Roll_View
AS
SELECT DISTINCT a.Timestamp, a.DocGamingDate,
a.Document_ID, a.DocType,
t1.InternalStatus, a.DocStatus, a.DocNumber, b.Name, t1.Description, T1.XXXXGameName, b.Location_ID, T1.PitID, T1.TableID
From OPENQUERY([XXXXITDB], 'Select DISTINCT b.PitID, b.TableID, b.Description, a.GameCode, b.XXXXGameName, b.XXXXTableName, b.InternalStatus,
b.ExternalGameDay, b.InternalGameDay
From [XXXXIT].[dbo].[CurrentTableConfig] a
INNER JOIN [XXXXIT].[dbo].[Tables] b
on a.SAKEY = b.TableSAKEY'
) t1
INNER JOIN [Win].[dbo].[XX_Location] b
on b.Name = t1.XXXXTableName
INNER JOIN [Win].[dbo].[XX_Document] a
ON a.Location_ID = b.location_ID
Where a.DocGamingDate = CAST(GETDATE() -1 AS Date) --Looking at yesterday's table processes
--OR a.DocGamingDate = GETDATE() --Including today's table openers
and t1.description NOT LIKE '%zz%'
Execute the job (I'm a sys admin on all servers)
SQL History Message:
Executed as user: xxxdba. Failed to initialize sqlcmd library with error number -2147467259. [SQLSTATE 42000] (Error 22050). The step failed.
Windows Application Log:
SQL Server Scheduled Job 'Previous Day Table Roll Information' (0x74F50053DC25DD439C6438B586359845) - Status: Failed - Invoked on: 2021-02-24 10:19:46 - Message: The job failed. The Job was invoked by User XXXXXXXXXXX\jwendt. The last step to run was step 1 (Execute and email).
Additional notes:
I made the linked server credential a system admin for testing purposes and got the same error.
I get the same result if I execute the SP by itself outside of the job
If I run as myself, the results are the same.
Shared Memory, Named Pipes and TCP/IP are enabled on both servers
Data Access, RPC, RPC Out, Use Remote Collation and Enable Promotion of Distributed Transaction is set to True in the linked server properties
I have the same issue on another server and here is where it gets VERY confusing. I have the same setup and permissions for the user above. The difference is, I have an email that goes out preload and post load.
Management gets an email for a point multiplier that customers earn playing Bingo on Sunday's. This email works correctly (Preload). Once the post load time has passed, the email fails. If the local server time is 10AM then management would already have received their email at 7AM. If the post load time is set to 11AM and the local server time >= 11AM the job fails with the SQLCMD Library error.
I am trying to create an audit trail for actions that are performed within a web application, SQL server agent jobs and manually run queries to the database. I am trying to use triggers to catch updates, inserts and deletes on certain tables.
In the whole this process is working. Example, user performs update in web application and the trigger writes the updated data to an audit trail table I have defined, including the username of the person who performed the action. This works fine from a web application or manual query perspective, but we also have dozens of SQL Server Agent Jobs that I would like to capture which one ran specific queries. Each of the agent jobs are ran with the same username. This works fine also and inputs the username correctly into the table but I can't find which job calls this query.
My current "solution" was to find which jobs are currently running at the time of the trigger, as one of them must be the correct one. Using:
CREATE TABLE #xp_results
(
job_id UNIQUEIDENTIFIER NOT NULL,
last_run_date INT NOT NULL,
last_run_time INT NOT NULL,
next_run_date INT NOT NULL,
next_run_time INT NOT NULL,
next_run_schedule_id INT NOT NULL,
requested_to_run INT NOT NULL, -- BOOL
request_source INT NOT NULL,
request_source_id sysname COLLATE database_default NULL,
running INT NOT NULL, -- BOOL
current_step INT NOT NULL,
current_retry_attempt INT NOT NULL,
job_state INT NOT NULL
)
INSERT INTO #xp_results
EXECUTE master.dbo.xp_sqlagent_enum_jobs 1, 'sa'
SELECT #runningJobs = STUFF((SELECT ',' + j.name
FROM #xp_results r
INNER JOIN msdb..sysjobs j ON r.job_id = j.job_id
WHERE running = 1
FOR XML PATH(''), TYPE).value('.', 'NVARCHAR(MAX)'), 1, 1, '')
DROP TABLE #xp_results
I ran a specific job to test and it seems to work, in that any OTHER job which is running will be listed in #runningJobs, but it doesn't record the job that runs it. I assume that by the time the trigger runs the job has finished.
Is there a way I can find out what job calls the query that kicks off the trigger?
EDIT: I tried changing the SELECT query above to get any job that ran within the past 2 mins or is currently running. The SQL query is now:
SELECT #runningJobs = STUFF((SELECT ',' + j.name
FROM #xp_results r
INNER JOIN msdb..sysjobs j ON r.job_id = j.job_id
WHERE (last_run_date = CAST(REPLACE(LEFT(CONVERT(VARCHAR, getdate(), 120), 10), '-', '') AS INT)
AND last_run_time > CAST(REPLACE(LEFT(CONVERT(VARCHAR,getdate(),108), 8), ':', '') AS INT) - 200)
OR running = 1
FOR XML PATH(''), TYPE).value('.', 'NVARCHAR(MAX)'), 1, 1, '')
When I run a job, then run the above query while the job is running, the correct jobs are returned. But when the SSIS package is run, either via the SQL Server Agent job or manually ran in SSIS, the #runningJobs is not populated and just returns NULL.
So I am now thinking it is a problem with permissions of SSIS and master.dbo.xp_sqlagent_enum_jobs. Any other ideas?
EDIT #2: Actually don't think it is a permissions error. There is an INSERT statement below this code, if it IS a permissions error the INSERT statement does not run and therefore the audit line does not get added to the database. So, as there IS a line added to the database, just not with the runningJobs field populated. Strange times.
EDIT #3: I just want to clarify, I am searching for a solution which DOES NOT require me to go into each job and change anything. There are too many jobs to make this a feasible solution.
WORKING CODE IS IN FIRST EDIT - (anothershrubery)
Use the app_name() function http://msdn.microsoft.com/en-us/library/ms189770.aspx in your audit trigger to get the name of the app running the query.
For SQL Agent jobs, app_name includes the job step id in the app name (if a T-SQL step). We do this in our audit triggers and works great. An example of the app_name() results when running from within an audit trigger:
SQLAgent - TSQL JobStep (Job 0x96EB56A24786964889AB504D9A920D30 : Step
1)
This job can be looked up via the job_id column in msdb.dbo.sysjobs_view.
Since SSIS packages initiate the SQL connection outside of the SQL Agent job engine, those connections will have their own application name, and you need to set the application name within the connection strings of the SSIS packages. In SSIS packages, Web apps, WinForms, or any client that connects to SQL Server, you can set the value that is returned by the app_name function by using this in your connection string :
"Application Name=MyAppNameGoesHere;"
http://www.connectionstrings.com/use-application-name-sql-server/
If the "Application Name" is not set within a .NET connection string, then the default value when using the System.Data.SqlClient.SqlConnection is ".Net SqlClient Data Provider".
Some other fields that are commonly used for auditing:
HOST_NAME(): http://technet.microsoft.com/en-us/library/ms178598.aspx Returns the name of the client computer that is connecting. This is helpful if you have an intranet app.
CONNECTIONPROPERTY('local_net_address'): For getting the client IP address.
CONTEXT_INFO(): http://technet.microsoft.com/en-us/library/ms187768.aspx You can use this to store information for the duration of the connection/session. Context_Info is a binary 128 byte field, so you might need to do conversions to/from strings when using it.
Here are SQL helper methods for setting/getting context info:
CREATE PROC dbo.usp_ContextInfo_SET
#val varchar(128)
as
begin
set nocount on;
DECLARE #c varbinary(128);
SET #c=cast(#val as varbinary(128));
SET CONTEXT_INFO #c;
end
GO
CREATE FUNCTION [dbo].[ufn_ContextInfo_Get] ()
RETURNS varchar(128)
AS
BEGIN
--context_info is binary data type, so will pad any values will CHAR(0) to the end of 128 bytes, so need to replace these with empty string.
RETURN REPLACE(CAST(CONTEXT_INFO() AS varchar(128)), CHAR(0), '')
END
EDIT:
The app_name() is the preferred way to get the application that is involved in the query, however since you do not want to update any of the SSIS packages, then here is an updated query to get currently executing jobs using the following documented SQL Agent tables. You may have to adjust the GRANTs for SELECT in the msdb database for these tables in order for the query to succeed, or create a view using this query, and adjust the grants for that view.
msdb.dbo.sysjobactivity http://msdn.microsoft.com/en-us/library/ms190484.aspx
msdb.dbo.syssessions http://msdn.microsoft.com/en-us/library/ms175016.aspx
msdb.dbo.sysjobs http://msdn.microsoft.com/en-us/library/ms189817.aspx
msdb.dbo.sysjobhistory http://msdn.microsoft.com/en-us/library/ms174997.aspx
Query:
;with cteSessions as
(
--each time that SQL Agent is started, a new record is added to this table.
--The most recent session is the current session, and prior sessions can be used
--to identify the job state at the time that SQL Agent is restarted or stopped unexpectedly
select top 1 s.session_id
from msdb.dbo.syssessions s
order by s.agent_start_date desc
)
SELECT runningJobs =
STUFF(
( SELECT N', [' + j.name + N']'
FROM msdb.dbo.sysjobactivity a
inner join cteSessions s on s.session_id = a.session_id
inner join msdb.dbo.sysjobs j on a.job_id = j.job_id
left join msdb.dbo.sysjobhistory h2 on h2.instance_id = a.job_history_id
WHERE
--currently executing jobs:
h2.instance_id is null
AND a.start_execution_date is not null
AND a.stop_execution_date is null
ORDER BY j.name
FOR XML PATH(''), ROOT('root'), TYPE
).query('root').value('.', 'nvarchar(max)') --convert the xml to nvarchar(max)
, 1, 2, '') -- replace the leading comma and space with empty string.
;
EDIT #2:
Also if you are on SQL 2012 or higher, then checkout the SSISDB.catalog.executions view http://msdn.microsoft.com/en-us/library/ff878089(v=sql.110).aspx to get the list of currently running SSIS packages, regardless of if they were started from within a scheduled job. I have not seen an equivalent view in SQL Server versions prior to 2012.
I would add an extra column to your table e.g. Update_Source, and get all the source apps (including SSIS) to set it when they update the table.
You could use the USER as a DEFAULT for that column to minimize the changes needed.
You could try using CONTEXT_INFO
Try adding a T-SQL step with SET CONTEXT_INFO 'A Job' in to your job
Then try reading that in your trigger using sys.dm_exec_sessions
I'm curious to see if it works - please post your findings.
http://msdn.microsoft.com/en-us/library/ms187768(v=sql.105).aspx
I have a SQL table that all of a sudden cannot return data unless I include with (nolock) on the end, which indicates some kind of lock left on my table.
I've experimented a bit with sys.dm_tran_locks to identify that there are in fact a number of locks on the table, but how do I identify what is locking them (ie the request element of the sys.dm_tran_locks)?
EDIT: I know about sp_lock for pre SQL 2005, but now that that sp is deprecated, AFAIK the right way to do this is with sys.dm_tran_locks. I'm using SQL Server 2008 R2.
Take a look at the following system stored procedures, which you can run in SQLServer Management Studio (SSMS):
sp_who
sp_lock
Also, in SSMS, you can view locks and processes in different ways:
Different versions of SSMS put the activity monitor in different places. For example, SSMS 2008 and 2012 have it in the context menu when you right-click on a server node.
For getting straight to "who is blocked/blocking" I combined/abbreviated sp_who and sp_lock into a single query which gives a nice overview of who has what object locked to what level.
--Create Procedure WhoLock
--AS
set nocount on
if object_id('tempdb..#locksummary') is not null Drop table #locksummary
if object_id('tempdb..#lock') is not null Drop table #lock
create table #lock ( spid int, dbid int, objId int, indId int, Type char(4), resource nchar(32), Mode char(8), status char(6))
Insert into #lock exec sp_lock
if object_id('tempdb..#who') is not null Drop table #who
create table #who ( spid int, ecid int, status char(30),
loginame char(128), hostname char(128),
blk char(5), dbname char(128), cmd char(16)
--
, request_id INT --Needed for SQL 2008 onwards
--
)
Insert into #who exec sp_who
Print '-----------------------------------------'
Print 'Lock Summary for ' + ##servername + ' (excluding tempdb):'
Print '-----------------------------------------' + Char(10)
Select left(loginame, 28) as loginame,
left(db_name(dbid),128) as DB,
left(object_name(objID),30) as object,
max(mode) as [ToLevel],
Count(*) as [How Many],
Max(Case When mode= 'X' Then cmd Else null End) as [Xclusive lock for command],
l.spid, hostname
into #LockSummary
from #lock l join #who w on l.spid= w.spid
where dbID != db_id('tempdb') and l.status='GRANT'
group by dbID, objID, l.spid, hostname, loginame
Select * from #LockSummary order by [ToLevel] Desc, [How Many] Desc, loginame, DB, object
Print '--------'
Print 'Who is blocking:'
Print '--------' + char(10)
SELECT p.spid
,convert(char(12), d.name) db_name
, program_name
, p.loginame
, convert(char(12), hostname) hostname
, cmd
, p.status
, p.blocked
, login_time
, last_batch
, p.spid
FROM master..sysprocesses p
JOIN master..sysdatabases d ON p.dbid = d.dbid
WHERE EXISTS ( SELECT 1
FROM master..sysprocesses p2
WHERE p2.blocked = p.spid )
Print '--------'
Print 'Details:'
Print '--------' + char(10)
Select left(loginame, 30) as loginame, l.spid,
left(db_name(dbid),15) as DB,
left(object_name(objID),40) as object,
mode ,
blk,
l.status
from #lock l join #who w on l.spid= w.spid
where dbID != db_id('tempdb') and blk <>0
Order by mode desc, blk, loginame, dbID, objID, l.status
(For what the lock level abbreviations mean, see e.g. https://technet.microsoft.com/en-us/library/ms175519%28v=sql.105%29.aspx)
Copied from: sp_WhoLock – a T-SQL stored proc combining sp_who and sp_lock...
NB the [Xclusive lock for command] column can be misleading -- it shows the current command for that spid; but the X lock could have been triggered by an earlier command in the transaction.
exec sp_lock
This query should give you existing locks.
exec sp_who SPID -- will give you some info
Having spids, you could check activity monitor(processes tab) to find out what processes are locking the tables ("details" for more info and "kill process" to kill it).
I have a stored procedure that I have put together, that deals not only with locks and blocking, but also to see what is running in a server.
I have put it in master.
I will share it with you, the code is below:
USE [master]
go
CREATE PROCEDURE [dbo].[sp_radhe]
AS
BEGIN
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
-- the current_processes
-- marcelo miorelli
-- CCHQ
-- 04 MAR 2013 Wednesday
SELECT es.session_id AS session_id
,COALESCE(es.original_login_name, '') AS login_name
,COALESCE(es.host_name,'') AS hostname
,COALESCE(es.last_request_end_time,es.last_request_start_time) AS last_batch
,es.status
,COALESCE(er.blocking_session_id,0) AS blocked_by
,COALESCE(er.wait_type,'MISCELLANEOUS') AS waittype
,COALESCE(er.wait_time,0) AS waittime
,COALESCE(er.last_wait_type,'MISCELLANEOUS') AS lastwaittype
,COALESCE(er.wait_resource,'') AS waitresource
,coalesce(db_name(er.database_id),'No Info') as dbid
,COALESCE(er.command,'AWAITING COMMAND') AS cmd
,sql_text=st.text
,transaction_isolation =
CASE es.transaction_isolation_level
WHEN 0 THEN 'Unspecified'
WHEN 1 THEN 'Read Uncommitted'
WHEN 2 THEN 'Read Committed'
WHEN 3 THEN 'Repeatable'
WHEN 4 THEN 'Serializable'
WHEN 5 THEN 'Snapshot'
END
,COALESCE(es.cpu_time,0)
+ COALESCE(er.cpu_time,0) AS cpu
,COALESCE(es.reads,0)
+ COALESCE(es.writes,0)
+ COALESCE(er.reads,0)
+ COALESCE(er.writes,0) AS physical_io
,COALESCE(er.open_transaction_count,-1) AS open_tran
,COALESCE(es.program_name,'') AS program_name
,es.login_time
FROM sys.dm_exec_sessions es
LEFT OUTER JOIN sys.dm_exec_connections ec ON es.session_id = ec.session_id
LEFT OUTER JOIN sys.dm_exec_requests er ON es.session_id = er.session_id
LEFT OUTER JOIN sys.server_principals sp ON es.security_id = sp.sid
LEFT OUTER JOIN sys.dm_os_tasks ota ON es.session_id = ota.session_id
LEFT OUTER JOIN sys.dm_os_threads oth ON ota.worker_address = oth.worker_address
CROSS APPLY sys.dm_exec_sql_text(er.sql_handle) AS st
where es.is_user_process = 1
and es.session_id <> ##spid
and es.status = 'running'
ORDER BY es.session_id
end
GO
this procedure has done very good for me in the last couple of years.
to run it just type sp_radhe
Regarding putting sp_radhe in the master database
I use the following code and make it a system stored procedure
exec sys.sp_MS_marksystemobject 'sp_radhe'
as you can see on the link below
Creating Your Own SQL Server System Stored Procedures
Regarding the transaction isolation level
Questions About T-SQL Transaction Isolation Levels You Were Too Shy to Ask
Jonathan Kehayias
Once you change the transaction isolation level it only changes when
the scope exits at the end of the procedure or a return call, or if
you change it explicitly again using SET TRANSACTION ISOLATION LEVEL.
In addition the TRANSACTION ISOLATION LEVEL is only scoped to the
stored procedure, so you can have multiple nested stored procedures
that execute at their own specific isolation levels.
This should give you all the details of the existing locks.
DECLARE #tblVariable TABLE(SPID INT, Status VARCHAR(200), [Login] VARCHAR(200), HostName VARCHAR(200),
BlkBy VARCHAR(200), DBName VARCHAR(200), Command VARCHAR(200), CPUTime INT,
DiskIO INT, LastBatch VARCHAR(200), ProgramName VARCHAR(200), _SPID INT,
RequestID INT)
INSERT INTO #tblVariable
EXEC Master.dbo.sp_who2
SELECT v.*, t.TEXT
FROM #tblVariable v
INNER JOIN sys.sysprocesses sp ON sp.spid = v.SPID
CROSS APPLY sys.dm_exec_sql_text(sp.sql_handle) AS t
ORDER BY BlkBy DESC, CPUTime DESC
You can then kill, with caution, the SPID that blocks your table.
kill 104 -- Your SPID
You can also use sp_who2 which gives more information
Here is some info http://dbadiaries.com/using-sp_who2-to-help-with-sql-server-troubleshooting
As per the official docs the sp_lock is mark as deprecated:
This feature is in maintenance mode and may be removed in a future
version of Microsoft SQL Server. Avoid using this feature in new
development work, and plan to modify applications that currently use
this feature.
and it is recommended to use sys.dm_tran_locks instead. This dynamic management object returns information about currently active lock manager resources. Each row represents a currently active request to the lock manager for a lock that has been granted or is waiting to be granted.
It generally returns more details in more user friendly syntax then sp_lock does.
The whoisactive routine written by Adam Machanic is very good to check the current activity in your environment and see what types of waits/locks are slowing your queries. You can very easily find what is blocking your queries and tons of other handy information.
For example, let's say we have the following queries running in the default SQL Server Isolation Level - Read Committed. Each query is executing in separate query window:
-- creating sample data
CREATE TABLE [dbo].[DataSource]
(
[RowID] INT PRIMARY KEY
,[RowValue] VARCHAR(12)
);
INSERT INTO [dbo].[DataSource]([RowID], [RowValue])
VALUES (1, 'samle data');
-- query window 1
BEGIN TRANSACTION;
UPDATE [dbo].[DataSource]
SET [RowValue] = 'new data'
WHERE [RowID] = 1;
--COMMIT TRANSACTION;
-- query window 2
SELECT *
FROM [dbo].[DataSource];
Then execute the sp_whoisactive (only part of the columns are displayed):
You can easily seen the session which is blocking the SELECT statement and even its T-SQL code. The routine has a lot of parameters, so you can check the docs for more details.
If we query the sys.dm_tran_locks view we can see that one of the session is waiting for a share lock of a resource, that has exclusive lock by other session:
Plot twist!
You can have orphaned distributed transactions holding exclusive locks and you will not see them if your script assumes there is a session associated with the transaction (there isn't!). Run the script below to identify these transactions:
;WITH ORPHANED_TRAN AS (
SELECT
dat.name,
dat.transaction_uow,
ddt.database_transaction_begin_time,
ddt.database_transaction_log_bytes_reserved,
ddt.database_transaction_log_bytes_used
FROM
sys.dm_tran_database_transactions ddt,
sys.dm_tran_active_transactions dat,
sys.dm_tran_locks dtl
WHERE
ddt.transaction_id = dat.transaction_id AND
dat.transaction_id = dtl.request_owner_id AND
dtl.request_session_id = -2 AND
dtl.request_mode = 'X'
)
SELECT DISTINCT * FROM ORPHANED_TRAN
Once you have identified the transaction, use the transaction_uow column to find it in MSDTC and decide whether to abort or commit it. If the transaction is marked as In Doubt (with a question mark next to it) you will probably want to abort it.
You can also kill the Unit Of Work (UOW) by specifying the transaction_uow in the KILL command:
KILL '<transaction_uow>'
References:
https://learn.microsoft.com/en-us/sql/t-sql/language-elements/kill-transact-sql?view=sql-server-2017#arguments
https://www.mssqltips.com/sqlservertip/4142/how-to-kill-a-blocking-negative-spid-in-sql-server/
A colleague and I have created a tool just for this.
It's a visual representation of all the locks that your sessions produce.
Give it a try (http://www.sqllockfinder.com), it's open source (https://github.com/LucBos/SqlLockFinder)