DELETE all with ORDER BY - sql

Question

I'm building a poor's man FIFO message queue based on tables and trying to implement a receiveAll operation where all messages currently in the queue would be retrieved.
To receive a single message, I do:
WITH receiveCte AS (
SELECT TOP 1 body
FROM MyQueue WITH (ROWLOCK, READPAST)
ORDER BY id
)
DELETE FROM receiveCte
OUTPUT deleted.body;
From what I understand, the ORDER BY clause is necessary to guarantee the delete order, even if id is an identity primary key with a clustered index.
Now, to perform the receiveAll operation I need to delete all rows ORDER BY id and obviously that doesn't work without a TOP clause.
Therefore, I was thinking of performing a SELECT of the rows not locked and lock these for the entire transaction, then go on with the DELETE, however it seems I can't find a way to lock the rows affected by the select for the entire transaction.
BEGIN TRAN
DECLARE #msgCount int;
SELECT #msgCount = COUNT(*)
FROM MyQueue WITH (UPDLOCK, ROWLOCK, READPAST);
...
COMMIT TRAN
If I execute the above except the COMMIT TRAN and then execute the following statement in another connection, it still returns all rows while I expected to return 0 because of READPAST and the fact that there's an ongoing transaction holding UPDLOCK on the rows.
SELECT COUNT(*)
FROM MyQueue WITH (READPAST)
Obviously, I must be doing something wrong...
EDIT #1:
#king.code already gave the perfect answer in this case, however I found out what was going on.
It turns out that COUNT(*) seems to be ignoring the lock hints so it wasn't adequate for testing.
Also, it seems that you need an XLOCK to make sure that READPAST does it's job.
EDIT #2:
WARNING: SELECT TOP 100 PERCENT ... ORDER BY doesn't work because SQL Server seems to disregard the ORDER BY clause in that case. However, it seems that we can trick the optimizer using a variable e.g. SELECT TOP (#hundred) PERCENT, but I'm not sure how reliable that is.

Answer 1

I tried this and it worked for me:
**EDIT **
Updated based upon this technet article which states:
"If you have to use TOP to delete rows in a meaningful chronological order, you must use TOP together with ORDER BY in a subselect statement."
Setup
CREATE TABLE MyQueue
(
ID INT Identity Primary Key,
Body VARCHAR(MAX)
)
INSERT INTO MyQueue
VALUES ('Message 1'),
('Message 2'),
('Message 3'),
('Message 4'),
('Message 5'),
('Message 6'),
('Message 7'),
('Message 8'),
('Message 9'),
('Message 10'),
('Message 11')
In one query analyser window I performed this:
Query session 1
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
GO
Begin Tran
;WITH receiveCte AS (
SELECT TOP 1 body
FROM MyQueue WITH (READPAST)
ORDER BY id
)
DELETE FROM receiveCte WITH (ROWLOCK)
OUTPUT deleted.body;
-- note no COMMIT
Query Session 2
--Second window
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
GO
BEGIN TRAN
DELETE FROM MYQueue WITH (ROWLOCK)
WHERE ID IN
(
SELECT TOP 100 PERCENT ID
FROM MyQueue WITH (READPAST)
ORDER BY Id
)
OUTPUT deleted.Body
-- Note again no COMMIT
and then in the third window:
Query Session 3
SELECT COUNT(*)
FROM MyQueue WITH (READPAST)
which correctly returned a result of 0

Answer 2

From DB point of view there is no big sense in deleting rows in any particular order if you are going to delete them all. Simple DELETE without ordering is just fine.
If you are going to process row-by-row from application side, then start serializable transaction, block entire table and process\delete row-by-row based on ID, no ordering required.

Answer 3

From what I understand, the ORDER BY clause is necessary to guarantee the delete order, even if id is an identity primary key with a clustered index.
You're right.
Now, to perform the receiveAll operation I need to delete all rows ORDER BY id and obviously that doesn't work without a TOP clause.
Remember that you can use PERCENT in TOP:
DECLARE #Hundred FLOAT = 100;
SELECT TOP (#Hundred) PERCENT body
FROM MyQueue WITH (ROWLOCK, READPAST)
ORDER BY id;
UPDATE:
I've just made some test. And it looks like that if I ORDER BY the Clustered Index, I get the same execution plan with and without TOP(100) PERCENT.
If I ORDER BY another column I see the Sort operation in execution plan even if I place TOP(100) PERCENT. So it looks like it is not ignored.
Anyway, since the #Hundred variable and the TOP expression are FLOAT, you can try to set it to something like this 99.99999 and see what happens.

Answer 4

Try this using Broker Queues:
USE MASTER
CREATE DATABASE SBTest
GO
ALTER DATABASE SBTest SET ENABLE_BROKER;
GO
USE SBTest
GO
CREATE Message TYPE MyMessage
VALIDATION = NONE
GO
CREATE CONTRACT MyContract (MyMessage SENT BY INITIATOR)
GO
CREATE QUEUE MYSendQueue
GO
CREATE QUEUE MyReceiveQueue
GO
CREATE SERVICE MySendService
ON QUEUE MySendQueue (MyContract)
GO
CREATE SERVICE MyReceiveService
ON QUEUE MyReceiveQueue (MyContract)
GO
-- Send Messages
DECLARE #MyDialog uniqueidentifier
DECLARE #MyMessage NVARCHAR(128)
BEGIN DIALOG CONVERSATION #MyDialog
FROM SERVICE MySendService
TO SERVICE 'MyReceiveService'
ON CONTRACT MyContract
WITH ENCRYPTION = OFF
-- Send messages on Dialog
SET #MyMessage = N'My First Message';
SEND ON CONVERSATION #MyDialog
MESSAGE TYPE MyMessage (#MyMessage)
SET #MyMessage = N'My Second Message';
SEND ON CONVERSATION #MyDialog
MESSAGE TYPE MyMessage (#MyMessage)
SET #MyMessage = N'My Third Message';
SEND ON CONVERSATION #MyDialog
MESSAGE TYPE MyMessage (#MyMessage)
GO
-- View messages from Receive Queue
SELECT CONVERT(NVARCHAR(MAX), message_body) AS Message
FROM MyReceiveQueue
GO
-- Receive 1 message from Queue
RECEIVE TOP(1) CONVERT(NVARCHAR(MAX), message_body) AS Message
FROM MyReceiveQueue
GO
-- Receive All messages from Receive Queue
RECEIVE CONVERT(NVARCHAR(MAX), message_body) AS Message
FROM MyReceiveQueue
GO
-- Clean Up
USE master
GO
DROP DATABASE SBTest
GO

Answer 5

I believe SQL Server is determining that the action within the transaction does not require it to lock the rows that are "in use". I attempted to lock the table with something more obvious:
OPEN TRANSACTION
UPDATE MyQueue SET body = body
But even with that, it would still return the rows with the READPAST hint. However, if I actually change the rows:
OPEN TRANSACTION
UPDATE MyQueue SET body = body + '.'
Only then would my SELECT statement with READPAST return 0 rows. Seems that MSSQL is intelligent enough to minimize the locks on your table!
I suggest that if you want to hide rows from a READPAST hint, add a new column that can actually be edited without the fear of changing important data, and then in the process that locks the rows, update that row with an actual data change:
ALTER TABLE MyQueue ADD LockRow bit DEFAULT(0)
...
BEGIN TRANSACTION
UPDATE MyQueue SET LockRow = 1
If you do the above, your READPAST query should return 0 rows.

Answer 6

If you simply need to delete all of the records as of now without blocking inserts and deletes that happen concurrently, you should simply issue this command:
DELETE FROM MyQueue WITH (ROWLOCK, READPAST)
OUTPUT deleted.id, deleted.body;
This won't block neither inserts into MyQueue table, nor executions of the same statement concurrently. Concurrent executions will only pick up the records that were inserted after the previous DELETE transaction start time. Similarly, there is no need to do any ORDER BY, since the subject of deletion will be all records that existed in the table at the transaction start time.
Also I must mention that I strongly recommend not using the ROWLOCK hint, and let the SQL server decide which lock level to use for the top efficiency.

Answer 7

Have you tried issuing a HOLDLOCK hint, when selecting the rows? This should ensure that no other query can select the rows, until after the transaction is finished:
BEGIN TRAN
DECLARE #msgCount int;
SELECT #msgCount = COUNT(*)
FROM MyQueue WITH (HOLDLOCK, ROWLOCK, READPAST);
...
COMMIT TRAN

Answer 8

According to MSDN Output documentation:
There is no guarantee that the order in which the changes are applied
to the table and the order in which the rows are inserted into the
output table or table variable will correspond.
You can however insert the results of OUTPUT into a table variable, and then select back the results while keeping them ordered. Please note that using rowlock will require more resources if there are many records in the queue.
DECLARE #processQueue
TABLE
( id int NOT NULL
, body nvarchar(max) /*<- set to appropriate data type!*/
);
DELETE q
OUTPUT deleted.id, deleted.body
INTO #processQueue(id, body)
FROM MyQueue q WITH (ROWLOCK, READPAST)
;
SELECT body
FROM #processQueue q
ORDER BY q.id --ensure output ordering here.
;
Also since you have mentioned you do not want table locks, you can disable lock escalation on the MyQueue table.
--Disable lock escalation, to ensure that locks do get escalated to table locks.
ALTER TABLE MyQueue SET ( LOCK_ESCALATION = DISABLE );
go