How to ignore duplicate records in CTE Select statement? - sql

Question

I am trying to ignore duplicate records in CTE but I am not able to do that, It seems like a SELECT statement inside CTE does not allow to use ROWNUM() variable numrows to condition in WHERE clause as it is showing Invalid column name 'numrows' error while trying to do so.
SQL Query:
DECLARE #BatchID uniqueidentifier = NEWID();
DECLARE #ClusterID SMALLINT = 1;
DECLARE #BatchSize integer = 20000;
DECLARE #myTableVariable TABLE(EventID BIGINT,HotelID int, BatchStatus varchar(50),BatchID uniqueidentifier);
WITH PendingExtResSvcEventsData_Batch
AS(
SELECT TOP (#BatchSize) t.EventID, t.HotelID, t.BatchStatus, t.BatchID, ROW_NUMBER() OVER (PARTITION BY t.EventID ORDER BY t.EventID) numrows
FROM ExtResSvcPendingMsg t WITH (NOLOCK)
WHERE t.ClusterID = #ClusterID AND numrows = 1 AND NOT EXISTS -- not allowed to use WHERE numrows = 1 here showing *Invalid Column Name*
(select 1 from ExtResSvcPendingMsg t2 where t2.BatchStatus = 'Batched'
and t2.EventID = t.EventID and t2.HotelID = t.HotelID)
)
UPDATE PendingExtResSvcEventsData_Batch
SET BatchStatus='Batched',
BatchID = #BatchID
-- WHERE numrows = 1 (not allowed to use WHERE here because of OUTPUT Clause)
OUTPUT INSERTED.* INTO #myTableVariable
SELECT e.ExtResSvcEventID,e.HotelID,e.ID1,e.ID2,e.ExtResSvcEventType,e.HostID,e.StatusCode,e.ChannelID,e.RequestAtTime,e.ProcessTime,e.DateBegin,e.DateEnd,
e.StatusMsg,em.MsgBodyOut,em.MsgBodyIn,e.ChannelResID
FROM ExtResSvcEvent e WITH (NOLOCK)
INNER JOIN #myTableVariable t ON e.ExtResSvcEventID = t.EventID
INNER JOIN ExtResSvcEventXML em with (nolock) on t.EventID = em.ExtResSvcEventID
ORDER BY e.ExtResSvcEventID
I have also tried to use numrows in final SELECT like INNER JOIN #myTableVariable t ON e.ExtResSvcEventID = t.EventID AND t.numrows = 1 but this gives me a error i.e. The column reference "inserted.numrows" is not allowed because it refers to a base table that is not being modified in this statement.
How do I ignore the duplicate records while using SELECT in CTE?

Answer 1

You can't refer to the numrows column in the WHERE clause of the CTE because that column is not calculated at this point in the plan execution. You need to add a second CTE with a select statement where you can refer to the numrows column:
WITH Base AS (
SELECT TOP (#BatchSize) t.EventID, t.HotelID, t.BatchStatus, t.BatchID, ROW_NUMBER() OVER (PARTITION BY t.EventID ORDER BY t.EventID) numrows
FROM ExtResSvcPendingMsg t WITH (NOLOCK)
WHERE t.ClusterID = #ClusterID
AND NOT EXISTS (select 1 from ExtResSvcPendingMsg t2 where t2.BatchStatus = 'Batched' and t2.EventID = t.EventID and t2.HotelID = t.HotelID)
), PendingExtResSvcEventsData_Batch AS (
SELECT EventID,
HotelID,
BatchStatus,
BatchID
WHERE numrows = 1
)
UPDATE...
I can't vouch for the update statement working as you expect it but the PendingExtResSvcEventsData_Batch should now have one row per EventID.