Establishing business hierarchy in log files – replace CURSOR by something else? - sql

Question

I'm using SQL Server as a warehouse to analyze log files. Those log files carry a kind of bussiness hierarchy (worker in this example):
Log Entry Id, Log Message
1 , Start Worker
2 , Do Cool Stuff
3 , Start Worker
4 , Do further cool stuff
5 , Start Worker
6 , This is a lot of working
7 , End worker
8 , End worker
9 , End worker
I need to relate the log entries to the current worker. The rule is quite simple: once a "Start worker" message is found, assign all following log entries to this worker. In the example hierarchy this means:
Log Entry Id, Log Message , Worker
1 , Start Worker , 1 (we take the entry id as worker id)
2 , Do Cool Stuff , 1
3 , Start Worker , 3
4 , Do further cool stuff , 3
5 , Start Worker , 5
6 , This is a lot of working , 5
7 , End worker , 5
8 , End worker , 3
9 , End worker , 1
Currently I'm using a stored procedure iterating all log entries with a cursor which basically uses a stack to establish the relationship between log entries and workers:
CREATE PROCEDURE CalculateRelations
AS
BEGIN
DECLARE entries_cur CURSOR FOR
SELECT Id, LogMessage
FROM LogEntries
ORDER BY Id;
DECLARE #Id BIGINT;
DECLARE #LogMessage VARCHAR(128);
DECLARE #ParentWorker BIGINT;
DECLARE #WorkerStack VARCHAR(MAX) = '';
OPEN entries_cur;
FETCH NEXT FROM entries_cur INTO #Id, #LogMessage;
WHILE ##FETCH_STATUS = 0
BEGIN
EXEC dbo.GetParentWorker #WorkerStack OUT, #Id, #LogMessage, #ParentWorker OUT;
UPDATE LogEntries
SET ParentWorker = #ParentWorker
WHERE Id = #Id;
FETCH NEXT FROM entries_cur INTO #Id, #LogMessage;
END;
CLOSE entries_cur;
DEALLOCATE entries_cur;
END;
GO
GetParentWorker is a stored procedure which uses the given VARCHAR variable WorkerStack as a stack. This means
"Start worker" message leads to adding (push) the Id to that VARCHAR
"End worker" message leads to removing and returning (pop) the last Id from that VARCHAR
all other messages lead to just returning (read) the last Id from that VARCHAR without modifying it
Now I'm wondering if it's possible to replace this cursor construct by a UPDATE statement. I'm not that deep in SQL and SQL Server, but might it be possible to realize this by dynamic variable assignment, CASE and the usage of the return value of GetParentWorker?

Answer 1

I think this is similar to Ian's, but I'll post for a slightly different approach to the indent level. I think you definitely want to put that indent level into the table with some indexing or this is going to be slow on large tables.
I'm using a CTE to calculate the indent level (basically just adding and subtracting one whenever we hit start or end, using a window function on preceding rows with a special case on ending worker in the current row). Outside of this toy solution, you'd want to limit the preceding rows to rows without an assigned worker and also rows up to the last time the level was zero before that.
Then we can just find the prior 'Start Worker' with the same level. These could probably be marked in pre-processing and indexed for quicker lookups.
UPDATE:
Simplified update statement by introducing window function CTE to calculate the worker id. This should reduce individual row lookups and improve performance in the update. See SQL Fiddle
WITH
WorkerNestingLevel AS (
SELECT
AuditLog.LogId
, AuditLog.LogMessage
, SUM( CASE LogMessage WHEN 'Start Worker' THEN 1 WHEN 'End Worker' THEN -1 ELSE 0 END ) OVER (ORDER BY LogId ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)
+ CASE LogMessage WHEN 'End Worker' THEN 1 ELSE 0 END AS [WorkerLevel]
FROM
AuditLog
)
, WorkerBatch AS (
SELECT
WorkerNestingLevel.LogId
, MAX( CASE WorkerNestingLevel.LogMessage WHEN 'Start Worker' THEN WorkerNestingLevel.LogId ELSE NULL END) OVER (PARTITION BY WorkerNestingLevel.WorkerLevel ORDER BY WorkerNestingLevel.LogId ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS WorkerId
FROM
WorkerNestingLevel
)
UPDATE
AuditLog
SET
WorkerId = WorkerBatch.WorkerId
FROM
AuditLog
JOIN
WorkerBatch ON (WorkerBatch.LogID = AuditLog.LogId);

Answer 2

My apologies for misunderstanding at the first attempt, hopefully I've understood this time that each 'End Worker' value cancels out one of the 'Start Worker's that precedes it. Here it is using a WITH statement that generates a dataset with a field called indent which you need to establish how far back to look for the correct [Log Entry ID]. Does that meet the requirement?
WITH indenttable AS (SELECT [Log Entry ID]
, [Log Message]
, ((SELECT COUNT(*)
FROM yourtable y2
WHERE [Log Message]='Start Worker'
AND y2.[Log Entry ID]<=yourtable.[Log Entry ID])
-(SELECT COUNT(*)
FROM yourtable y2
WHERE [Log Message]='End Worker'
AND y2.[Log Entry ID]<yourtable.[Log Entry ID])) indent
FROM yourtable)
UPDATE yourtable
SET worker=(
SELECT TOP(1) [Log Entry ID]
FROM indenttable y2
WHERE [Log Message]='Start Worker'
AND y2.[Log Entry ID]<=indenttable.[Log Entry ID]
AND y2.indent<=indenttable.indent
ORDER BY [Log Entry ID] DESC)
FROM indenttable JOIN yourtable ON indenttable.[Log Entry ID]=yourtable.[Log Entry ID];