Materializing the path of Nested Set hierarchy in T-SQL - sql
I have a table containing details on my company's chart of accounts - this data is essentially stored in nested sets (on SQL Server 2014), with each record having a left and right anchor - there are no Parent IDs.
Sample Data:
ID LeftAnchor RightAnchor Name
1 0 25 Root
2 1 16 Group 1
3 2 9 Group 1.1
4 3 4 Account 1
5 5 6 Account 2
6 7 8 Account 3
7 10 15 Group 1.2
8 11 12 Account 4
9 13 14 Account 5
10 17 24 Group 2
11 18 23 Group 2.1
12 19 20 Account 1
13 21 22 Account 1
I need to materialize the path for each record, so that my output looks like this:
ID LeftAnchor RightAnchor Name MaterializedPath
1 0 25 Root Root
2 1 16 Group 1 Root > Group 1
3 2 9 Group 1.1 Root > Group 1 > Group 1.1
4 3 4 Account 1 Root > Group 1 > Group 1.1 > Account 1
5 5 6 Account 2 Root > Group 1 > Group 1.1 > Account 2
6 7 8 Account 3 Root > Group 1 > Group 1.1 > Account 3
7 10 15 Group 1.2 Root > Group 1 > Group 1.2
8 11 12 Account 4 Root > Group 1 > Group 1.2 > Acount 4
9 13 14 Account 5 Root > Group 1 > Group 1.2 > Account 5
10 17 24 Group 2 Root > Group 2
11 18 23 Group 2.1 Root > Group 2 > Group 2.1
12 19 20 Account 1 Root > Group 2 > Group 2.1 > Account 10
13 21 22 Account 1 Root > Group 2 > Group 2.1 > Account 11
Whilst I've managed to achieve this using CTEs, the query is deathly slow. It takes just shy of two minutes to run with around 1200 records in the output.
Here's a simplified version of my code:
;with accounts as
(
-- Chart of Accounts
select AccountId, LeftAnchor, RightAnchor, Name
from ChartOfAccounts
-- dirty great where clause snipped
)
, parents as
(
-- Work out the Parent Nodes
select c.AccountId, p.AccountId [ParentId]
from accounts c
left join accounts p on (p.LeftAnchor = (
select max(i.LeftAnchor)
from accounts i
where i.LeftAnchor<c.LeftAnchor
and i.RightAnchor>c.RightAnchor
))
)
, path as
(
-- Calculate the Account path for each node
-- Root Node
select c.AccountId, c.LeftAnchor, c.RightAnchor, 0 [Level], convert(varchar(max), c.name) [MaterializedPath]
from accounts c
where c.LeftAnchor = (select min(LeftAnchor) from chart)
union all
-- Children
select n.AccountId, n.LeftAnchor, n.RightAnchor, p.level+1, p.path + ' > ' + n.name
from accounts n
inner join parents x on (n.AccountId=x.AccountId)
inner join path p on (x.ParentId=p.AccountId)
)
select * from path order by LeftAnchor
Ideally this query should only take a couple of seconds (max) to run. I can't make any changes to the database itself (read-only connection), so can anyone come up with a better way to write this query?
After your comments, I realized no need for the CTE... you already have the range keys.
Example
Select A.*
,Path = Replace(Path,'>','>')
From YourTable A
Cross Apply (
Select Path = Stuff((Select ' > ' +Name
From (
Select LeftAnchor,Name
From YourTable
Where A.LeftAnchor between LeftAnchor and RightAnchor
) B1
Order By LeftAnchor
For XML Path (''))
,1,6,'')
) B
Order By LeftAnchor
Returns
First you can try to rearrange your preparing CTEs (accounts and parents) to have it that each CTE contains all data from previous, so you only use the last one in path CTE - no need for multiple joins:
;with accounts as
(
-- Chart of Accounts
select AccountId, LeftAnchor, RightAnchor, Name
from ChartOfAccounts
-- dirty great where clause snipped
)
, parents as
(
-- Work out the Parent Nodes
select c.*, p.AccountId [ParentId]
from accounts c
left join accounts p on (p.LeftAnchor = (
select max(i.LeftAnchor)
from accounts i
where i.LeftAnchor<c.LeftAnchor
and i.RightAnchor>c.RightAnchor
))
)
, path as
(
-- Calculate the Account path for each node
-- Root Node
select c.AccountId, c.LeftAnchor, c.RightAnchor, 0 [Level], convert(varchar(max), c.name) [MaterializedPath]
from parents c
where c.ParentID IS NULL
union all
-- Children
select n.AccountId, n.LeftAnchor, n.RightAnchor, p.level+1, p.[MaterializedPath] + ' > ' + n.name
from parents n
inner join path p on (n.ParentId=p.AccountId)
)
select * from path order by LeftAnchor
This should give some improvement (50% in my test), but to have it really better, you can split first half of preparing data into #temp table, put clustered index on ParentID column in #temp table and use it in second part
if (Object_ID('tempdb..#tmp') IS NOT NULL) DROP TABLE #tmp;
with accounts as
(
-- Chart of Accounts
select AccountId, LeftAnchor, RightAnchor, Name
from ChartOfAccounts
-- dirty great where clause snipped
)
, parents as
(
-- Work out the Parent Nodes
select c.*, p.AccountId [ParentId]
from accounts c
left join accounts p on (p.LeftAnchor = (
select max(i.LeftAnchor)
from accounts i
where i.LeftAnchor<c.LeftAnchor
and i.RightAnchor>c.RightAnchor
))
)
select * into #tmp
from parents;
CREATE CLUSTERED INDEX IX_tmp1 ON #tmp (ParentID);
With path as
(
-- Calculate the Account path for each node
-- Root Node
select c.AccountId, c.LeftAnchor, c.RightAnchor, 0 [Level], convert(varchar(max), c.name) [MaterializedPath]
from #tmp c
where c.ParentID IS NULL
union all
-- Children
select n.AccountId, n.LeftAnchor, n.RightAnchor, p.level+1, p.[MaterializedPath] + ' > ' + n.name
from #tmp n
inner join path p on (n.ParentId=p.AccountId)
)
select * from path order by LeftAnchor
Hard to tell on small sample data, but it should be an improvement. Please tell if you try it.
Seems odd to me that you don't have a Parent ID, but with the aid of an initial OUTER APPLY, we can generate a Parent ID and then run a standard recursive CTE.
Example
Declare #Top int = null --<< Sets top of Hier Try 12 (Just for Fun)
;with cte0 as (
Select A.*
,B.*
From YourTable A
Outer Apply (
Select Top 1 Pt=ID
From YourTable
Where A.LeftAnchor between LeftAnchor and RightAnchor and LeftAnchor<A.LeftAnchor
Order By LeftAnchor Desc
) B
)
,cteP as (
Select ID
,Pt
,LeftAnchor
,RightAnchor
,Lvl=1
,Name
,Path = cast(Name as varchar(max))
From cte0
Where IsNull(#Top,-1) = case when #Top is null then isnull(Pt ,-1) else ID end
Union All
Select r.ID
,r.Pt
,r.LeftAnchor
,r.RightAnchor
,p.Lvl+1
,r.Name
,cast(p.path + ' > '+r.Name as varchar(max))
From cte0 r
Join cteP p on r.Pt = p.ID
)
Select *
From cteP
Order By LeftAnchor
Returns
Related
Bottom Up Recursive SUM (lowest level only has values)
I have a tree-based structure of SKUs within a Product Hierarchy in SQL Server. The lowest level SKUs will only ever have values (these are consumption values). I then want to generate aggregates up the hierarchy at every level. Here's is the sample table structure: Id ParentId Name Volume IsSku 1 -1 All 0 0 2 1 Cat A 0 0 3 1 Cat B 0 0 4 2 Cat A.1 0 0 5 2 Cat A.2 0 0 6 3 Cat B.1 0 0 7 3 Cat B.2 0 0 8 4 SKU1 10 1 9 4 SKU2 5 1 10 5 SKU3 7 1 11 5 SKU4 4 1 12 6 SKU1 10 1 13 6 SKU2 5 1 14 7 SKU3 9 1 15 7 SKU4 7 1 I need a query that will start at the sku level (IsSku=1) and then working up, will sum the SKUs and carry the sum up the product category levels to get a cumulative running total. I've seen several queries where there are recursive sums in a hierarchical structure where each level already has values, but I need one that will start at the lowest level that has values and recursively calculate the sum as it moves upward. I was trying these, but they look like they are mainly summing hierarchical data where each node already has a value (Volume, in my case). I need to start at the lowest level and carry the aggregate up as I go up the hierarchy. I tried to emulate the answers in these posts with my data, but wasn't successful so far with my data setup. 24394601 29127163 11408878 The output for the query should be like so: Id ParentId Name Volume IsSku 1 -1 All 54 0 2 1 Cat A 26 0 3 1 Cat B 28 0 4 2 Cat A.1 15 0 5 2 Cat A.2 11 0 6 3 Cat B.1 12 0 7 3 Cat B.2 16 0 8 4 SKU1 10 1 9 4 SKU2 5 1 10 5 SKU3 7 1 11 5 SKU4 4 1 12 6 SKU1 10 1 13 6 SKU2 2 1 14 7 SKU3 9 1 15 7 SKU4 7 1 I've got a start with a recursive CTE that returns the hierarchy can can aggregate the volume if that node has volume already, but can't seem to figure out how to start at the SKU levels and continue aggregating up the hierarchy. Here's the start with my CTE: DECLARE #tblData TABLE ( [ID] INT NOT NULL, [ParentId] INT NULL, [Name] varchar(50) NOT NULL, [Volume] int NOT NULL, [IsSku] bit ) INSERT INTO #tblData VALUES (1,-1,'All',0,0) ,(2,1,'Cat A',0,0) ,(3,1,'Cat B',0,0) ,(4,2,'Cat A.1',0,0) ,(5,2,'Cat A.2',0,0) ,(6,3,'Cat B.1',0,0) ,(7,3,'Cat B.2',0,0) ,(8,4,'SKU1',10,1) ,(9,4,'SKU2',5,1) ,(10,5,'SKU3',7,1) ,(11,5,'SKU4',4,1) ,(12,6,'SKU1',10,1) ,(13,6,'SKU2',5,1) ,(14,7,'SKU3',7,1) ,(15,7,'SKU4',4,1) ;WITH cte AS ( SELECT a.ID ,a.ParentID ,a.Name ,a.Volume ,CAST('/' + cast(ID as varchar) + '/' as varchar) Node ,0 AS level ,IsSku FROM #tblData AS a WHERE a.ParentID = -1 UNION ALL SELECT b.ID ,b.ParentID ,b.Name ,b.Volume ,CAST(c.Node + CAST(b.ID as varchar) + '/' as varchar) ,level = c.level + 1 ,b.IsSku FROM #tblData AS b INNER JOIN cte c ON b.ParentId = c.ID ) SELECT c1.ID, c1.ParentID, c1.Name, c1.Node ,ISNULL(SUM(c2.Volume),0) FROM cte c1 LEFT OUTER JOIN cte c2 ON c1.Node <> c2.Node AND LEFT(c2.Node, LEN(c1.Node)) = c1.Node GROUP BY c1.ID, c1.ParentID, c1.Name, c1.Node Any help is appreciated!
This should do it: DECLARE #tbl TABLE(Id INT, ParentId INT, Name NVARCHAR(255), Volume INTEGER, IsSku BIT) INSERT INTO #tbl VALUES (1,-1,'All',0,0) ,(2,1,'Cat A',0,0) ,(3,1,'Cat B',0,0) ,(4,2,'Cat A.1',0,0) ,(5,2,'Cat A.2',0,0) ,(6,3,'Cat B.1',0,0) ,(7,3,'Cat B.2',0,0) ,(8,4,'SKU1',10,1) ,(9,4,'SKU2',5,1) ,(10,5,'SKU3',7,1) ,(11,5,'SKU4',4,1) ,(12,6,'SKU1',10,1) ,(13,6,'SKU2',5,1) ,(14,7,'SKU3',7,1) ,(15,7,'SKU4',4,1) SELECT * FROM #tbl ; WITH cte AS ( SELECT Id,ParentId, Name, Volume, IsSku, CAST(Id AS VARCHAR(MAX)) AS Hierarchy FROM #tbl WHERE ParentId=-1 UNION ALL SELECT t.Id,t.ParentId, t.Name, t.Volume, t.IsSku, CAST(c.Hierarchy + '|' + CAST(t.Id AS VARCHAR(MAX)) AS VARCHAR(MAX)) FROM cte c INNER JOIN #tbl t ON c.Id = t.ParentId ) SELECT Id,ParentId, Name, ChildVolume AS Volume, IsSku FROM ( SELECT c1.Id, c1.ParentId, c1.Name, c1. Volume, c1.IsSku, SUM(c2.Volume) AS ChildVolume FROM cte c1 LEFT JOIN cte c2 ON c2.Hierarchy LIKE c1.Hierarchy + '%' GROUP BY c1.Id, c1.ParentId, c1.Name, c1. Volume, c1.IsSku ) x Basically the computation happens in three steps: Capture the Hierarchy recursively for each descendant by concatenating the Ids: CAST(c.Hierarchy + '|' + CAST(t.Id AS VARCHAR(MAX)) AS VARCHAR(MAX)) Join the resulting table with itself so each record is joined with itself and all its descendants: FROM cte c1 LEFT JOIN cte c2 ON c2.Hierarchy LIKE c1.Hierarchy + '%' Finally aggregate the Volume of each hierarchy by grouping: SUM(c2.Volume) AS ChildVolume This is in reference to Ed Harper's answer to a similar question here: Hierarchy based aggregation
Due to the way that recursive CTEs work in SQL Server, it is very difficult to get this kind of logic working efficiently. It often either requires self-joining the whole resultset, or using something like JSON or XML. The problem is that at each recursion of the CTE, although it appears you are working on the whole set at once, it actually only feeds back one row at a time. Therefore grouping is disallowed over the recursion. Instead, it's much better to simply recurse with a WHILE loop and insert into a temp table or table variable, then read it back to aggregate Use the OUTPUT clauses to view the intermediate results DECLARE #tmp TABLE ( Id INTEGER, ParentId INTEGER, Name VARCHAR(7), Volume INTEGER, IsSku INTEGER, Level INT, INDEX ix CLUSTERED (Level, ParentId, Id) ); INSERT INTO #tmp (Id, ParentId, Name, Volume, IsSku, Level) -- OUTPUT inserted.Id, inserted.ParentId, inserted.Name, inserted.Volume, inserted.IsSku, inserted.Level SELECT p.Id, p.ParentId, p.Name, p.Volume, p.IsSku, 1 FROM Product p WHERE p.IsSku = 1; DECLARE #level int = 1; WHILE (1=1) BEGIN INSERT INTO #tmp (Id, ParentId, Name, Volume, IsSku, Level) -- OUTPUT inserted.Id, inserted.ParentId, inserted.Name, inserted.Volume, inserted.IsSku, inserted.Level SELECT p.Id, p.ParentId, p.Name, t.Volume, p.IsSku, #level + 1 FROM ( SELECT t.ParentID, Volume = SUM(t.Volume) FROM #tmp t WHERE t.Level = #level GROUP BY t.ParentID ) t JOIN Product p ON p.Id = t.ParentID; IF (##ROWCOUNT = 0) BREAK; SET #level += 1; END; SELECT * FROM #tmp ORDER BY Id; db<>fiddle This solution does involve a blocking operator, due to Halloween protection (in my case I saw an "unnecessary" sort). You can avoid it by using Itzik Ben-Gan's Divide and Conquer method, utilizing two table variables and flip-flopping between them.
Select non existing Numbers from Table each ID
I‘m new in learning TSQL and I‘m struggling getting the numbers that doesn‘t exist in my table each ID. Example: CustomerID Group 1 1 3 1 6 1 4 2 7 2 I wanna get the ID which does not exist and select them like this CustomerID Group 2 1 4 1 5 1 5 2 6 2 .... .. The solution by usin a cte doesn‘t work well or inserting first the data and do a not exist where clause. Any Ideas?
If you can live with ranges rather than a list with each one, then an efficient method uses lead(): select group_id, (customer_id + 1) as first_missing_customer_id, (next_ci - 1) as last_missing_customer_id from (select t.*, lead(customer_id) over (partition by group_id order by customer_id) as next_ci from t ) t where next_ci <> customer_id + 1
Cross join 2 recursive CTEs to get all the possible combinations of [CustomerID] and [Group] and then LEFT join to the table: declare #c int = (select max([CustomerID]) from tablename); declare #g int = (select max([Group]) from tablename); with customers as ( select 1 as cust union all select cust + 1 from customers where cust < #c ), groups as ( select 1 as gr union all select gr + 1 from groups where gr < #g ), cte as ( select * from customers cross join groups ) select c.cust as [CustomerID], c.gr as [Group] from cte c left join tablename t on t.[CustomerID] = c.cust and t.[Group] = c.gr where t.[CustomerID] is null and c.cust > (select min([CustomerID]) from tablename where [Group] = c.gr) and c.cust < (select max([CustomerID]) from tablename where [Group] = c.gr) See the demo. Results: > CustomerID | Group > ---------: | ----: > 2 | 1 > 4 | 1 > 5 | 1 > 5 | 2 > 6 | 2
get count of reference id and it is their child recursively
I have the below table( I need them for oracle and sql server): id id_reference 1 0 2 1 3 1 4 1 6 2 7 2 8 3 9 8 10 0 11 10 12 10 13 12 I want to get the count of the id_reference for each id. the result id count(1) 1 7 -- because id 1 2 3 4 and the child 6 7 8 9 are referring to the id 2 2 -- because id 6 and 7 are referring to it 3 2 -- because id 8 and the child 9 referring to it 4 0 -- non are referring to them 6 0 -- non are referring to them 7 0 -- non are referring to them 8 1 -- because 9 is referring to the id 10 3 -- because 11 , 12 and 13 are referring 11 0 -- none are referring 12 1 -- 13 is referring to id 13 0 -- none is referring to id this what I tried but I need it to be recursive. select count(1), id, (select count(1) from tab e2 where e2.id <=e1.id and id_ref in ( select id from tab e3 where e3.id_ref= e2.id ) from tab e1 group by id order by id desc
Oracle version: dbfiddle demo select distinct id, nvl(cnt, 0) from tab left join ( select root, count(1) cnt from ( select tab.*, connect_by_root(id) root from tab where level > 1 connect by id_reference = prior id) group by root) r on root = tab.id order by id
In SQL Server (2016+), this is how I'd achieve the above result set: USE Sandbox; GO WITH VTE AS( SELECT * FROM (VALUES (1 ,0 ), (2 ,1 ), (3 ,1 ), (4 ,1 ), (6 ,2 ), (7 ,2 ), (8 ,3 ), (9 ,8 ), (10,0 ), (11,10), (12,10), (13,12)) V(ID, ID_ref)), CTE AS ( SELECT ID, CONVERT(varchar(30),CONVERT(varchar(4),ID)) AS Delimited FROM VTE V WHERE V.ID_ref = 0 UNION ALL SELECT V.ID, CONVERT(varchar(30),CONCAT(C.Delimited,',' + CONVERT(varchar(4),V.ID))) FROM CTE C JOIN VTE V ON V.ID_ref = C.ID), Splits AS( SELECT C.ID, SS.value FROM CTE C CROSS APPLY STRING_SPLIT(C.Delimited,',') SS) SELECT V.ID, COUNT(S.ID) - 1 AS [Count] FROM VTE V JOIN Splits S ON S.[value] = V.ID GROUP BY V.ID; This firstly creates a delimited list of each ID at each layer. It then splits them out and finally does a Count -1. If you aren't on SQL Server 2016+, then you can use a XML Splitter or delimitedsplit8k(_lead). Note that a rCTe will stop recursing at 100 loops. You'll need to use OPTION (MAXRECURSION N) to increase the loops (where N is a suitable number of the maximum layer you might have).
SQL query for tree table menu
I have a table with tree structure: ID Title ParentID Orderby ---------------------------------------- 1 All 0 2 2 Banking 1 5 3 USAA Checking 0 0 4 USAA Mastercard 1 9 5 Medical 3 0 6 Jobs 3 100 7 Archive 0 1 8 Active 7 0 9 BoA Amex 1 1 I need to write a SQL query to produce a result like this (ORDER by column Orderby): ID Title Path Orderby ---------------------------------------- 3 USAA Checking 1 0 5 Medical 1.1 0 6 Jobs 3.2 100 7 Archive 2 1 8 Active 2.1 0 1 All 3 2 9 BoA Amex 3.1 1 2 Banking 3.2 5 4 USAA Mastercard 3.3 9 Who can help me to write a SQL query? Thanks!
We can do this using a recursive CTE: WITH children AS ( SELECT NULL AS ParentID, ID, Title, Orderby, CAST(ID AS VARCHAR(500)) AS Path FROM Categories WHERE ParentID = 0 UNION ALL SELECT d.ParentID, t.counter + 1, d.ID, d.Title, d.Orderby, CAST(CAST(t.Path AS VARCHAR(50)) + '.' + CAST(ROW_NUMBER() OVER (PARTITION BY d.ParentID ORDER BY d.ID) AS VARCHAR(50)) AS VARCHAR(500)) FROM children t INNER JOIN Categories AS d ON d.ParentID = t.ID ) SELECT ID, Title, Path, Orderby FROM children; Demo Note that you never provided fixed logic for what should be used to determine the minor version numbers, for a given parent version. That is, it is not clear why Medical appears earlier than Jobs in the hierarchy.
You can try below using row_number() DEMO select Id,title, concat(val,'.',case when row_number() over(partition by val order by Id)-1=0 then null else row_number() over(partition by val order by Id)-1 end) as path, orderby from ( select *,case when parentid=0 then id else parentid end as val from Categories )A
You Can try Below query if you have one level of the hierarchy Select C.ID as ID, C.Title as Title, Case when C.ParentID =0 then cast(C.ID as varchar(2)) else cast(C.ParentID as varchar(2)) + '.' + cast(C.Order as varchar(3)) END as Path, C.Order from Categories as C You Need to create Temp tables if you have multiple level hierarchy. and you need to update order so that we have a simpler query for the desired output. Thanks
RECURSIVE CTE SQL - Find next available Parent
I have parent child relation SQL table LOCATIONDETAIL Table OID NAME PARENTOID 1 HeadSite 0 2 Subsite1 1 3 subsite2 1 4 subsubsite1 2 5 subsubsite2 2 6 subsubsite3 3 RULESETCONFIG OID LOCATIONDETAILOID VALUE 1 1 30 2 4 15 If i provide Input as LOCATIONDETAIL 6, i should get RULESETCONFIG value as 30 because for LOCATIONDETAIL 6, parentid is 3 and for LOCATIONDETAIL 3 there is no value in RULESETCONFIG, LOCATIONDETAIL 3 has parent 1 which has value in RULESETCONFIG if i provide Input as LOCATIONDETAIL 4, i should get RULESETCONFIG value 15 i have code to populate the tree, but don't know how to find the next available Parent ;WITH GLOBALHIERARCHY AS ( SELECT A.OID,A.PARENTOID,A.NAME FROM LOCATIONDETAIL A WHERE OID = #LOCATIONDETAILOID UNION ALL SELECT A.OID,A.PARENTOID,A.NAME FROM LOCATIONDETAIL A INNER JOIN GLOBALHIERARCHY GH ON A.PARENTOID = GH.OID ) SELECT * FROM GLOBALHIERARCHY
This will return the next parent with a value. If you want to see all, remove the top 1 from the final select. dbFiddle Example Declare #Fetch int = 4 ;with cteHB as ( Select OID ,PARENTOID ,Lvl=1 ,NAME From LOCATIONDETAIL Where OID=#Fetch Union All Select R.OID ,R.PARENTOID ,P.Lvl+1 ,R.NAME From LOCATIONDETAIL R Join cteHB P on P.PARENTOID = R.OID) Select Top 1 Lvl = Row_Number() over (Order By A.Lvl Desc ) ,A.OID ,A.PARENTOID ,A.NAME ,B.Value From cteHB A Left Join RULESETCONFIG B on A.OID=B.OID Where B.VALUE is not null and A.OID <> #Fetch Order By 1 Desc Returns when #Fetch=4 Lvl OID PARENTOID NAME Value 2 2 1 Subsite1 15 Returns when #Fetch=6 Lvl OID PARENTOID NAME Value 1 1 0 HeadSite 30
This should do the job: ;with LV as ( select OID ID,PARENTOID PID,NAME NAM, VALUE VAL FROM LOCATIONDETAIL left join RULESETCONFIG ON LOCATIONDETAILOID=OID ), GH as ( select ID gID,PID gPID,NAM gNAM,VAL gVAL from LV where ID=#OID union all select ID,PID,NAM,VAL FROM LV INNER JOIN GH ON gVAL is NULL AND gPID=ID ) select * from GH WHERE gVAL>0 See here for e little demo: http://rextester.com/OXD40496