Optimizing slow query with multiple withs - sql

Question

I have the following query:
with matched_urls as
(
select l.Url, min(f.urlfiltertype) as Filter
from landingpages l
join landingpageurlfilters lpf on lpf.landingpageid = l.Url
join urlfilters f on lpf.urlfilterid = f.id
where f.groupid = 3062
group by l.Url
),
all_urls as
(
select l.Url, 5 as Filter
from landingpages l
where 'iylsuqnzukwv0milinztea' in (select domainid
from domainlandingpages dlp
where l.Url = dlp.landingpageid)
and l.Url not in (select Url from matched_urls)
union
select * from matched_urls
)
select l.*
from landingpages l
join all_urls u on l.Url = u.Url
order by u.Filter asc
offset 0 rows fetch next 30 rows only
These are the tables used in the query:
And this is the DDL for the tables:
CREATE TABLE [dbo].[LandingPages]
(
[Url] [nvarchar](448) NOT NULL,
[LastUpdated] [datetime2](7) NOT NULL,
CONSTRAINT [PK_LandingPages]
PRIMARY KEY CLUSTERED ([Url] ASC)
WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO
CREATE TABLE [dbo].[DomainLandingPages]
(
[LandingPageId] [nvarchar](448) NOT NULL,
[DomainId] [nvarchar](128) NOT NULL,
CONSTRAINT [PK_DomainLandingPages]
PRIMARY KEY CLUSTERED ([DomainId] ASC, [LandingPageId] ASC)
WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[DomainLandingPages] WITH CHECK
ADD CONSTRAINT [FK_DomainLandingPages_Domains_DomainId]
FOREIGN KEY([DomainId]) REFERENCES [dbo].[Domains] ([Id])
GO
ALTER TABLE [dbo].[DomainLandingPages] CHECK CONSTRAINT [FK_DomainLandingPages_Domains_DomainId]
GO
ALTER TABLE [dbo].[DomainLandingPages] WITH CHECK
ADD CONSTRAINT [FK_DomainLandingPages_LandingPages_LandingPageId]
FOREIGN KEY([LandingPageId]) REFERENCES [dbo].[LandingPages] ([Url])
GO
ALTER TABLE [dbo].[DomainLandingPages] CHECK CONSTRAINT [FK_DomainLandingPages_LandingPages_LandingPageId]
GO
CREATE TABLE [dbo].[UrlFilters]
(
[Id] [int] IDENTITY(1,1) NOT NULL,
[GroupId] [int] NOT NULL,
[UrlFilterType] [int] NOT NULL,
[Filter] [nvarchar](max) NOT NULL,
CONSTRAINT [PK_UrlFilters]
PRIMARY KEY CLUSTERED ([Id] ASC)
WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
ALTER TABLE [dbo].[UrlFilters] WITH CHECK
ADD CONSTRAINT [FK_UrlFilters_Groups_GroupId]
FOREIGN KEY([GroupId]) REFERENCES [dbo].[Groups] ([Id])
ON DELETE CASCADE
GO
ALTER TABLE [dbo].[UrlFilters] CHECK CONSTRAINT [FK_UrlFilters_Groups_GroupId]
GO
CREATE TABLE [dbo].[LandingPageUrlFilters]
(
[LandingPageId] [nvarchar](448) NOT NULL,
[UrlFilterId] [int] NOT NULL,
CONSTRAINT [PK_LandingPageUrlFilters]
PRIMARY KEY CLUSTERED ([LandingPageId] ASC, [UrlFilterId] ASC)
WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[LandingPageUrlFilters] WITH CHECK
ADD CONSTRAINT [FK_LandingPageUrlFilters_LandingPages_LandingPageId]
FOREIGN KEY([LandingPageId]) REFERENCES [dbo].[LandingPages] ([Url])
GO
ALTER TABLE [dbo].[LandingPageUrlFilters] CHECK CONSTRAINT [FK_LandingPageUrlFilters_LandingPages_LandingPageId]
GO
ALTER TABLE [dbo].[LandingPageUrlFilters] WITH CHECK
ADD CONSTRAINT [FK_LandingPageUrlFilters_UrlFilters_UrlFilterId]
FOREIGN KEY([UrlFilterId]) REFERENCES [dbo].[UrlFilters] ([Id])
GO
ALTER TABLE [dbo].[LandingPageUrlFilters] CHECK CONSTRAINT [FK_LandingPageUrlFilters_UrlFilters_UrlFilterId]
GO
Here is the execution plan:
https://www.brentozar.com/pastetheplan/?id=H1tHt5pvP
The query is pulling all urls for a given domain and are then supposed to be ordered by UrlFilterType - however not all landing pages have a match, hence the two with clauses.
As far as I can see from the execution plan it's mainly doing index seeks, so I think I have the right indexes. However, the query takes very long to execute, so I hope there might be a smarter way of doing this.
Any input will be greatly appreciated!

Answer 1

First up, your statistics look wildly out of line. Estimated 3,700 rows, actual 219,000. That suggests, at the very least, a statistics update will possibly change the choices the optimizer is making. Because of those row estimates, the optimizer is choosing a seek and a nested loops operation where, based on the data distribution, you're reading 1/3 of the table, 200k rows of a 600k table. A scan here, probably with a hash join, would be more efficient.
The query itself isn't giving much to filter the [aarhus-cluster-onesearch-staging].[dbo].[LandingPages].[PK_LandingPages] table on. So it's pulling 200k rows in order to filter them down to 30. If you can find a way to additional filtering there, you should see a performance improvement.

Answer 2

OK, the big hit is at the bottom of that query plan - where it's reading from LandingPageUrlFilters where the URLfilterID comes from URLfilters.
It's getting completely the wrong estimates (out by 70x) and then sorts your URLs taking 30 seconds or so.
If you run the first CTE on its own, I think it will take a long time. That's what you need to optimise.
select l.Url, min(f.urlfiltertype) as Filter
from landingpages l
join landingpageurlfilters lpf on lpf.landingpageid = l.Url
join urlfilters f on lpf.urlfilterid = f.id
where f.groupid = 3062
group by l.Url
Suggestions
First thing to try is to also add a nonclustered index on LandingPageUrlFilters in the opposite order to your clustered index (e.g., CREATE NONCLUSTERED INDEX myindex ON LandingPageUrlFilters ([UrlFilterId] ASC, [LandingPageId] ASC). Note this will make a full copy of that table which may be rather large. It appears you already have a non-clustered index like this (based on the fact it's referring to IX_LandingPageUrlFilters_UrlFilterId)
The sort on the string field that is nvarchar(448) - which is actually close to 900 bytes per row - will take a much bigger memory grant. Consider adding ID int values as primary keys - it will require less memory and therefore less likely to spill to disk
Consider, instead of a CTE, create a temporary table (with appropriate PKs) of the section and LandingPageURLFilters and urlfilters. However, you will still need to do a sort when inserting these - which is likely to take just as long.
Part of the problem (another 10s or so) is a nested loop join to LandingPages. It was expecting less than 4000 rows (so a nested loop is OK) but had to do 220,000 loops. If necessary, consider a join hint (e.g., INNER HASH JOIN rather than INNER JOIN). However, it appears that landingpages isn't actually required in that query - just remove the table landingpages from the CTE, and use landingpageurlfilters.landingpageid
e.g.,
select lpf.landingpageid AS [Url], min(f.urlfiltertype) as Filter
from landingpageurlfilters lpf
join urlfilters f on lpf.urlfilterid = f.id
where f.groupid = 3062
group by lpf.landingpageid
I think that gives the same results as the CTE I copied above.