Big performance loss when using IN(subquery). Why? - sql

Question

I am using SQL Server 2005 and I have noticed something strange when I want to filter some results when using a subquery within an IN clause. For instance, this is my current query and it runs in 70 seconds on average:
select Phone, ZipCode, sum(Calls) as Calls, sum(Sales) as Sales
from Archive
where CustomerID = 20
and ReportDate = '2/3/2011'
and Phone in (
select Phone
from PlanDetails
where Phone is not null
and Length is not null
and PlannedImp > 0
and CustomerID = 20
and (StatusID <> 2 and StatusID <> 7)
and SubcategoryID = 88
)
group by Phone, ZipCode
However, if I break them down into 2 separate queries, they take under 1 second each to run.
select Phone
from PlanDetails
where Phone is not null
and Length is not null
and PlannedImp > 0
and CustomerID = 20
and (StatusID <> 2 and StatusID <> 7)
and SubcategoryID = 88
and
select Phone, ZipCode, sum(Calls) as Calls, sum(Sales) as Sales
from Archive
where CustomerID = 20
and ReportDate = '2/3/2011'
group by Phone, ZipCode
Finally, if I do this it returns the same results as the first query, but in about 2-3 seconds:
select Phone
into #tempTable
from PlanDetails
where Phone is not null
and Length is not null
and PlannedImp > 0
and CustomerID = 20
and (StatusID <> 2 and StatusID <> 7)
and SubcategoryID = 88
select Phone, ZipCode, sum(Calls) as Calls, sum(Sales) as Sales
from Archive
where CustomerID = 20
and ReportDate = '2/3/2011'
and Phone in (
select Phone
from #tempTable
)
group by Phone, ZipCode
For the past few weeks I have been noticing that not just this query is slow, but any query using a (somewhat complicated) subquery inside of an IN clause just ruins the performance. What is the reason for that?
The only indexes that are available for either of these queries to use is a nonclustered index on CustomerID for both tables. I looked at the execution plans of both the slow query and the fast ones, and saw that the nonclustered index seek on the Archive table is what takes the highest percentage of cost by far (80-90%). However, the only difference is that that step in the slow query has a CPU cost of 7.1 while the fast ones have a CPU cost of 1.7.

Answer 1

It depends upon the database system, the version, the settings, etc. but generally what ends up happening is the database fails (or refuses) to cache that inner query, so it is being executed every single iteration of the outer query. You are changing your problem from a O(n) efficiency class to O(n^2).

Answer 2

Quote IN vs. JOIN vs. EXISTS:
We see now that contrary to the popular opinion, IN / EXISTS queries are not less efficient than a JOIN query in SQL Server.
In fact, JOIN queries are less efficient on non-indexed tables, since Semi Join methods allow aggregation and matching against a single hash table, while a JOIN needs to do these two operations in two steps.
Beyond that, indexing and how current table statistics are plays a large role in how the optimizer decides to execute a query.

Answer 3

What if you rewrite the query with a join?
select a.Phone, a.ZipCode, sum(a.Calls) as Calls, sum(a.Sales) as Sales
from Archive a
inner join PlanDetails pd
on a.CustomerID = pd.CustomerID
and a.Phone = pd.Phone
where a.CustomerID = 20
and a.ReportDate = '2/3/2011'
and pd.Length is not null
and pd.PlannedImp > 0
and (pd.StatusID <> 2 and pd.StatusID <> 7)
and pd.SubcategoryID = 88
group by a.Phone, a.ZipCode

Answer 4

I'd propose 2 solutions :
1. Try to rewrite your query using EXISTS instead of IN. It might help if you use the older SQL Server versions (If my memory serves me well, prior to SQL Server 2005 EXITST and IN generated different execution plans).
2. Try to use INNER JOIN (you can also use CTE):
select Phone, ZipCode, sum(Calls) as Calls, sum(Sales) as Sales
from Archive
INNER JOIN
(
select DISTINCT Phone // DISTINCT to avoid duplicates
from PlanDetails
where Phone is not null
and Length is not null
and PlannedImp > 0
and CustomerID = 20
and (StatusID <> 2 and StatusID <> 7)
and SubcategoryID = 88
)XX ON (XX.Phone = Archive.Phone)
where CustomerID = 20 and ReportDate = '2/3/2011'
group by Phone, ZipCode
Personally, I'd expect the second approach to give you much better results.