Where condition with many parameters - sql

Question

I have to run a query with say around 30 columns in where clause. Each column have 1000+ value to compare. I know IN clause is not the best way to do so. Can any one suggest how to run this query without processing error. E.g below
select *
from table
where column1 not in (1,2,3,4......1000+ )
and column2 not in (1,2,3,4......1000+ ) and column3 not in (1,2,3,4......1000+) and so on upto column30.
I am getting error:
SQL Server query processor ran out of internal resources.
I explored other link but did not find solution or suggestion to implement it in a best way.

Answer 1

Create a temp table holding all the possible values:
CREATE TABLE #temp(column1 int, column2 int ....)
INSERT INTO #temp values
(1,1,1...),
(1,2,2...),
.
.
Now, apply SELECT query filter accordingly
Select * from table where column1 not in (SELECT column1 FROM #temp)
and column2 not in (SELECT column2 from #temp)
there is one more approach of using left outer joins
Select * from table as t
LEFT OUTER JOIN #temp as t1
on t.column1 = t1.column1
LEFT OUTER JOIN #temp as t2
on t.column2 = t1.column2
.
.
WHERE t.column1 IS NULL AND t.column2 IS NULL

Answer 2

There are a couple routes you can take based on assumptions.
Assumption 1: 1,2,3,4......1000+ values are same across all columns' NOT IN comparisons
With this assumption, I believe the problem is that SQL Server's resources are exhausted in a query like this:
Note that I am calling your table as MYTABLE.
select *
from MYTABLE -- <---- Remember the name I gave your table
where column1 not in (2, 5, 100, 22, 44, ... thousand other values)
and column2 not in (2, 5, 100, 22, 44, ... thousand other values)
and column3 not in (2, 5, 100, 22, 44, ... thousand other values)
Notice the values we are comparing. All columns are compared to the same set of values.
Solution
Create a table called values_to_compare like so:
create table values_to_compare (comparison_value int primary key);
Just remember that it might be much, much easier to populate values_to_compare table if you just import a text file with values into it rather than typing out a giant insert statement. But, if you choose to write an insert statement, here's how you'd write it. You might have to break your insert statement up in batches of a few hundred entries if SQL Server complains about a large insert statement.
insert into comparison values (2), (5), (100), (22), (44), ... thousand other values;
Then, create an index on MYTABLE like so:
create index on MYTABLE (column1);
Then, write your query in pieces. First, do this:
select *
from MYTABLE
where not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column1);
Hopefully, this will run fast. If that runs fast enough for you, it's time to add 4 more indexes:
create index on MYTABLE (column2);
create index on MYTABLE (column3);
create index on MYTABLE (column4);
create index on MYTABLE (column5);
Then, add 4 more lines so that the query looks like this:
select *
from MYTABLE
where not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column1)
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column2)
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column3)
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column4)
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column5);
If this works well, add 25 more indexes on each of the columns in MYTABLE. Then, expand the query above to compare all 30 columns. I believe SQL Server will perform well.
Assumption 2: Values you compare with column1..30 are all different
With this assumption, I believe the problem is that SQL Server's resources are exhausted in a query like this:
select *
from MYTABLE
where column1 not in (2, 5, 100, 22, 44, ... thousand other values)
and column2 not in (1, 225, 5619, 8, 9000, ... thousand other values)
and column3 not in (2024, 5223, 0, 552, 4564, ... thousand other values)
Notice the values we are comparing. All columns are compared to different set of values.
Solution
Create a table called values_to_compare like so:
create table values_to_compare (compare_column as varchar(50), comparison_value int, primary key (compare_column, compare_value));
Just remember that it might be much, much easier to populate values_to_compare table if you just import a text file with values into it rather than typing out a giant insert statement. But, if you choose to write an insert statement, here's how you'd write it. You might have to break your insert statement up in batches of a few hundred entries if SQL Server complains about a large insert statement.
insert into comparison values
('column1', 2), ('column1', 5), ('column1', 100), ('column1', 22), ('column1', 44), ... thousand other values ...
, ('column2', 1), ('column2', 225), ('column2', 5619), ('column2', 8), ('column2', 9000), ... thousand other values ...
, ('column3', 2024), ('column3', 5223), ('column3', 0), ('column3', 552), ('column3', 4564) ... thousand other values ...
;
Then, create an index on MYTABLE like so:
create index on MYTABLE (column1);
Then, write your query in pieces. First, do this:
select *
from MYTABLE
where not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column1 and compare_column = 'column1');
Hopefully, this will run fast. If that runs fast enough for you, it's time to add 4 more indexes:
create index on MYTABLE (column2);
create index on MYTABLE (column3);
create index on MYTABLE (column4);
create index on MYTABLE (column5);
Then, add 4 more lines so that the query looks like this:
select *
from MYTABLE
where not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column1 and compare_column = 'column1')
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column2 and compare_column = 'column2')
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column3 and compare_column = 'column3')
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column4 and compare_column = 'column4')
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column5 and compare_column = 'column5');
If this works well, add 25 more indexes on each of the columns in MYTABLE. Then, expand the query above to compare all 30 columns. I believe SQL Server will perform well.
Give this a shot.
EDIT
Based on the new information that SQL Server is able to do comparisons up to ~20 columns, we can do split operation.
select * into MYTABLE_TEMP from MYTABLE where 1=2;
We now have a temporary table to store data. Then, execute query comparing only 15 columns. Take the output and dump it into MYTABLE_TEMP.
insert into MYTABLE_TEMP
select *
from MYTABLE
where not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column1)
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column2)
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column3)
...
...
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column15);
Create 15 indexes on MYTABLE_TEMP.
create index on MYTABLE_TEMP (column16);
create index on MYTABLE_TEMP (column17);
...
...
create index on MYTABLE_TEMP (column30);
Then, run a query on MYTABLE_TEMP.
select *
from MYTABLE_TEMP
where not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column16)
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column17)
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column18)
...
...
and not exists (select 1 from values_to_compare where comparison_value = MYTABLE.column30);
See if that helps.