Optimization - sql :How to show all data exists in multiple tables - sql

Question

I have two table. I want to find all the rows in table One that exists in table Two, and back. I had the answer, but I want it faster.
Example:
Create table One (ID INT, Value INT, location VARCHAR(10))
Create table Two (ID INT, Value INT, location VARCHAR(10))
INSERT INTO One VALUES(1,2,'Hanoi')
INSERT INTO One VALUES(2,1,'Hanoi')
INSERT INTO One VALUES(1,4,'Hanoi')
INSERT INTO One VALUES(3,5,'Hanoi')
INSERT INTO Two VALUES(1,5,'Saigon')
INSERT INTO Two VALUES(4,6,'Saigon')
INSERT INTO Two VALUES(5,7,'Saigon')
INSERT INTO Two VALUES(2,8,'Saigon')
INSERT INTO Two VALUES(2,8,'Saigon')
And answers:
SELECT * FROM One WHERE ID IN (SELECT ID FROM Two)
UNION ALL
SELECT *FROM Two WHERE ID IN (SELECT ID FROM One)
With this query, the system scan the table 4 times
enter image description here
I want the system scan the table twice (table One once, table Two once).
Am I crazy?

Answer 1

You can try something like:
-- CREATE TABLES
IF OBJECT_ID ( 'tempdb..#One' ) IS NOT NULL
DROP TABLE #One;
IF OBJECT_ID ( 'tempdb..#Two' ) IS NOT NULL
DROP TABLE #Two;
CREATE TABLE #One (ID INT, Value INT, location VARCHAR(10))
CREATE TABLE #Two (ID INT, Value INT, location VARCHAR(10))
-- INSERT DATA
INSERT INTO #One VALUES(1,2,'Hanoi')
INSERT INTO #One VALUES(2,1,'Hanoi')
INSERT INTO #One VALUES(1,4,'Hanoi')
INSERT INTO #One VALUES(3,5,'Hanoi')
INSERT INTO #Two VALUES(1,5,'Saigon')
INSERT INTO #Two VALUES(4,6,'Saigon')
INSERT INTO #Two VALUES(5,7,'Saigon')
INSERT INTO #Two VALUES(2,8,'Saigon')
INSERT INTO #Two VALUES(2,8,'Saigon')
-- CREATE INDEX
CREATE NONCLUSTERED INDEX IX_One ON #One (ID) INCLUDE (Value, location)
CREATE NONCLUSTERED INDEX IX_Two ON #Two (ID) INCLUDE (Value, location)
-- SELECT DATA
SELECT o.ID
,o.Value
,o.location
FROM #One o
WHERE EXISTS (SELECT 1 FROM #Two t WHERE o.ID = t.ID)
UNION ALL
SELECT t.ID
,t.Value
,t.location
FROM #Two t
WHERE EXISTS (SELECT 1 FROM #One o WHERE t.ID = o.ID)
but it depends how "big" data you have. If the data is really big (millions of rows) and you are running Enterprise version of SQL Server, you may consider using columnstore indexes.

Answer 2

The reason you're scanning the tables twice is because you are reading from Table X and looking up the corresponding value from table Y. And once that's finished you do the same but starting from table Y and then looking for matches in Table Y. After that, both results are combined and returned to the caller.
In a way, that's not a bad thing although if tables are 'wide' and contain a lot of columns you don't need then you are doing a lot of IO for no good reason. Additionally, in your example, the quest for a matching ID in the other table requires scanning the whole table because there is no 'logic' to the ID field. It simply is a list of values. To speed up things you should add an index on the ID field that helps the system find a particular ID value MUCH MUCH faster. Additionally, this also limits the amount of data that needs to be read for the lookup phase: the server will only read from the index which only contains the ID values (**) and not all the other, unneeded fields.
To be honest, I find your requirement a bit strange, but I'm guessing it's mostly due to simplification as to make it understandable here on SO. My first reaction was to suggest to use a JOIN between both tables, but since the ID fields are non-unique this then results in duplicates! To work around that I added a DISTINCT but then things slowed down severely. In the end, doing just the WHERE ID IN (...) turned out to be the most efficient approach.
Adding indexes on the ID field made it faster although the effect wasn't as big as I expected, probably because there are few other fields and the gain in IO is negligible (read: it all fits in memory even though I tried this on 5 million rows).
FYI: Personally I prefer the construction WHERE EXISTS() over WHERE IN (...) but they're both equivalent and actually produced the exact same query plan.
(**: aside from the indexed fields, every index also contains the clustered index -- which usually is the Primary Key of the table -- fields in its leaf data. For more information Kimberly L. Tripp has some interesting articles about indexes and how they work.)