Random selection in SQL Server - sql

Question

I have two sets and for each value in the first set I want to apply a number of random values from the second. The approach I have chosen uses a select from the first with a cross apply from the second. A simplified MWE is as follows:
DROP TABLE IF EXISTS #S;
CREATE TABLE #S (c CHAR(1));
INSERT INTO #S VALUES ('A'), ('B');
DROP TABLE IF EXISTS #T;
WITH idGen(id) AS (
SELECT 1
UNION ALL
SELECT id + 1 FROM idGen WHERE id < 1000
)
SELECT id INTO #T FROM idGen OPTION(MAXRECURSION 0);
DROP TABLE IF EXISTS #R;
SELECT c, id INTO #R FROM #S
CROSS APPLY (
SELECT id, ROW_NUMBER() OVER (
/*
-- this gives 100% overlap
PARTITION BY c
ORDER BY RAND(CHECKSUM(NEWID()))
*/
-- this gives the expected ~10% overlap
ORDER BY RAND(CHECKSUM(NEWID()) + CHECKSUM(c))
) AS R
FROM #T
) t
WHERE t.R <= 100;
SELECT COUNT(*) AS PercentOverlap -- ~10%
FROM #R rA JOIN #R rB
ON rB.id = rA.id AND rB.c = 'B'
WHERE rA.c = 'A';
While this solution works, I am wondering why changing to the (commented) partitioning method does not? Also, are there any caveats using this solution, seeing as it feels sort of dirty to add two checksums?
In the actual problem there is also a count in the first set containing the number of random values to select from the second set, which replaces the static 100 in the example above. However, using the fixed 100 made it easy to verify the expected overlap.

Answer 1

RAND() function is a run-time constant in SQL Server. It means that usually it is evaluated once for the query. When you pass a value to RAND this value serves as a starting seed.
You need to examine execution plan and you'll see where optimiser puts evaluation of the functions. It the case which doesn't produce expected result most likely optimiser has optimised it too aggressively and moved all "randomness" outside the loop.
Also, there is no point wrapping NEWID() into CHECKSUM() and into RAND().
Simple NEWID() is enough. Or, even better, a function that is designed to produce a random number, such as CRYPT_GEN_RANDOM()
Either version of your query looks a bit strange. I'd write it like this:
SELECT c, id INTO #R
FROM #S
CROSS APPLY
(
SELECT TOP(100) -- or #S.SomeField instead of 100
id
FROM #T
ORDER BY CRYPT_GEN_RANDOM(4) -- generate 4 random bytes, usually it is enough
) AS t
;
This gives 100 random rows from #T for each row from #S.
Actually, the query above is not good. Optimiser again sees that inner query (inside the CROSS APPLY) doesn't depend on outer query and optimises it away.
End result is that random rows are selected only once.
We need something to make optimiser run the inner query for each row from #S.
One way would be something like this:
SELECT c, id INTO #R
FROM #S
CROSS APPLY
(
SELECT TOP(100) -- or #S.SomeField instead of 100
id
FROM #T
ORDER BY CRYPT_GEN_RANDOM(4) + CHECKSUM(c)
) AS t
;
Something in the inner query to reference the row from the outer query. If you put TOP(#S.SomeField) instead of constant TOP(100), then + CHECKSUM(c) is not needed.
This is the plan for the first variant. You can see that #T is scanned once (1000 rows are read).
This is the plan for the second variant. You can see that #T is scanned twice (2000 rows are read).