SQL Query against multiple databases - sql

Question

I'm trying to run a query against multiple databases on the same server.
I need to pull all the values from 2 tables in a database based on a criteria from a 3rd table in the database, if the database was created after a certain date.
I have a query to find when the database was created:
SELECT *
FROM sys.databases
WHERE STATE = 0 --ignores offline databases
AND database_id > 4 --does not include master, model, msdb, tempdb
AND create_date > CONVERT(datetime, '2021-01-01')
And the query that I need run on each database is generally as follows:
SELECT *
FROM Table1
INNER JOIN Table3 ON Table3.Column6=Table1.Column2
AND Table3.Column3='Value1'
AND Table3.Column4='Value2'
INNER JOIN Table2 ON Table3.Column6=Table2.Column2
I did find this question, which is essentially would I would like to do, but when I look at the INFORMATION_SCHEMA.TABLES the TABLE_CATALOG column does not have the table names I would like to query against. I thought I could try pulling the names from the sys.databases table like above, so I tried modifying it to:
DECLARE #cmd VARCHAR(max) = N''
SELECT #cmd += COALESCE(#cmd + ' UNION ALL ', '') + 'SELECT *
FROM [' + name + '].dbo.Table1
INNER JOIN [' + name + '].dbo.Table3 on Table3.Column6=Table1.Column2
AND Table3.Column3= ''Value1''
AND Table3.Column4=''Value2''
INNER JOIN [' + name + '].dbo.Table2 on Table3.Column6=Table2.Column2'
FROM sys.databases
WHERE STATE = 0
AND database_id>4
AND create_date>CONVERT(datetime,'2021-08-26')
SET #cmd = STUFF(#cmd, CHARINDEX('UNION ALL', #cmd), 10, '')
PRINT #cmd
EXEC(#cmd)
But when I run it with a date earlier than 2021-08-26 (which grabs more than 5 tables), I get a memory error. I need to run this at least to the beginning of April (preferably up to the beginning of the year) which will grab around 500 tables.
What is the recommended way to run a query against multiple databases in SQL?

Answer 1

My recommendation would be instead of trying to build one massive UNION ALL dynamic SQL statement, that you build a #temp table to hold the results of each output, and then it's much easier to send the same string to each database:
CREATE TABLE #hold(dbname sysname, Column1 {data type}, ...);
DECLARE #sql nvarchar(max), #exec nvarchar(1024);
SET #sql = N'SELECT DB_NAME(), *
FROM dbo.Table1
INNER JOIN dbo.Table3
ON Table3.Column6 = Table1.Column2
AND Table3.Column3 = ''Value1''
AND Table3.Column4 = ''Value2''
INNER JOIN dbo.Table2
ON Table3.Column6 = Table2.Column2;';
DECLARE #dbname sysname, #c cursor;
SET #c = CURSOR FORWARD_ONLY STATIC READ_ONLY FOR
SELECT name FROM sys.databases
WHERE state = 0 -- ignores offline databases
AND database_id > 4 -- does not include master, model, msdb, tempdb
AND create_date > CONVERT(datetime, '20210101');
OPEN #c;
FETCH NEXT FROM #c INTO #dbname;
WHILE ##FETCH_STATUS = 0
BEGIN
SET #exec = QUOTENAME(#dbname) + N'.sys.sp_executesql';
INSERT #hold EXEC #exec #sql;
FETCH NEXT FROM #c INTO #dbname;
END;
SELECT * FROM #hold;
You might also consider investing in sp_ineachdb, a procedure I wrote to help simplify running the same command in the context of each database.
Execute a Command in the Context of Each Database in SQL Server using sp_ineachdb
Execute a Command in the Context of Each Database in SQL Server - Part 2