I'm preparing three temp tables and then, I assigned one of them as the data to be processed further. The choice is done by commenting out all the irrelevant temps.
select * into #Daily from ...
select * into #Monthly from ...
select * into #Yearly from ...
-- select * into #Data from #Daily
select * into #Data from #Montly
-- select * into #Data from #Yearly
Naturally, I'd like to control that by a parameter and make the selection dynamically controlled. I've only found examples like this with conditions for the subset of a static source.
declare #Type as varchar(max) = 'Daily'
...
select * into #Data from case(...)
Trying different versions of the above gave me a lot of info in red color. Due to ignorance and limited competency, it's totally useless to me. (Although, I'm sure it's pretty obvious once one gets it right. It's definitely PICNIC situation.)
What should I google for? I sense it's something like select into case source conditional but haven't got lucky (or didn't realized that I have). Quite a lot is about inserting into and not selecting into like here, which is irrelevant to me.
You could use a Cursor to loop through the tables and populate the Table Name from a cursor variable in some Dynamic SQL.
SSIS can also do this as everything can be parameterized including data destinations and it's perfect for this sort of workload.
Related
I have developed a SQL query in SSMS-2017 like this:
DECLARE #property NVARCHAR(MAX) = #p;
SET #property = REPLACE(#property, '''', '');
DECLARE #propList TABLE (hproperty NUMERIC(18, 0));
IF CHARINDEX('SELECT', #property) > 0 OR CHARINDEX('select', #property) > 0
BEGIN
INSERT INTO #propList
EXECUTE sp_executesql #property;
END;
ELSE
BEGIN
DECLARE #x TABLE (val NUMERIC(18, 0));
INSERT INTO #x
SELECT CONVERT(NUMERIC(18, 0), strval)
FROM dbo.StringSplit(#property, ',');
INSERT INTO #propList
SELECT val
FROM #x;
END;
SELECT ...columns...
FROM ...tables and joins...
WHERE ...filters...
AND HMY IN (SELECT hproperty FROM #propList)
The issue is, it is possible that the value of the parameter #p can be a list of IDs (Example: 1,2,3,4) or a direct select query (Example: Select ID from mytable where code='A123').
The code is working well as shown above. However it causes a problem in our system (as we use Yardi7-Voyager), and we need to leave only the select statement as a query. To manage it, I was planning to create a function and use it in the where clause like:
WHERE HMY IN (SELECT myFunction(#p))
However I could not manage it as I see I cannot execute a dynamic query in an SQL Function. Then I am stacked. Any idea at this point to handle this issue will be so appreciated.
Others have pointed out that the best fix for this would be a design change, and I agree with them. However, I'd also like to treat your question as academic and answer it in case any future readers ever have the same question in a use case where a design change wouldn't be possible/desirable.
I can think of two ways you might be able to do what you're attempting in a single select, as long as there are no other restrictions on what you can do that you haven't mentioned yet. To keep this brief, I'm just going to give you psuedo-code that can be adapted to your situation as well as those of future readers:
OPENQUERY (or OPENROWSET)
You can incorporate your code above into a stored procedure instead of a function, since stored procedures DO allow dynamic sql, unlike functions. Then the SELECT query in your app would be a SELECT from OPENQUERY(Execute Your Stored Prodedure).
UNION ALL possibilities.
I'm about 99% sure no one would ever want to use this, but I'm mentioning it to be as academically complete as I know how to be.
The second possibility would only work if there are a limited, known, number of possible queries that might be supported by your application. For instance, you can only get your Properties from either TableA, filtered by column1, or from TableB, filtered by Column2 and/or Column3.
Could be more than these possibilities, but it has to be a limited, known quantity, and the more possibilities, the more complex and lengthy the code will get.
But if that's the case, you can simply SELECT from a UNION ALL of every possible scenario, and make it so that only one of the SELECTs in the UNION ALL will return results.
For example:
SELECT ... FROM TableA WHERE Column1=fnGetValue(#p, 'Column1')
AND CHARINDEX('SELECT', #property) > 0
AND CHARINDEX('TableA', #property) > 0
AND CHARINDEX('Column1', #property) > 0
AND (Whatever other filters are needed to uniquely identify this case)
UNION ALL
SELECT
...
Note that fnGetValue() isn't a built-in function. You'd have to write it. It would parse the string in #p, find the location of 'Column1=', and return whatever value comes after it.
At the end of your UNION ALL, you'd need to add a last UNION ALL to a query that will handle the case where the user passed a comma-separated string instead of a query, but that's easy, because all the steps in your code where you populated table variables are unnecessary. You can simply do the final query like this:
WHERE NOT CHARINDEX('SELECT', #p) > 0
AND HMY IN (SELECT strval FROM dbo.StringSplit(#p, ','))
I'm pretty sure this possibility is way more work than its worth, but it is an example of how, in general, dynamic SQL can be replaced with regular SQL that simply covers every possible option you wanted the dynamic sql to be able to handle.
With GDPR in the UK on the looming horizon and already have a team of 15 users creating spurious SELECT statements (in excess of 2,000) across 15 differing databases I need to be able to create a method to capture an already created SELECT statement and be able to assign surrogate keys/data WITHOUT rewriting every procedure we already have.
There will be a need to run the original team members script as normal and there will be requirements to pseudo the values.
My current thinking is to create a stored procedure along the lines of:
CREATE PROC Pseudo (#query NVARCHAR(MAX))
INSERT INTO #TEMP FROM #query
Do something with the data via a mapping table of real and surrogate/pseudo data.
UPDATE #TEMP
SET FNAME = (SELECT Pseudo_FNAME FROM PseudoTable PT WHERE #TEMP.FNAME = PT.FNAME)
SELECT * FROM #TEMP
So that team members can run their normal SELECT statements and get pseudo data simply by using:
EXEC Pseudo (SELECT FNAME FROM CUSTOMERS)
The problem I'm having is you can't use:
INSERT INTO #TEMP FROM #query
So I tried via CTE:
WITH TEMP AS (#query)
..but I can't use that either.
Surely there's a way of capturing the recordset from an existing select that I can pull into a table to amend it or capture the SELECT statement; without having to amend the original script. Please bear in mind that each SELECT statement will be unique so I can't write COLUMN or VALUES etc.
Does any anyone have any ideas or a working example(s) on how to best tackle this?
There are other lengthy methods I could externally do to carry this out but I'm trying to resolve this within SQL if possible.
So after a bit of deliberation I resolved it.
I passed the Original SELECT SQL to SP that used some SQL Injection, which when executed INSERTed data. I then Updated from that dataset.
The end result was "EXEC Pseudo(' Orginal SQL ;')
I will have to set some basic rules around certain columns for now as a short term fix..but at least users can create NonPseudo and Pseudo data as required without masses of reworking :)
I have a (not normalized) legacy SQL database and am working on a task to gather code from one of several similar tables. I want to execute a lot of identical code against the #Temp table, but the tables it draws from are very dissimilar (with some columns being the same).
My code is:
IF #Variable = 'X'
BEGIN
SELECT * INTO #Temp FROM TABLE1 WHERE Condition1 = #Condition
END
IF #Variable = 'Y'
BEGIN
SELECT * INTO #Temp FROM TABLE2 WHERE Condition1 = #Condition
END
At this point, I execute some common code. There is quite a lot and I want to just use #Temp, not have another IF condition with the code copied in multiple times. I cannot really declare the table ahead of time (it is very wide - and they are not the same) and I cannot really normalize the DB (the legacy system is far to 'mature' and my time frame is far to small). Also, at the end of the query, the #Temp table is used for creating new rows back in the original table (so again, I cannot just declare the common parts).
At this point, I cannot make my stored proc because
There is already an object named '#Temp' in the database.
This error highlights the 2nd IF block. Adding a DROP TABLE #Temp in the IF block does not help either. So I'm having to offload the work in additional SPROCs or repeat the code in conditional statements. For readability, I don't like either of these options.
Any way to use #Temp within multiple IF blocks as above ( I really have more IF conditions, only 2 shown to give an idea of the issue).
Example SqlFiddle
I am looking for a way of dealing with the following situation:
We have a database server with multiple databases on it (all have the same schema, different data).
We are looking for a way to query across all the databases (and for it to be easy to configure, as more databases may be added at any time). This data access must be realtime.
Say, as an example, you have an application that inserts orders - each application has its own DB etc. What we are then looking for is an efficient way for a single application to then access the order information in all the other databases in order to query it and subsequently action it.
My searches to date have not revealed very much, however I think I may just be missing the appropriate keywords in order to find the correct info...
You must specify the database name before any database object.
Single database:
SELECT * FROM [dbo].[myTable]
Multiple dabases:
SELECT * FROM [DB01].[dbo].[myTable]
UNION ALL
SELECT * FROM [DB02].[dbo].[myTable]
UNION ALL
SELECT * FROM [DB03].[dbo].[myTable]
It's not going to be the cleanest solution ever, but you could define a view on a "Master database" (if your individual databases are not going to stay constant) that includes the data from the individual databases, and allows you to execute queries on a single source.
For example...
CREATE VIEW vCombinedRecords AS
SELECT * FROM DB1.dbo.MyTable
UNION ALL
SELECT * FROM DB2.dbo.MyTable
Which allows you to do...
SELECT * FROM vCombinedRecords WHERE....
When your databases change, you just update the view definition to include the new tables.
You can build the union dynamically:
select name from sys.databases
and then check if the database has the table:
select name from [dbname_from_above].sys.tables where name = 'YourTable'
That gives you all databases for the union. You can build the query client side or in dynamic SQL.
Note - My answer below received a couple down votes, but only one comment giving any reason why it might be down-voted. The comment was that this answer is very similar to the accepted answer, but even less performant. I disagree with this opinion and I reproduce my response here - in the actual answer - so that anyone else reading my answer might have a better chance at seeing why this is not the same as the accepted answer at all, and in fact better addresses the original question.
My response to the suggestion this is similar to the accepted answer:
on the contrary - the original question notes that new databases are added regularly. The accepted solution will require maintenance each time a new database is added. The solution here will work regardless of whether any new databases are added (in line with the original question that states they all have the same schema). Further, the accepted answer requires you to duplicate the query once per database queried. If the query is complex, that gets ugly fast. The proposal here ensures a single source of truth for the logic being used in the query.
And the answer itself:
Shooting from the hip here.
use master;
go
create table #Temp (sourceDBName varchar(128), colA varchar(128), colB varchar(128));
exec sp_MSforeachDB ' USE [?];
insert into #Temp
SELECT DISTINCT
''?'',
tableA.colA,
tableB.colB
FROM tableA JOIN tableB on some_conditions
WHERE someCol LIKE ''%some_term%''
'
select sourceDBName, colA, colB from #Temp order by 1, 2, 3;
drop table #Temp;
This logic should allow you to apply a single query to all databases. To use it though, you will want to add logic to filter out system databases, or explicitly include only the databases you specify. To achieve that, you might like to put this logic into a stored procedure which then returns a result set, so in the end, your call to this logic is a select statement that returns a rowset you can join, filter, etc.
Check out https://www.mssqltips.com/sqlservertip/2855/sql-server-multi-database-query-with-registered-servers/
SELECT * FROM (
SELECT
##SERVERNAME as [ServerName],
##version [Version],
Format(##CONNECTIONS,'N0') [Conections],
Format(##CPU_BUSY ,'N0')[CPUBusy]
) SQLInfo
LEFT JOIN (
SELECT
##SERVERNAME AS [ServerName],
SERVERPROPERTY('ProductVersion') [Version Build],
SERVERPROPERTY ('Edition') AS [Edition],
SERVERPROPERTY('ProductLevel') AS [Service Pack],
CASE SERVERPROPERTY('IsIntegratedSecurityOnly')
WHEN 0 THEN 'SQL Server and Windows Authentication mode'
WHEN 1 THEN 'Windows Authentication mode'
END AS [Server Authentication],
CASE SERVERPROPERTY('IsClustered')
WHEN 0 THEN 'False'
WHEN 1 THEN 'True'
END AS [Is Clustered?],
SERVERPROPERTY('ComputerNamePhysicalNetBIOS') AS [Current Node Name],
SERVERPROPERTY('Collation') AS [ SQL Collation],
[cpu_count] AS [CPUs],
[physical_memory_kb]*0.00000095367432 AS [RAM (GB)]
FROM [sys].[dm_os_sys_info]
) SQLInfo2 on SQLInfo.[ServerName]=SQLInfo2.[ServerName]
LEFT JOIN (
SELECT
##SERVERNAME as [ServerName],
NodeName,
Status_Description,
is_Current_Owner
FROM [MASTER].[sys].[fn_virtualservernodes]()
)Clusterinfo on SQLInfo.[ServerName]=Clusterinfo.[ServerName]
I am passing in a comma-delimited list of values that I need to compare to the database
Here is an example of the values I'm passing in:
#orgList = "1123, 223%, 54%"
To use the wildcard I think I have to do LIKE but the query runs a long time and only returns 14 rows (the results are correct, but it's just taking forever, probably because I'm using the join incorrectly)
Can I make it better?
This is what I do now:
declare #tempTable Table (SearchOrg nvarchar(max) )
insert into #tempTable
select * from dbo.udf_split(#orgList) as split
-- this splits the values at the comma and puts them in a temp table
-- then I do a join on the main table and the temp table to do a like on it....
-- but I think it's not right because it's too long.
select something
from maintable gt
join #tempTable tt on gt.org like tt.SearchOrg
where
AYEAR= ISNULL(#year, ayear)
and (AYEAR >= ISNULL(#yearR1, ayear) and ayear <= ISNULL(#yearr2, ayear))
and adate = ISNULL(#Date, adate)
and (adate >= ISNULL(#dateR1, adate) and adate <= ISNULL(#DateR2 , adate))
The final result would be all rows where the maintable.org is 1123, or starts with 223 or starts with 554
The reason for my date craziness is because sometimes the stored procedure only checks for a year, sometimes for a year range, sometimes for a specific date and sometimes for a date range... everything that's not used in passed in as null.
Maybe the problem is there?
Try something like this:
Declare #tempTable Table
(
-- Since the column is a varchar(10), you don't want to use nvarchar here.
SearchOrg varchar(20)
);
INSERT INTO #tempTable
SELECT * FROM dbo.udf_split(#orgList);
SELECT
something
FROM
maintable gt
WHERE
some where statements go here
And
Exists
(
SELECT 1
FROM #tempTable tt
WHERE gt.org Like tt.SearchOrg
)
Such a dynamic query with optional filters and LIKE driven by a table (!) are very hard to optimize because almost nothing is statically known. The optimizer has to create a very general plan.
You can do two things to speed this up by orders of magnitute:
Play with OPTION (RECOMPILE). If the compile times are acceptable this will at least deal with all the optional filters (but not with the LIKE table).
Do code generation and EXEC sp_executesql the code. Build a query with all LIKE clauses inlined into the SQL so that it looks like this: WHERE a LIKE #like0 OR a LIKE #like1 ... (not sure if you need OR or AND). This allows the optimizer to get rid of the join and just execute a normal predicate).
Your query may be difficult to optimize. Part of the question is what is in the where clause. You probably want to filter these first, and then do the join using like. Or, you can try to make the join faster, and then do a full table scan on the results.
SQL Server should optimize a like statement of the form 'abc%' -- that is, where the wildcard is at the end. (See here, for example.) So, you can start with an index on maintable.org. Fortunately, your examples meet this criteria. However, if you have '%abc' -- the wildcard comes first -- then the optimization won't work.
For the index to work best, it might also need to take into account the conditions in the where clause. In other words, adding the index is suggestive, but the rest of the query may preclude the use of the index.
And, let me add, the best solution for these types of searches is to use the full text search capability in SQL Server (see here).