We have a scenario in which we wish to use Azure Elastic Query so as to allow us to run aggregate queries on multiple databases geographically distributed, and which might be added to with time. However, we can't yet find useful docs or advise on how to design and run Azure Elastic Queries that can operate reliably without being modified (by hand), while data sources are added or removed.
Any advise from someone with experience on this db tech would be very welcome.
As a further, specific constraint, the disparate source databases are all SQL Express DBs - we are considering mapping these to online Azure SQL instances (PaaS).
UPDATE: I've seen something similar being asked/answered here, but am seeking a better answer.
You can create external source with a specific name that will be used on your queries but programmatically change the location and database name used by sources using Dynamic SQL:
ALTER PROCEDURE CETFromNewLocation AS
BEGIN
DECLARE #location varchar(100)
SET #location = 'myserver.database.windows.net'
DECLARE #CreateExternalTableString varchar(100)
SET #CreateExternalTableString = 'CREATE EXTERNAL DATA SOURCE MyExtSrc
WITH
(
TYPE=SHARD_MAP_MANAGER,
LOCATION=' + #location + ' DATABASE_NAME='ShardMapDatabase',
CREDENTIAL= SMMUser,
SHARD_MAP_NAME='ShardMap'
);'
EXEC sp_executesql #CreateExternalTableString
END
Related
I am using al lot of sql queries and tyred of typing the complete prefixes of
[LINKED_SERVER_ALIAS].[LINKED_SERVER_ON_LINKED_SERVER].[DATABASEPATH].[SCHMEMA].TABLE
No way to change the serverstructure or direct login to linked server on another linked server.
Question: Is there som transact sql command to create a global alias like
create
alias my_linked_connection
for
[LINKED_SERVER_ALIAS].[LINKED_SERVER_ON_LINKED_SERVER].[DATABASEPATH].[SCHMEMA].TABLE
that it is possible to use:
select * from my_linked_connection.TABLE
an additional problem is, that these are to many prefixes, so a normal select query is only possible by openquery or declare #cmd ... exec #cmd
Thanks
Combine a part of the prefixes inside the linked server alias sp_addlinked server.
Synonym is what you are looking for here
CREATE SYNONYM schema.tablename for linkedservername.remotedatabasename.schema.tablename
This has the advantage (which I expect is what you are looking for) that you can move views, functions and procedures through your development environments whithout having to modify the object code; the only thing that should be different is that the target database for the synonym will be different each time.
Note that Synonym is a MSSQL feature and may not be supported by your ODBC/JDBC drivers so please test fully before deployment.
Is there a way to programmatically set the table names used in the FROM clause?
The reason is we have a different table names in our prod vs. dev environment therefore we need to set the table names accordingly to be used in our reports, based on the different environments.
For example:
In prod the database name is 123prd, in dev it would be 123dev
In prod the database name is 456prd, in dev it would be 456dev
The report runs against database 123prd and we need to INNER JOIN to another table in the 456prd database.
So for the Prod environment it would be something like below:
USE 123prd
SELECT *
FROM aTable a
JOIN 456prd.dbo.bTable b
ON a.id = b.id
However since the report needs to work correctly according to the different environment Prod vs. Dev I will need to programatically change the database name in the FROM clause.
So this is what I have:
DECLARE #456DBName VARCHAR(16)
SET #456DBName = REPLACE(DB_NAME(), '123', '456')
USE 123prd
SELECT *
FROM aTable a
JOIN CONCAT(#456DbName, '.dbo.bTable') b
ON a.id = b.id
I got error invalid syntax when using CONCAT or +
Is there a correct way on how to do linked server programmatically? Sorry about bad English by the way, hopefully my question makes sense.
You have limited options, because you cannot just set the project that way.
One option is to do away with three part naming. Just reference tables within the database. This works if production/development only consist of one database.
Another option is to wrap all references with views (these could also incorporate business logic). This is particularly helpful if "production" and "development" can span multiple databases. The views can be in one place -- but they probably need to be created using deployment scripts that use dynamic SQL.
A related option would use synonyms. I have not personally used these for this purpose, but they should work.
And finally, there is dynamic SQL. One method is to store the queries and have a stored procedure do the replacement. Heck, you could even store the queries as views for a specific database -- and have a stored procedure do the substitution for other environments.
You can query across multiple servers using 'server groups' in ssms.https://learn.microsoft.com/en-us/sql/ssms/register-servers/execute-statements-against-multiple-servers-simultaneously?view=sql-server-2017
Option 1. You can do it with code, using different methods depending on the environment variable and calling different store procedures or same name different schema.
Option 2. You can use Dynamic Queries. You could create a table to manage the configuration to what schema/table to use depending on the parameter you send from your server. Then just execute the dynamic query.
Ie.
DECLARE #Prefix VARCHAR(10) = 'dev';
DECLARE #Params NVARCHAR(200) = '#Prefix VARCHAR(10)'
, #Query NVARCHAR(MAX) = 'SELECT [Id] FROM [Schema].['+#Prefix+'TableName]';
EXECUTE [sp_executesql]
#Query ,
#Params ,
#Prefix = #Prefix
I am creating a new SQL database that needs to connect to several existing DB's, all of which have the same schema.
I would like to create stored procedures that can be used to access any of the existing DB's whilst re-using code as much as possible. My question is, what is the best way to write generic SP's that can be used to access any one of these existing databases?
Here are some options I have considered so far. Please note, these are just simple code snippets to illustrate the question, not real-world code. First I tried switching on the DB name:
IF #db = 'A'
BEGIN
SELECT * FROM A.dbo.SERIES_DATA
END
IF #db = 'B'
BEGIN
SELECT * FROM B.dbo.SERIES_DATA
END
This has the disadvantage that the same SQL statement is repeated several times. So then I thought of using dynamic SQL:
DECLARE #Command varchar(100)
SET #Command = 'select * from ' + #db + '.dbo.SERIES_DATA'
EXEC (#Command)
This solves the problem of duplicate staements, but has the risks of dynamic SQL (e.g. injection attacks etc). So finally I hit on the idea of creating a Table-valued Function:
CREATE FUNCTION [dbo].[ufnSeries_Data] (#db varchar(10))
RETURNS TABLE
AS
RETURN (
SELECT * FROM A.dbo.Series_Data WHERE #db = 'A'
UNION
SELECT * FROM B.dbo.Series_Data WHERE #db = 'B'
)
GO
This gives me the ability to write single, generic code to access any of the DB's:
SELECT * FROM ufnSeries_Data(#db)
The problem with this approach is that I have to give all users of the system read-access to all of the databases in order for this query to work, otherwise I get an access-denied error. I'm also not certain of the performance impact of creating a TVF like this.
Does anyone has any other ideas for how I can structure my code to minimize duplication of statements? Maybe there is a much simpler approach that I'm missing? I would be very grateful for any suggestions.
You can do this by removing the reference to your database and schema
SELECT * FROM SERIES_DATA
It is a question of creating the schema in all your databases, and granting the login access to the databases and schemas (in this case you are using the default dbo schema).
If you are using the management studio, when you create logins, you can handle the rest with user mapping.
If you have the same schema (e.g. dbo), you can change your select to:
SELECT * FROM dbo.SERIES_DATA
You really don't want to include the database name unless you are joining tables across databases.
I am working with some commercial schemas, which have a a set of similar tables, which differ only in language name e.g.:
Products_en
Products_fr
Products_de
I also have several stored procedures which I am using to access these to perform some administrative functions, and I have opted to use synonyms since there is a lot of code, and writing everything as dynamic SQL is just painful:
declare #lang varchar(50) = 'en'
if object_id('dbo.ProductsTable', 'sn') is not null drop synonym dbo.ProductsTable
exec('create synonym dbo.ProductsTable for dbo.Products_' + #lang)
/* Call the synonym table */
select top 10 * from dbo.ProductsTable
update ProductsTable set a = 'b'
My question is how does SQL Server treat synonyms when it comes to concurrent access? My fear is that a procedure could start, then a second come along and change the table the synonym points to halfway through causing major issues. I could wrap everything in a BEGIN TRAN and COMMIT TRAN which should theoretically remove the risk of two processes changing a synonym, however the documentation is scarce on this matter and I can not get a definitive answer.
Just to note, although this system is concurrent, it is not high traffic, so the performance hits of using synonyms/transactions are not really an issue here.
Thanks for any suggestions.
Your fear is correct. Synonyms are not intended to used in this way. Wrapping it is a transaction (not sure what isolation level would be required) might solve the issue, but only by making the system single user.
If I was dealing with this then I would probably have gone with dynamic SQL becuase I am familiar with it. However, having thought about it I wonder if schemas could solve your problem.
If you created schema for each language and then had a table called products in each schema. Your stored proc can then reference an un-qualified table name and SQL should resolve the reference to the table that is in the default schema of the current user. You'll then need to either change what account your application authenticates as to determine which schema it uses or use EXECUTE AS in a stored proc to decide which schema is default.
I haven't tested this schema idea, I may not have thought of everything and I don't know enough about your application to know if it is actually workable in your case. Let us know if you decide to try it.
How would you handle cross database queries in different environments. For example, db1-development and db2-development, db1-production and db2-production.
If I want to do a cross-database query in development from db2 to db1 I could use the fully qualified name, [db1-development].[schema].[table]. But how do I maintain the queries and stored procedures between the different environments? [db1-development].[schema].[table] will not work in production because the database names are different.
I can see search and replace as a possible solution but I am hoping there is a more elegant way to solve this problem. If there are db specific solutions, I am using SQL Server 2005.
Why are the database names different between dev and prod? It'd, obviously, be easiest if they were the same.
If it's a single table shared, then you could create a view over it - which only requires that you change that view when moving to production.
Otherwise, you'll want to create a SYNONYM for the objects, and make sure to always reference that. You'll still need to change the SYNONYM creation scripts, but that can be done in a build script fairly easily, I think.
For this reason, it's not practical to use different names for development and production databases. Using the same db name on development, production, and optionally, acceptance/Q&A environments, makes your SQL code much easier to maintain.
However, if you really have to, you could get creative with views and dynamic SQL. For example, you put the actual data retrieval query inside a view, and then you select like this:
declare #environment varchar(10)
set #environment = 'db-dev' -- input parameter, comes from app layer
declare #sql varchar(8000)
set #sql = 'select * from ' + #environment + '.dbo.view'
execute(#sql)
But it's far from pretty...