I have 2 DB on same SQL Azure server and i have same table(TB1) on both DB, now i want to read data from TB1 of DB2 and insert data into TB1 of DB1.
I am using below query but getting error.
insert into TB1 select 1,* from [DB2].dbo.TB1
Error Message
Msg 40515, Level 15, State 1, Line 16
Reference to database and/or server name in 'DB2.dbo.TB1' is not supported in this version of SQL Server.
Yes, you can use the Elastic Query Features on SQL Azure.It's the only way you can perform the cross database Queries.
Here are the detailed Queries to follow:
Run the below Query in your DB1(Since you said like reading the TB1 from DB2 and insert those Data's into your TB2 in your DB1)
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'STro*ngPaSSe0rD';
CREATE DATABASE SCOPED CREDENTIAL Login
WITH IDENTITY = 'Login',
SECRET = 'STro*ngPaSSe0rD';
CREATE EXTERNAL DATA SOURCE RemoteReferenceData
WITH
(
TYPE=RDBMS,
LOCATION='myserver.database.windows.net',
DATABASE_NAME='DB2',
CREDENTIAL= Login
);
CREATE EXTERNAL TABLE [dbo].[TB1]
(
[Columns] [DataTypes]
)
WITH (DATA_SOURCE = [RemoteReferenceData])
After these step, you can Query the external table like the Normal table. Though some limitations while using the External table, like you couldn't able to Insert Data's into a EXTERNAL TABLE(Reference table)
Azure supports this cross database query feature since 2015 but needs some extra setup to work and Elastic Query.
The first step is create a security credential:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<password>';
CREATE DATABASE SCOPED CREDENTIAL DB2Security
WITH IDENTITY = '<username>',
SECRET = '<password>';
The "username" and "password" should be the username and password used to login into the DB2 database.
Now you can use it to define a external datasource, so DB1 can connect to DB2:
CREATE EXTERNAL DATA SOURCE DB2Access
WITH (
TYPE=RDBMS,
LOCATION='myservernotyours.database.secure.windows.net',
DATABASE_NAME='DB2',
CREDENTIAL= DB2Security);
Finally, you map the TB1 as a external table from the DB2 database, using the previous external datasource:
CREATE EXTERNAL TABLE dbo.TB1FromDB2(
ID int,
Val varchar(50))
WITH
(
DATA_SOURCE = DB2Access);
You can also accomplish this using the Azure SQL Data Sync, but the data are replicated in one single database and this feature are still a preview version (May/2018) and you always see oldest data (the minimal configurable interval for each synchronization is 5 minutes).
You can perform cross database queries using the elastic query features on SQL Azure.
You will have to create an external data source and an external table to be able to query tables on other SQL Azure databases. This article shows how to do it.
Hope this helps.
Related
How to get the list of queries executed in Azure Synapse Dedicated SQL Pool by user?
You can get the query details by either using Azure Query Store service which automatically captures a history of queries, plans, and runtime statistics, and retains these for your review or you can create your custom table in your database to capture all the required details.
Only Synapse Dedicated pool supports Query Store but it isn't enabled by default for new Azure Synapse Analytics databases.
Use the ALTER DATABASE statement to enable the query store for a given database. For example:
ALTER DATABASE <database_name>
SET QUERY_STORE = ON (OPERATION_MODE = READ_WRITE);
The following query returns information about queries and plans in the Query Store.
SELECT Txt.query_text_id, Txt.query_sql_text, Pl.plan_id, Qry.*
FROM sys.query_store_plan AS Pl
INNER JOIN sys.query_store_query AS Qry
ON Pl.query_id = Qry.query_id
INNER JOIN sys.query_store_query_text AS Txt
ON Qry.query_text_id = Txt.query_text_id;
On the other side, if you want to create a table to track the history of all the queries, you can refer the answer given by SqlWorldWide on this similar thread.
I have a need to connect to Synapse Analytics Serverless SQL Pool database using SQL Authentication.
I created a serverless SQL Pool database and created a SQL User and provided db_owner access.
Then created an external table below
IF NOT EXISTS (SELECT * FROM sys.external_file_formats
WHERE name = 'SynapseDeltaFormat')
CREATE EXTERNAL FILE FORMAT [SynapseDeltaFormat]
WITH ( FORMAT_TYPE = PARQUET)
GO
IF NOT EXISTS (SELECT * FROM sys.external_data_sources WHERE name =
'test_dfs_core_windows_net')
CREATE EXTERNAL DATA SOURCE [test_dfs_core_windows_net]
WITH (
LOCATION = 'abfss://test.dfs.core.windows.net'
)
GO
CREATE EXTERNAL TABLE table_staging (
<columns here>
)
WITH (
LOCATION = 'bronze/table_staging/',
DATA_SOURCE = [test_dfs_core_windows_net],
FILE_FORMAT = [SynapseDeltaFormat]
)
GO
SELECT TOP 100 * FROM dbo.table_staging
GO
Get below error when trying to access data of the table using SQL User
External table 'dbo.table_staging' is not accessible because location does not exist or it is used by another process.
Table data is accessible using AD user. Created DataSource using SQL User.
It seems like that SQL Server User does not have access to data lake/data storage. How to grant that access?
Just started delta lakes myself :)
Per default the serverless sql authenticates using the user context. When you are querying from within the synapse data studio you are using your AD users context which is why you can connect to the external storage.
However for SQL users and AD users that do not have access to be able to query you need to use credentials when setting up the query.
You can finde detailed instructions here:
https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/develop-storage-files-storage-access-control?tabs=user-identity
You need to have way of accessing the storage - either service principal, SAS, Managed Identity
I just set up credentials using a service principal "App registration"
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<secret>';
go
CREATE DATABASE SCOPED CREDENTIAL [credentialname] WITH
IDENTITY = '<Client-ID>#https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token'
, SECRET = '<token>'
GO
CREATE EXTERNAL DATA SOURCE mydatasource WITH (
LOCATION = 'https://<storageaccount>.dfs.core.windows.net/onedatahubtest',
CREDENTIAL = credentialname
);
CREATE VIEW MYVIEW AS
SELECT *
FROM OPENROWSET(
BULK 'Example/table',
DATA_SOURCE='mydatasource',
FORMAT = 'delta') as rows;
And now my sql user can access the view
I have a linked server hosted externally (SQL Server) and would like to refresh the data everyday. What I have tried is truncate the data in the destination table and insert the data from the source database.
I have about 100 tables to refresh from external source. Is there a better way to refresh the entire database without having to refresh each table?
Source Database : MySQL
Destination Database: SQL Server
use [Audits]
GO
TRUNCATE TABLE [Products_data].[r_rooms]
GO
INSERT INTO [Products_data].[r_rooms]
Select * from [ProdcutRemote]...[Products_data.r_rooms]
Use case-
After learning that AD Passthrough is not working as expected on Synapse Serverless pool with ADLS Gen-2 ; I am trying to use traditional method of creating external tables on Serverless Pool and granting READ ONLY access to users to a set of tales and enable WRITE BACK option to another ADLS Gen-2 container using CETAS option .
Looks like I am stuck there as well - to move forward.
I have tried to explain my scenario in below image.
Now - I have 5 external tables on a database where I have a READ ONLY access to the schema's where those table exists.
I wanted to create few more tables - which ideally does a JOIN between those 5 tables and aggregates the data and writes back to ADLS Gen-2 for reporting/data science purpose.
What access should I grant for WRITE back purpose ?
I tried creating new schema and granting ALTER, CONTROL, SELECT access to that schema along with CREATE TABLE access at database level . I dont want to grant more access to database level - as it has data scoped credential having managed identity referenced- which will grant full access on ROC container objects.
Grant select on SCHEMA ::sandbox to sls_svc ;
Grant ALTER on SCHEMA ::sandbox to sls_svc ;
GRANT CONTROL ON SCHEMA::[sandbox ] TO [sls_svc];
Grant CREATE TABLE to sls_svc;
CREATE EXTERNAL TABLE sanbox.revenue-by-month
WITH (
LOCATION = '/ROW/revenue-by-month/',
DATA_SOURCE = ADLS-ROW,
FILE_FORMAT = EF_PARQUET
)
AS
SELECT * from table1;
all users in sls_svc role has STORAGE DATA CONTRIBUTOR access on READ-WRITE-CONTAINER (ROW)
Below are the error messages I am getting
I also tried creating a new database. hoping that i can grant full access on that database - so that cross DB query can work - but I am out of luck there as well.
Any thoughts ?
It seems that you have correctly set permissions https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/develop-storage-files-overview?tabs=impersonation#permissions
Are you sure that you can successfully execute just select statement and that the issue is not in SELECT part?
GRANT CONNECT to the database that was created
+
GRANT DDL_ADMIN access
resolved the issue
I am trying to create a master key for my database, needed because I follow a tutorial to make cross db queries.
Tutorial: https://www.scarydba.com/2016/03/21/cross-database-queries-in-azure-sql-database/
It is a similar question to Cannot create master key for master database in azure sql, but I get the message:
Msg 15578, Level 16, State 1, Line 1 There is already a master key in
the database. Please drop it before performing this statement.
...when I am trying to create a master key which is encrypted. What do I do wrong and is there another option to make cross db calls with Azure?
I execute the following:
CREATE MASTER KEY ENCRYPTION BY PASSWORD='SUPERSTRONGPASSWORD'
GO
Regards
According my experience, the master key has already created and exist when we create the Azure SQL database.
When we create master key :
-- Create a db master key if one does not already exist, using your own password.
CREATE MASTER KEY ENCRYPTION BY PASSWORD='<EnterStrongPasswordHere>';
Since the master key is already exist, we could not create a new one.
For most situations, Azure SQL database cross db queries don't need create the master key.
For example, I use the bellow query to do cross db query from Mydatabase to Mydatabase2:
--The "username" and "password" should be the username and password used to log in into the Customers database.
CREATE DATABASE SCOPED CREDENTIAL ElasticDBQueryCred
WITH IDENTITY = '<username>',
SECRET = '<password>';
--To create an external data source, execute the following command on the Orders database:
CREATE EXTERNAL DATA SOURCE MyElasticDBQueryDataSrc WITH
(TYPE = RDBMS,
LOCATION = '<server_name>.database.windows.net',
DATABASE_NAME = 'Customers',
CREDENTIAL = ElasticDBQueryCred,
) ;
--Create an external table on the Orders database, which matches the definition of the CustomerInformation table:
CREATE EXTERNAL TABLE [dbo].[test]
( [id] [int] ,
[age] [int]
)
WITH
( DATA_SOURCE = MyElasticDBQueryDataSrc)
--query the table
select * from test
Test table in Mydatabase2:
Cross db query in Mydatabase:
For details, please reference: Get started with cross-database queries.
Hope this helps.