Azure SQL - Cannot create encrypted master key - azure-sql-database

I am trying to create a master key for my database, needed because I follow a tutorial to make cross db queries.
Tutorial: https://www.scarydba.com/2016/03/21/cross-database-queries-in-azure-sql-database/
It is a similar question to Cannot create master key for master database in azure sql, but I get the message:
Msg 15578, Level 16, State 1, Line 1 There is already a master key in
the database. Please drop it before performing this statement.
...when I am trying to create a master key which is encrypted. What do I do wrong and is there another option to make cross db calls with Azure?
I execute the following:
CREATE MASTER KEY ENCRYPTION BY PASSWORD='SUPERSTRONGPASSWORD'
GO
Regards

According my experience, the master key has already created and exist when we create the Azure SQL database.
When we create master key :
-- Create a db master key if one does not already exist, using your own password.
CREATE MASTER KEY ENCRYPTION BY PASSWORD='<EnterStrongPasswordHere>';
Since the master key is already exist, we could not create a new one.
For most situations, Azure SQL database cross db queries don't need create the master key.
For example, I use the bellow query to do cross db query from Mydatabase to Mydatabase2:
--The "username" and "password" should be the username and password used to log in into the Customers database.
CREATE DATABASE SCOPED CREDENTIAL ElasticDBQueryCred
WITH IDENTITY = '<username>',
SECRET = '<password>';
--To create an external data source, execute the following command on the Orders database:
CREATE EXTERNAL DATA SOURCE MyElasticDBQueryDataSrc WITH
(TYPE = RDBMS,
LOCATION = '<server_name>.database.windows.net',
DATABASE_NAME = 'Customers',
CREDENTIAL = ElasticDBQueryCred,
) ;
--Create an external table on the Orders database, which matches the definition of the CustomerInformation table:
CREATE EXTERNAL TABLE [dbo].[test]
( [id] [int] ,
[age] [int]
)
WITH
( DATA_SOURCE = MyElasticDBQueryDataSrc)
--query the table
select * from test
Test table in Mydatabase2:
Cross db query in Mydatabase:
For details, please reference: Get started with cross-database queries.
Hope this helps.

Related

How do I create a DATABASE SCOPED CREDENTIAL in Synapse Serverless?

I have a Serverless SQL pool set up in Azure Synapse Analytics, and I am trying to run this query:
CREATE DATABASE SCOPED CREDENTIAL myCredential
WITH IDENTITY = 'test',
SECRET = 'test2';
When I run the query I get this error:
Incorrect syntax near 'IDENTITY'.
How can I correct this issue?
Please use the below format :
USE [master]
GO
-- Create the lake house logic database
IF db_id('nyctaxidwdelta') IS NULL
EXEC('CREATE DATABASE nyctaxidwdelta COLLATE Latin1_General_100_BIN2_UTF8')
GO
USE [nyctaxidwdelta]
GO
-- Create a master key
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'blabla!'
GO
-- Create database scoped credential that use Synapse Managed Identity
CREATE DATABASE SCOPED CREDENTIAL WorkspaceIdentity
WITH IDENTITY = 'Managed Identity'
GO
-- Create external data source
IF NOT EXISTS (SELECT * FROM sys.external_data_sources WHERE name = 'eds_nyctaxi')
CREATE EXTERNAL DATA SOURCE [eds_nyctaxi]
WITH (
LOCATION = 'https://mystorage.dfs.core.windows.net/lakedata/',
CREDENTIAL = WorkspaceIdentity
)
GO

What would happen if I were to drop this CREDENTIAL in my Azure DB

My main goal is to get Cross-Database queries working between two Azure Databases. In order to do so, I've come across Elastic Queries. The examples I've seen on how to get started with Elastic Queries show people creating a Master Key with encryption by password like so:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<password>';
GO
CREATE DATABASE SCOPED CREDENTIAL ElasticDBCredential WITH IDENTITY = 'ElasticUser', SECRET = '<password>';
GO
Although my Database already has a Master Key, and I get this message when I try and drop the Master Key
Cannot drop master key because CREDENTIAL
'https://****.windows.net/sqldbauditlogs' is
encrypted by it.
I have Audit Logs turned on for this DB and it is encrypted by the existing Master Key, apparently. I tried dropping the Credential doing this:
DROP DATABASE SCOPED CREDENTIAL [https://****.windows.net/sqldbauditlogs]
But this also returns the error:
Cannot drop the credential 'https://****.windows.net/sqldbauditlogs' because it is being used.
How would I make it so it's "not in use" so that I can drop it OR I could use the existing Master Key, I just don't know how to figure out what it is currently...
You need to first ALTER the master key and regenerate it using the new password as follow:
ALTER MASTER KEY REGENERATE WITH ENCRYPTION BY PASSWORD = 'NewPassword';
Any CREDENTIAL using old master key will automatically inherit the new master key since there can only be one master key.
Now, drop the DATABASE ENCRYPTION KEY and then drop the CERTIFICATE and last drop the MASTER KEY.
The following example removes the database encryption and drops the database encryption key.
ALTER DATABASE AdventureWorks2012
SET ENCRYPTION OFF;
GO
/* Wait for decryption operation to complete, look for a
value of 1 in the query below. */
SELECT encryption_state
FROM sys.dm_database_encryption_keys;
GO
USE AdventureWorks2012;
GO
DROP DATABASE ENCRYPTION KEY;
GO
Drop the credentials.
DROP DATABASE SCOPED CREDENTIAL credential_name
Drop external Table [EXTERNAL TABLE Name]
Drop external data source [Data Source Name]
Drop DATABASE SCOPED CREDENTIAL [Name]
Drop Master key

Azure Syanpse Analytics

I have a need to connect to Synapse Analytics Serverless SQL Pool database using SQL Authentication.
I created a serverless SQL Pool database and created a SQL User and provided db_owner access.
Then created an external table below
IF NOT EXISTS (SELECT * FROM sys.external_file_formats
WHERE name = 'SynapseDeltaFormat')
CREATE EXTERNAL FILE FORMAT [SynapseDeltaFormat]
WITH ( FORMAT_TYPE = PARQUET)
GO
IF NOT EXISTS (SELECT * FROM sys.external_data_sources WHERE name =
'test_dfs_core_windows_net')
CREATE EXTERNAL DATA SOURCE [test_dfs_core_windows_net]
WITH (
LOCATION = 'abfss://test.dfs.core.windows.net'
)
GO
CREATE EXTERNAL TABLE table_staging (
<columns here>
)
WITH (
LOCATION = 'bronze/table_staging/',
DATA_SOURCE = [test_dfs_core_windows_net],
FILE_FORMAT = [SynapseDeltaFormat]
)
GO
SELECT TOP 100 * FROM dbo.table_staging
GO
Get below error when trying to access data of the table using SQL User
External table 'dbo.table_staging' is not accessible because location does not exist or it is used by another process.
Table data is accessible using AD user. Created DataSource using SQL User.
It seems like that SQL Server User does not have access to data lake/data storage. How to grant that access?
Just started delta lakes myself :)
Per default the serverless sql authenticates using the user context. When you are querying from within the synapse data studio you are using your AD users context which is why you can connect to the external storage.
However for SQL users and AD users that do not have access to be able to query you need to use credentials when setting up the query.
You can finde detailed instructions here:
https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/develop-storage-files-storage-access-control?tabs=user-identity
You need to have way of accessing the storage - either service principal, SAS, Managed Identity
I just set up credentials using a service principal "App registration"
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<secret>';
go
CREATE DATABASE SCOPED CREDENTIAL [credentialname] WITH
IDENTITY = '<Client-ID>#https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token'
, SECRET = '<token>'
GO
CREATE EXTERNAL DATA SOURCE mydatasource WITH (
LOCATION = 'https://<storageaccount>.dfs.core.windows.net/onedatahubtest',
CREDENTIAL = credentialname
);
CREATE VIEW MYVIEW AS
SELECT *
FROM OPENROWSET(
BULK 'Example/table',
DATA_SOURCE='mydatasource',
FORMAT = 'delta') as rows;
And now my sql user can access the view

Not able to do cross database query in SQL Azure

I have 2 DB on same SQL Azure server and i have same table(TB1) on both DB, now i want to read data from TB1 of DB2 and insert data into TB1 of DB1.
I am using below query but getting error.
insert into TB1 select 1,* from [DB2].dbo.TB1
Error Message
Msg 40515, Level 15, State 1, Line 16
Reference to database and/or server name in 'DB2.dbo.TB1' is not supported in this version of SQL Server.
Yes, you can use the Elastic Query Features on SQL Azure.It's the only way you can perform the cross database Queries.
Here are the detailed Queries to follow:
Run the below Query in your DB1(Since you said like reading the TB1 from DB2 and insert those Data's into your TB2 in your DB1)
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'STro*ngPaSSe0rD';
CREATE DATABASE SCOPED CREDENTIAL Login
WITH IDENTITY = 'Login',
SECRET = 'STro*ngPaSSe0rD';
CREATE EXTERNAL DATA SOURCE RemoteReferenceData
WITH
(
TYPE=RDBMS,
LOCATION='myserver.database.windows.net',
DATABASE_NAME='DB2',
CREDENTIAL= Login
);
CREATE EXTERNAL TABLE [dbo].[TB1]
(
[Columns] [DataTypes]
)
WITH (DATA_SOURCE = [RemoteReferenceData])
After these step, you can Query the external table like the Normal table. Though some limitations while using the External table, like you couldn't able to Insert Data's into a EXTERNAL TABLE(Reference table)
Azure supports this cross database query feature since 2015 but needs some extra setup to work and Elastic Query.
The first step is create a security credential:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<password>';
CREATE DATABASE SCOPED CREDENTIAL DB2Security
WITH IDENTITY = '<username>',
SECRET = '<password>';
The "username" and "password" should be the username and password used to login into the DB2 database.
Now you can use it to define a external datasource, so DB1 can connect to DB2:
CREATE EXTERNAL DATA SOURCE DB2Access
WITH (
TYPE=RDBMS,
LOCATION='myservernotyours.database.secure.windows.net',
DATABASE_NAME='DB2',
CREDENTIAL= DB2Security);
Finally, you map the TB1 as a external table from the DB2 database, using the previous external datasource:
CREATE EXTERNAL TABLE dbo.TB1FromDB2(
ID int,
Val varchar(50))
WITH
(
DATA_SOURCE = DB2Access);
You can also accomplish this using the Azure SQL Data Sync, but the data are replicated in one single database and this feature are still a preview version (May/2018) and you always see oldest data (the minimal configurable interval for each synchronization is 5 minutes).
You can perform cross database queries using the elastic query features on SQL Azure.
You will have to create an external data source and an external table to be able to query tables on other SQL Azure databases. This article shows how to do it.
Hope this helps.

Postgresql create stored database link

I know that in Oracle it's possible to create stored dblink and after that use it in query. For example:
Script for creation dblink:
CREATE PUBLIC DATABASE LINK my_link CONNECT TO my_schema IDENTIFIED BY shema_password USING 'remote';
And after that we can use it in our queries:
SELECT * FROM some_table#my_link;
I didn't find same solution for Postgres. I undestood that we can create named dblink connection:
For this we must use dblink_connect with name param. But created named dblink will destroy after session close.
Or we can create dblink connection for every queries:
SELECT *
FROM dblink('host= port= dbname= user= password=',
'select table_schema, table_name from information_schema.tables where table_schema = ''data''') AS t1 (table_schema TEXT, table_name TEXT);
Is it possible create stored dblink in Postgres and use it in different queries? Or I should create some function that return dblink connection params which encapsulate them?
I try use foreign table and do next steps:
Create postgres_fdw extension:
CREATE EXTENSION IF NOT EXISTS postgres_fdw;
Create Server:
CREATE SERVER my_server FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '-', port '-', dbname '-');
And create mapping for user 'sys' where set remote user and password:
CREATE USER MAPPING FOR sys SERVER light_house OPTIONS ( USER 'remote_user', PASSWORD 'remove_password');
GRANT USAGE ON FOREIGN SERVER my_server TO sys;
Create foreign table in schema:
CREATE FOREIGN TABLE system.my_local_table (
colums ..
) SERVER my_server OPTIONS (schema_name 'remote_user', table_name 'remote_table'
);
GRANT SELECT ON TABLE system.home_measurement TO argus_sys;
after that I catch next exception:
[2F003] ERROR: password is required
Description: Non-superuser cannot connect if the server does not request a password.
Help: Target server's authentication method must be changed.
You should use a foreign table.
To get rid of the error message, change the pg_hba.conf file on the remote database server to use md5 authentication (don't forget to reload with pg_ctl reload).