Is there any option to query the Azure SQL Data warehouse (Cloud) from the Azure SQL Server database (Cloud)?
We have a central warehouse hosted in cloud with all the domains needed for the Application. It would be great if we can use those tables as external tables using the approach described at https://azure.microsoft.com/en-us/blog/querying-remote-databases-in-azure-sql-db.
Querying Data Warehouse from SQL DB is currently unsupported.
You can access Azure SQL Data Warehouse from Azure SQL using external tables
Here's the article: https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-table-transact-sql
Example:
CREATE MASTER KEY
CREATE DATABASE SCOPED CREDENTIAL dwcredential WITH IDENTITY = '<username>',
SECRET = '<password>';
CREATE EXTERNAL DATA SOURCE sqldw WITH
(
TYPE = RDBMS,
LOCATION = N'<dwserver>.database.windows.net',
DATABASE_NAME = N'<dwdatabasename>',
CREDENTIAL = dwcredential
)
GO
CREATE EXTERNAL TABLE [<azuresqltablename>](
[col1] [int] NOT NULL,
[col2] [int] NOT NULL
)
WITH
(
DATA_SOURCE = sqldw,
SCHEMA_NAME = 'dbo', --schema name of remote table
OBJECT_NAME = '<dwtablename>' --table name of remote table
);
Related
I am trying to Create and query external tables from a file in Azure Data Lake from Azure Synapse Serverless SQL Pool using the following guide:
https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/develop-tables-external-tables?tabs=hadoop
Everything appears to be going fine.
I have created the following script to create an external table:
IF NOT EXISTS (SELECT * FROM sys.external_file_formats WHERE name = 'SynapseDelimitedTextFormat')
CREATE EXTERNAL FILE FORMAT [SynapseDelimitedTextFormat]
WITH ( FORMAT_TYPE = DELIMITEDTEXT ,
FORMAT_OPTIONS (
FIELD_TERMINATOR = ',',
USE_TYPE_DEFAULT = FALSE
))
GO
IF NOT EXISTS (SELECT * FROM sys.external_data_sources WHERE name = 'synapsefilename_synapselakev2_dfs_core_windows_net')
CREATE EXTERNAL DATA SOURCE [synapsefilename_synapselakev2_dfs_core_windows_net]
WITH (
LOCATION = 'abfss://synapsefilename#synapselakev2.dfs.core.windows.net'
)
GO
CREATE EXTERNAL TABLE GlobalOptionsetMetadata (
[C1] nvarchar(4000),
[C2] nvarchar(4000),
[C3] nvarchar(4000),
[C4] nvarchar(4000),
[C5] nvarchar(4000),
[C6] nvarchar(4000),
[C7] nvarchar(4000)
)
WITH (
LOCATION = 'GlobalOptionsetMetadata.csv',
DATA_SOURCE = [synapsefilename_synapselakev2_dfs_core_windows_net],
FILE_FORMAT = [SynapseDelimitedTextFormat]
)
GO
SELECT TOP 100 * FROM dbo.GlobalOptionsetMetadata
GO
However, when I click run I get the following error:
Operation CREATE EXTERNAL FILE FORMAT is not allowed for a replicated database.
Any thoughts?
it is most likely Database1 is a "replicated" lake database which you can't create external file formats. you have to select a Serverless/Dedicated SQL database while creating the External table. Try to change the database in "Use Database" combobox near to "Connect to" at top of query window and select Serverless/Dedicated SQL Database instead.
We have a scenario where the source csv files are isolated by Customer i.e., each Customer will have a Container in the Azure Storage.
When creating External Table in SQL Synapse, is it possible to pass the Container name as parameter that way there are not multiple External Data Tables in SQL Synapse DB?
CREATE EXTERNAL DATA SOURCE AzureBlobStorage with (
TYPE = HADOOP,
LOCATION ='wasbs://<**container100**>#<accountname>.blob.core.windows.net',
CREDENTIAL = AzureStorageCredential
);
CREATE EXTERNAL TABLE [dbo].[res1_Data] (
[ID] INT,
[UniqueId] VARCHAR(50),
[Status] VARCHAR(50) NULL,
[JoinedDate] DATE
)
WITH (LOCATION='<**container2**>/<folder>/<file>.csv',
DATA_SOURCE = AzureBlobStorage,
FILE_FORMAT = CEFormat
);
Unfortunately you can't use variables within DDL commands. However, you can build dynamic statements and then execute with sp_executesql to do this.
More information here.
I am trying to access sys.SQL_Logins on Master DB from a User Database so that I could join sys.sql_logins with sys.sysusers.
UserDatabase
SELECT DP.NAME, DP.sid, SU.sid
FROM sys.sql_logins DP
INNER JOIN SYS.sysusers SU
ON DP.name=SU.name
In Azure I am not able to do this, since I am not able to call master DB from UserDatbase. My objective was to compare SID of sys.sql_logins with sys.sqlusers from User Database.
Is there any way I could join these two tables in Azure.
You can only execute queries against a single database. However, you can set up an external table to point to the master database views. This code should get you there:
CREATE MASTER KEY; -- if your DB doesn't already have one
GO
CREATE DATABASE SCOPED CREDENTIAL AccessToMaster
WITH IDENTITY = 'yourmasterlogin', SECRET = 'yourpassword'; --TODO set a login name and password that can access the master database
GO
CREATE EXTERNAL DATA SOURCE MasterDatabase
WITH
(
TYPE = RDBMS,
LOCATION = N'yourserver.database.windows.net', --TODO replace your server name
DATABASE_NAME = N'master',
CREDENTIAL = AccessToMaster
);
GO
CREATE SCHEMA MasterDB AUTHORIZATION dbo;
GO
CREATE EXTERNAL TABLE MasterDB.sql_logins
(
[name] sysname,
principal_id int,
[sid] varbinary(85),
[is_disabled] bit
)
WITH
(
DATA_SOURCE = MasterDatabase,
SCHEMA_NAME = 'sys',
OBJECT_NAME = 'sql_logins'
);
GO
SELECT * FROM MasterDB.sql_logins;
GO
I am following the Microsoft tutorial
for vertical elastic querying at https://learn.microsoft.com/en-us/azure/sql-database/sql-database-elastic-query-getting-started-vertical
I entered the code as follows:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'redacted';
CREATE DATABASE SCOPED CREDENTIAL ElasticDBQueryCred
WITH IDENTITY = 'redacted',
SECRET = 'redacted';
CREATE EXTERNAL DATA SOURCE MyElasticDBQueryDataSrc WITH
(TYPE = RDBMS,
LOCATION = 'redactedazure.database.windows.net',
DATABASE_NAME = 'Redacted_Staging',
CREDENTIAL = ElasticDBQueryCred,
);
CREATE EXTERNAL TABLE [dbo].[CustomerInformation]
( [CustomerID] [int] NOT NULL,
[CustomerName] [varchar](50) NOT NULL,
[Company] [varchar](50) NOT NULL)
WITH
( DATA_SOURCE = MyElasticDBQueryDataSrc)
SELECT OrderInformation.CustomerID, OrderInformation.OrderId, CustomerInformation.CustomerName, CustomerInformation.Company
FROM OrderInformation
INNER JOIN CustomerInformation
ON CustomerInformation.CustomerID = OrderInformation.CustomerID;
However, I am getting the following errors:
Msg 46823, Level 16, State 1, Line 1
Error retrieving data from redactedazure.database.windows.net.Redacted_Staging. The underlying error message received was: 'Error retrieving data from redactedazure.database.windows.net.Redacted_Staging. The underlying error message received was: 'Error retrieving data from... (repeats)
Any idea why?
Did you ensure you possess ALTER ANY EXTERNAL DATA SOURCE permission?
ALTER ANY EXTERNAL DATA SOURCE permissions are needed to refer to the underlying data source.
You can see: Vertical partitioning - cross-database queries
Hope this can helps you.
I have created a table in Azure SQL Data Warehouse as below:
CREATE TABLE dbo.test_lob_type
(
id VARCHAR(80) NOT NULL,
mime_type VARCHAR(80) NOT NULL,
binary_lob VARBINARY(MAX) NULL
)
WITH
(
DISTRIBUTION = HASH ( id ),
CLUSTERED INDEX ( id ASC )
);
I want to insert a BLOB object into this table. I tried to achieve this using the OPENROWSET command as pointed in the link How to insert a blob into a database using sql server management studio
But unfortunately this command does not work with Azure SQL DW. Can anyone provide any input on how to insert any BLOB object into a SQL DW table from the command line?
bcp is supported for this scenario. Here is a simple example using SQL Authentication and char format:
REM Example using SQL Authentication and character file
bcp dbo.test_lob_type in test_lob_type.bcp -S yourDWServer.database.windows.net -d yourDWName -U yourLogin -P yourPassword -c
If your file only contains the blob, consider loading to a staging table before inserting into the main table.