Here is my script which I am trying to run in Azure SQL Database:
CREATE DATABASE SCOPED CREDENTIAL some_cred WITH IDENTITY = user1,
SECRET = '<Key of Blob Storage container>';
CREATE EXTERNAL DATA SOURCE TEST
WITH
(
TYPE=BLOB_STORAGE,
LOCATION='wasbs://<containername>#accountname.blob.core.windows.net',
CREDENTIAL= <somecred>`enter code here`
);
CREATE EXTERNAL TABLE dbo.test
(
val VARCHAR(255)
)
WITH
(
DATA_SOURCE = TEST
)
I am getting the following error:
External tables are not supported with the provided data source type.
My goal is to create external table in blob storage so that Hive query in HDInsight references to the same blob. The table needs to be managed through Azure SQL. What's wrong with this script?
Azure SQL Database does have the feature to load files stored in Blob Storage but it only via the BULK INSERT and OPENROWSET language features. See here for more information.
BULK INSERT dbo.test
FROM 'data/yourFile.txt'
WITH ( DATA_SOURCE = 'YourAzureBlobStorageAccount');
The way you have scripted it is more like an external table using Polybase which is only available in SQL Server 2016 and Azure SQL Data Warehouse at this time.
I'm thinking External tables can be used for Cross Database Querying (Elastic queries). So it couldn't able to use the External Data Source which is BLOB_STORAGE
Related
From what I understand SQL Managed Instance cannot access local or external file share.
We are trying to load data from a file to SQL Managed Instance, wanted recommendations on the best approach for that.
Use BULK INSERT from Azure blob storage https://techcommunity.microsoft.com/t5/Azure-SQL-Database/Loading-files-from-Azure-Blob-Storage-into-Azure-SQL-Database/ba-p/386133
Upload your files to azure storage
Create external data source to that location (put SAS token if it is not public):
CREATE EXTERNAL DATA SOURCE MyAzureBlobStorage
WITH ( TYPE = BLOB_STORAGE, LOCATION = 'https://myazureblobstorage.blob.core.windows.net'); ```
Load files from this account.
BULK INSERT Product
FROM 'data/product.dat'
WITH ( DATA_SOURCE = 'MyAzureBlobStorageAccount');
I am working on a project designed to run on azure platform using Azure database and Locally using local sql server. We have stored procedure that contain cross database calls. However It doesn't works on Azure server.I need cross database queries that capable of working in both azure and local sql server.
We an use Elastic Query which allows us to query across Azure SQL Databases
https://azure.microsoft.com/en-us/documentation/articles/sql-database-elastic-query-overview/
We can setup external data source in azure server Using following code. In this case we can execute cross database calls likes joining tables in a single database.
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'yourPassword';
CREATE DATABASE SCOPED CREDENTIAL yourServeradminlogin
WITH IDENTITY = 'yourServeradminlogin',
SECRET = 'yourPassword';
CREATE EXTERNAL DATA SOURCE RefmyDemoDB2
WITH
(
TYPE=RDBMS,
LOCATION='testdbdemoserver.database.windows.net',
DATABASE_NAME='myDemoDB2',
CREDENTIAL= yourServeradminlogin
);
CREATE EXTERNAL TABLE [dbo].[Department](
[DeptId] [int] NOT NULL,
[Name] [varchar](50) NULL
)
WITH
(
DATA_SOURCE = RefmyDemoDB2
);
It doesn't seem possible to use above method on our local SQL server without adding a third party database engine.
Install something called PolyBase to do cross database query in local sql server version 2017 and above.It only supported with external data source of type such as HADOOP rather than referencing another database(or any data storage) within the sql server.
We can accomplish that using following code
CREATE MASTER KEY ENCRYPTION BY PASSWORD='MyP#ssword123secretword';
CREATE DATABASE SCOPED CREDENTIAL mycredential
WITH IDENTITY = 'credential', Secret = 'secretkey'
CREATE EXTERNAL DATA SOURCE mycustomers
WITH (
TYPE = HADOOP,
LOCATION = 'wasbs://azurestorage.blob.core.windows.net/',
CREDENTIAL = mycredential
);
CREATE EXTERNAL FILE FORMAT csvformat
WITH (
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS (
FIELD_TERMINATOR = ','
)
);
CREATE EXTERNAL TABLE TableName
(
[did] [int] NOT NULL,
[Dname] [varchar] (50) NULL
)
WITH
(
LOCATION = '/',
DATA_SOURCE = mycustomers,
FILE_FORMAT = csvformat
)
Using HADOOP and polybase we can create an external data source in local sql server. But it create external data source of that external data storage. That is External data table located in that external storage.Exactly, my requirement is create an external data source of database within the sql server. So that i can use same corss data base query in both azure and local sql server
Is there any solution to solve this. or any solution to run cross database queries in both azure and local sql server?
You should not assume that using external tables in SQL Azure is remotely similar to doing cross-database queries in SQL Server. They are not the same thing and they have very different performance profiles. External tables is closer to linked servers than cross-database queries in SQL Server.
In SQL Server, cross-database queries are:
- running in the same SQL instance
- run the same basic execution code path to read data vs. single-database SQL Azure
- you have some slightly different transactional semantics vs. single-database operaitons in SQL Server, but the experience is generally similar from your perspective.
In SQL Azure singletons ("traditional") SQL Azure, you generally do not have the databases on the same physical machines. So, you have to cross-server queries (which are exposed using a linked-server like mechanism called external tables which support sharding/fan-out scenarios mostly). Trying to use this feature to simulate cross-database queries may functionally work, but it is not really a great plan due to the performance differences. There are also no real transactional guarantees in this path at all (no DTC).
SQL Azure Managed Instance does support cross-database queries within a single SQL Server instance internally. So, this would be the path that would be most similar for you to use if you really want to use cross-database queries. If you want to use SQL Azure single databases, you generally would not want to do cross-db queries at all for a legacy workload and you would want to rewrite to avoid the dependency. (Otherwise it will just be an ongoing headache due to the performance differences)
Is it possible to create an external table in Azure SQL of the data residing in Azure Table storage?
Answer is no.
I am currently facing similiar issue and this is my research so far:
Azure SQL Database doesn't allow Azure Table Storage as a external data source.
Sources:
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-data-source-transact-sql?view=sql-server-2017
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-file-format-transact-sql?view=sql-server-2017
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-table-transact-sql?view=sql-server-2017
Reason:
The possible data source scenarios are to copy from Hadoop (DataLake/Hive,..), Blob (Text files,csv) or RDBMS (another sql server). The Azure Table Storage is not listed.
The possible external data formats are only variations of text files/hadoop: Delimited Text, Hive RCFile, Hive ORC,Parquet.
Note - even copying from blob in JSON format requires implementing custom data format.
Workaround:
Create a copy pipeline with Azure Data Factory.
Create a copy
function/script with Azure Functions using C# and manually transfer
the data
Yes, there are a couple options. Please see the following:
CREATE EXTERNAL TABLE (Transact-SQL)
APPLIES TO: SQL Server (starting with 2016) Azure SQL Database Azure SQL Data Warehouse Parallel Data Warehouse
Creates an external table for PolyBase, or Elastic Database queries. Depending on the scenario, the syntax differs significantly. An external table created for PolyBase cannot be used for Elastic Database queries. Similarly, an external table created for Elastic Database queries cannot be used for PolyBase, etc.
CREATE EXTERNAL DATA SOURCE (Transact-SQL)
APPLIES TO: SQL Server (starting with 2016) Azure SQL Database Azure SQL Data Warehouse Parallel Data Warehouse
Creates an external data source for PolyBase, or Elastic Database queries. Depending on the scenario, the syntax differs significantly. An external data source created for PolyBase cannot be used for Elastic Database queries. Similarly, an external data source created for Elastic Database queries cannot be used for PolyBase, etc.
What is your use case?
I am trying to get access to my Azure blob storage via SQL. This (https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-table-transact-sql) article describes how it works.
I have tried the following SQL command:
-- PolyBase only: Azure Storage Blob as data source
-- (on SQL Server 2016 and Azure SQL Data Warehouse)
CREATE EXTERNAL DATA SOURCE dataSourceNameTestBlob
WITH (
TYPE = HADOOP,
LOCATION = 'wasb[s]://container#account_name.blob.core.windows.net'
[, CREDENTIAL = credential_name ]
)
[;]
Which results in the following error:
Msg 102, Level 15, State 1, Line 5 Incorrect syntax near 'HADOOP'.
When I google on this error I find that I need to use sql DW (.dsql) instead of .sql queries. However the article mentions that I can use a Azure SQL Database.
What am I doing wrong? I just only want to access my blob storage in SQL.
PolyBase scenario with Hadoop is only supported on SQL Server 2016 (or higher), Azure SQL Data Warehouse, and Parallel Data Warehouse.
Below is the T-SQL script against Azure SQL Database to store to blob storage.
CREATE EXTERNAL DATA SOURCE data_source_name
WITH (
TYPE = BLOB_STORAGE,
LOCATION = 'https://storage_account_name.blob.core.windows.net/container_name'
[, CREDENTIAL = credential_name ]
)
I'm trying to create an external data source to access Azure Blob Storage. However, I'm having issues with creating the actual data source.
I've followed the instructions located here:
Examples of bulk access to data in azure blob storage and
Create external data source - transact sql. I'm using SQL Server 2016 on a VM accessing via SSMS on a client machine using Windows Authentication with no issues. Instructions say creating this external data source works for SQL Server 2016 and Azure Blob Storage.
I have created the Master Key:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = <password>
and, the database scoped credential
CREATE DATABASE SCOPED CREDENTIAL UploadCountries
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET = <key>;
I have verified both of these exist in the database by querying sys.symmetric_keys and sys.database_scoped_credentials.
However, when I try executing the following code it says 'Incorrect syntax near 'EXTERNAL'
CREATE EXTERNAL DATA SOURCE BlobCountries
WITH (
TYPE = BLOB_STORAGE,
LOCATION = 'https://<somewhere>.table.core.windows.net/<somewhere>',
CREDENTIAL = UploadCountries
);
Your thoughts and help are appreciated!
Steve.
In “Examples of Bulk Access to Data in Azure Blob Storage”, we can find:
Bulk access to Azure blob storage from SQL Server, requires at least SQL Server 2017 CTP 1.1.
And in Arguments section of “CREATE EXTERNAL DATA SOURCE (Transact-SQL)”, we can find similar information:
Use BLOB_STORAGE when performing bulk operations using BULK INSERT or OPENROWSET with SQL Server 2017
You are using SQL Server 2016, so you get Incorrect syntax near 'EXTERNAL' error when you create external data source for Azure Blob storage.