I am trying to Create and query external tables from a file in Azure Data Lake from Azure Synapse Serverless SQL Pool using the following guide:
https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/develop-tables-external-tables?tabs=hadoop
Everything appears to be going fine.
I have created the following script to create an external table:
IF NOT EXISTS (SELECT * FROM sys.external_file_formats WHERE name = 'SynapseDelimitedTextFormat')
CREATE EXTERNAL FILE FORMAT [SynapseDelimitedTextFormat]
WITH ( FORMAT_TYPE = DELIMITEDTEXT ,
FORMAT_OPTIONS (
FIELD_TERMINATOR = ',',
USE_TYPE_DEFAULT = FALSE
))
GO
IF NOT EXISTS (SELECT * FROM sys.external_data_sources WHERE name = 'synapsefilename_synapselakev2_dfs_core_windows_net')
CREATE EXTERNAL DATA SOURCE [synapsefilename_synapselakev2_dfs_core_windows_net]
WITH (
LOCATION = 'abfss://synapsefilename#synapselakev2.dfs.core.windows.net'
)
GO
CREATE EXTERNAL TABLE GlobalOptionsetMetadata (
[C1] nvarchar(4000),
[C2] nvarchar(4000),
[C3] nvarchar(4000),
[C4] nvarchar(4000),
[C5] nvarchar(4000),
[C6] nvarchar(4000),
[C7] nvarchar(4000)
)
WITH (
LOCATION = 'GlobalOptionsetMetadata.csv',
DATA_SOURCE = [synapsefilename_synapselakev2_dfs_core_windows_net],
FILE_FORMAT = [SynapseDelimitedTextFormat]
)
GO
SELECT TOP 100 * FROM dbo.GlobalOptionsetMetadata
GO
However, when I click run I get the following error:
Operation CREATE EXTERNAL FILE FORMAT is not allowed for a replicated database.
Any thoughts?
it is most likely Database1 is a "replicated" lake database which you can't create external file formats. you have to select a Serverless/Dedicated SQL database while creating the External table. Try to change the database in "Use Database" combobox near to "Connect to" at top of query window and select Serverless/Dedicated SQL Database instead.
Related
This query used to work in Azure Data Factory pipeline but stopped working few days ago. Nothing changed in case of file names/ formats etc in Azure Blob storage. Getting error in this line:
SELECT * FROM OPENROWSET (
BULK
'/filepath.csv#snapshot=*', FORMAT = 'CSV'
)
The error says .csv#snapshot=* has URL suffix which is not allowed.
Full code:
-- CREATE OR REPLACE VIEW clean.barriers AS
IF EXISTS (SELECT * FROM sys.tables t
JOIN sys.schemas s ON (t.schema_id = s.schema_id)
WHERE s.name = 'clean' AND t.name = 'barriers')
EXEC('DROP EXTERNAL TABLE [clean].[barriers]')
CREATE EXTERNAL TABLE [clean].[barriers]
WITH
(
LOCATION = 'clean/synapse/barriers',
DATA_SOURCE = "",
FILE_FORMAT = [SynapseParquetFormat]
)
AS
SELECT * FROM OPENROWSET (
BULK
'/filepath.csv#snapshot=*', FORMAT = 'CSV'
)
WITH(
-- Schema adjusted to what we have in clean/barriers in Bigquery
mshp_id INT,
prog_name NVARCHAR(256),
barrier_name NVARCHAR(256),
days INT
) AS load_clean_data
As per the Official Documentation, you should have a Data source for the source file also from which you are trying to copy the data.
So, try to create a data source for the source CSV file and check, it may work.
Also, as you are executing the above script using ADF, first try to execute it without ADF and if the error occurs then problem can be with the script not ADF. If not try to change the activity of ADF and check.
You can try this trouble shoot also in your BULK path. As you want the data from that csv files folder give the path like below and check.
/folder/*.csv
I'm trying to create a view of data that resides in remote Azure SQL Server. I can't seem to create a table or temp table in the view to store the results of the sp_exeucte_remote call because that is not allowed so I tried to use a function, but then I get an error complaining about the following error based on the provided function definition.
Invalid use of a side-effecting operator 'INSERT EXEC' within a function
CREATE OR ALTER FUNCTION [dbo].[fn_Test]()
RETURNS #Results TABLE
(
[ID] INT,
[$ShardName] VARCHAR(500)
)
AS
BEGIN
INSERT INTO #Results
EXEC sp_execute_remote N'MyExternalDatasource', N'SELECT 1'
RETURN;
END
How can one create a view of data that exists on a remote Azure SQL Server where that data also exists as a view? FYI - the server where I'm trying to create the view is also Azure SQL Server.
Why are you using a FUNCTION?
As per Microsoft Documentation you cannot call a Stored Procedure within a function.
Also, your remote execution is only returning 1 column and you haven't defined the destination column to insert to.
The fact that the remote object is a VIEW does not matter. If we assume that the VIEW on the remote database has columns also named ID and [$shardname] then why not just use something like:
CREATE TABLE #results ([ID] int, [$shardname] varchar(500))
INSERT #results ([ID], [$shardname])
EXEC sp_execute_remote N'ExternalSource', N'SELECT [ID], [$shardname] FROM RemoteTableName'
We have a scenario where the source csv files are isolated by Customer i.e., each Customer will have a Container in the Azure Storage.
When creating External Table in SQL Synapse, is it possible to pass the Container name as parameter that way there are not multiple External Data Tables in SQL Synapse DB?
CREATE EXTERNAL DATA SOURCE AzureBlobStorage with (
TYPE = HADOOP,
LOCATION ='wasbs://<**container100**>#<accountname>.blob.core.windows.net',
CREDENTIAL = AzureStorageCredential
);
CREATE EXTERNAL TABLE [dbo].[res1_Data] (
[ID] INT,
[UniqueId] VARCHAR(50),
[Status] VARCHAR(50) NULL,
[JoinedDate] DATE
)
WITH (LOCATION='<**container2**>/<folder>/<file>.csv',
DATA_SOURCE = AzureBlobStorage,
FILE_FORMAT = CEFormat
);
Unfortunately you can't use variables within DDL commands. However, you can build dynamic statements and then execute with sp_executesql to do this.
More information here.
I am trying to access sys.SQL_Logins on Master DB from a User Database so that I could join sys.sql_logins with sys.sysusers.
UserDatabase
SELECT DP.NAME, DP.sid, SU.sid
FROM sys.sql_logins DP
INNER JOIN SYS.sysusers SU
ON DP.name=SU.name
In Azure I am not able to do this, since I am not able to call master DB from UserDatbase. My objective was to compare SID of sys.sql_logins with sys.sqlusers from User Database.
Is there any way I could join these two tables in Azure.
You can only execute queries against a single database. However, you can set up an external table to point to the master database views. This code should get you there:
CREATE MASTER KEY; -- if your DB doesn't already have one
GO
CREATE DATABASE SCOPED CREDENTIAL AccessToMaster
WITH IDENTITY = 'yourmasterlogin', SECRET = 'yourpassword'; --TODO set a login name and password that can access the master database
GO
CREATE EXTERNAL DATA SOURCE MasterDatabase
WITH
(
TYPE = RDBMS,
LOCATION = N'yourserver.database.windows.net', --TODO replace your server name
DATABASE_NAME = N'master',
CREDENTIAL = AccessToMaster
);
GO
CREATE SCHEMA MasterDB AUTHORIZATION dbo;
GO
CREATE EXTERNAL TABLE MasterDB.sql_logins
(
[name] sysname,
principal_id int,
[sid] varbinary(85),
[is_disabled] bit
)
WITH
(
DATA_SOURCE = MasterDatabase,
SCHEMA_NAME = 'sys',
OBJECT_NAME = 'sql_logins'
);
GO
SELECT * FROM MasterDB.sql_logins;
GO
Is it possible to execute a SQL Select statement on a CSV file on a Sybase Database?
Update DBA.user_data
set user_data.date_Sent = '12/16/2015'
where user_data.caseid in (select caseid
from DBA.cases
where cases.caseid=user_data.caseid
And cases.caseid in (select * FROM 'C:\\example\\test.csv' WITH (FIELDTERMINATOR = ',',ROWTERMINATOR = '\n')));
Assuming you are using Sybase ASE, you can access flat files using the Component Integration Services (CIS).
I suggest you check out the Component Integration Services User Guide, which is part of the SAP/Sybase documentation.
Check out the section on File system access: File Access
You will create a proxy (or existing) table, using the file information in the definition.
create proxy_table <table_name>
external file at " pathname" [column delimiter “<string>”]
OR
create existing table fname (
column1 int null,
column2 datetime null,
column3 varchar(1024) null
etc. etc.
) external file at "pathname" [column delimiter “<string>”]
Only the select, insert, and truncate table statements are supported
for file access. update and delete result in errors if the file proxy
is the target of these commands.
Documentation: create proxy_table