We have a scenario where the source csv files are isolated by Customer i.e., each Customer will have a Container in the Azure Storage.
When creating External Table in SQL Synapse, is it possible to pass the Container name as parameter that way there are not multiple External Data Tables in SQL Synapse DB?
CREATE EXTERNAL DATA SOURCE AzureBlobStorage with (
TYPE = HADOOP,
LOCATION ='wasbs://<**container100**>#<accountname>.blob.core.windows.net',
CREDENTIAL = AzureStorageCredential
);
CREATE EXTERNAL TABLE [dbo].[res1_Data] (
[ID] INT,
[UniqueId] VARCHAR(50),
[Status] VARCHAR(50) NULL,
[JoinedDate] DATE
)
WITH (LOCATION='<**container2**>/<folder>/<file>.csv',
DATA_SOURCE = AzureBlobStorage,
FILE_FORMAT = CEFormat
);
Unfortunately you can't use variables within DDL commands. However, you can build dynamic statements and then execute with sp_executesql to do this.
More information here.
Related
This query used to work in Azure Data Factory pipeline but stopped working few days ago. Nothing changed in case of file names/ formats etc in Azure Blob storage. Getting error in this line:
SELECT * FROM OPENROWSET (
BULK
'/filepath.csv#snapshot=*', FORMAT = 'CSV'
)
The error says .csv#snapshot=* has URL suffix which is not allowed.
Full code:
-- CREATE OR REPLACE VIEW clean.barriers AS
IF EXISTS (SELECT * FROM sys.tables t
JOIN sys.schemas s ON (t.schema_id = s.schema_id)
WHERE s.name = 'clean' AND t.name = 'barriers')
EXEC('DROP EXTERNAL TABLE [clean].[barriers]')
CREATE EXTERNAL TABLE [clean].[barriers]
WITH
(
LOCATION = 'clean/synapse/barriers',
DATA_SOURCE = "",
FILE_FORMAT = [SynapseParquetFormat]
)
AS
SELECT * FROM OPENROWSET (
BULK
'/filepath.csv#snapshot=*', FORMAT = 'CSV'
)
WITH(
-- Schema adjusted to what we have in clean/barriers in Bigquery
mshp_id INT,
prog_name NVARCHAR(256),
barrier_name NVARCHAR(256),
days INT
) AS load_clean_data
As per the Official Documentation, you should have a Data source for the source file also from which you are trying to copy the data.
So, try to create a data source for the source CSV file and check, it may work.
Also, as you are executing the above script using ADF, first try to execute it without ADF and if the error occurs then problem can be with the script not ADF. If not try to change the activity of ADF and check.
You can try this trouble shoot also in your BULK path. As you want the data from that csv files folder give the path like below and check.
/folder/*.csv
I am trying to Create and query external tables from a file in Azure Data Lake from Azure Synapse Serverless SQL Pool using the following guide:
https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/develop-tables-external-tables?tabs=hadoop
Everything appears to be going fine.
I have created the following script to create an external table:
IF NOT EXISTS (SELECT * FROM sys.external_file_formats WHERE name = 'SynapseDelimitedTextFormat')
CREATE EXTERNAL FILE FORMAT [SynapseDelimitedTextFormat]
WITH ( FORMAT_TYPE = DELIMITEDTEXT ,
FORMAT_OPTIONS (
FIELD_TERMINATOR = ',',
USE_TYPE_DEFAULT = FALSE
))
GO
IF NOT EXISTS (SELECT * FROM sys.external_data_sources WHERE name = 'synapsefilename_synapselakev2_dfs_core_windows_net')
CREATE EXTERNAL DATA SOURCE [synapsefilename_synapselakev2_dfs_core_windows_net]
WITH (
LOCATION = 'abfss://synapsefilename#synapselakev2.dfs.core.windows.net'
)
GO
CREATE EXTERNAL TABLE GlobalOptionsetMetadata (
[C1] nvarchar(4000),
[C2] nvarchar(4000),
[C3] nvarchar(4000),
[C4] nvarchar(4000),
[C5] nvarchar(4000),
[C6] nvarchar(4000),
[C7] nvarchar(4000)
)
WITH (
LOCATION = 'GlobalOptionsetMetadata.csv',
DATA_SOURCE = [synapsefilename_synapselakev2_dfs_core_windows_net],
FILE_FORMAT = [SynapseDelimitedTextFormat]
)
GO
SELECT TOP 100 * FROM dbo.GlobalOptionsetMetadata
GO
However, when I click run I get the following error:
Operation CREATE EXTERNAL FILE FORMAT is not allowed for a replicated database.
Any thoughts?
it is most likely Database1 is a "replicated" lake database which you can't create external file formats. you have to select a Serverless/Dedicated SQL database while creating the External table. Try to change the database in "Use Database" combobox near to "Connect to" at top of query window and select Serverless/Dedicated SQL Database instead.
Is there any option to query the Azure SQL Data warehouse (Cloud) from the Azure SQL Server database (Cloud)?
We have a central warehouse hosted in cloud with all the domains needed for the Application. It would be great if we can use those tables as external tables using the approach described at https://azure.microsoft.com/en-us/blog/querying-remote-databases-in-azure-sql-db.
Querying Data Warehouse from SQL DB is currently unsupported.
You can access Azure SQL Data Warehouse from Azure SQL using external tables
Here's the article: https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-table-transact-sql
Example:
CREATE MASTER KEY
CREATE DATABASE SCOPED CREDENTIAL dwcredential WITH IDENTITY = '<username>',
SECRET = '<password>';
CREATE EXTERNAL DATA SOURCE sqldw WITH
(
TYPE = RDBMS,
LOCATION = N'<dwserver>.database.windows.net',
DATABASE_NAME = N'<dwdatabasename>',
CREDENTIAL = dwcredential
)
GO
CREATE EXTERNAL TABLE [<azuresqltablename>](
[col1] [int] NOT NULL,
[col2] [int] NOT NULL
)
WITH
(
DATA_SOURCE = sqldw,
SCHEMA_NAME = 'dbo', --schema name of remote table
OBJECT_NAME = '<dwtablename>' --table name of remote table
);
I have created a table in Azure SQL Data Warehouse as below:
CREATE TABLE dbo.test_lob_type
(
id VARCHAR(80) NOT NULL,
mime_type VARCHAR(80) NOT NULL,
binary_lob VARBINARY(MAX) NULL
)
WITH
(
DISTRIBUTION = HASH ( id ),
CLUSTERED INDEX ( id ASC )
);
I want to insert a BLOB object into this table. I tried to achieve this using the OPENROWSET command as pointed in the link How to insert a blob into a database using sql server management studio
But unfortunately this command does not work with Azure SQL DW. Can anyone provide any input on how to insert any BLOB object into a SQL DW table from the command line?
bcp is supported for this scenario. Here is a simple example using SQL Authentication and char format:
REM Example using SQL Authentication and character file
bcp dbo.test_lob_type in test_lob_type.bcp -S yourDWServer.database.windows.net -d yourDWName -U yourLogin -P yourPassword -c
If your file only contains the blob, consider loading to a staging table before inserting into the main table.
Is it possible to execute a SQL Select statement on a CSV file on a Sybase Database?
Update DBA.user_data
set user_data.date_Sent = '12/16/2015'
where user_data.caseid in (select caseid
from DBA.cases
where cases.caseid=user_data.caseid
And cases.caseid in (select * FROM 'C:\\example\\test.csv' WITH (FIELDTERMINATOR = ',',ROWTERMINATOR = '\n')));
Assuming you are using Sybase ASE, you can access flat files using the Component Integration Services (CIS).
I suggest you check out the Component Integration Services User Guide, which is part of the SAP/Sybase documentation.
Check out the section on File system access: File Access
You will create a proxy (or existing) table, using the file information in the definition.
create proxy_table <table_name>
external file at " pathname" [column delimiter “<string>”]
OR
create existing table fname (
column1 int null,
column2 datetime null,
column3 varchar(1024) null
etc. etc.
) external file at "pathname" [column delimiter “<string>”]
Only the select, insert, and truncate table statements are supported
for file access. update and delete result in errors if the file proxy
is the target of these commands.
Documentation: create proxy_table