I am following the Microsoft tutorial
for vertical elastic querying at https://learn.microsoft.com/en-us/azure/sql-database/sql-database-elastic-query-getting-started-vertical
I entered the code as follows:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'redacted';
CREATE DATABASE SCOPED CREDENTIAL ElasticDBQueryCred
WITH IDENTITY = 'redacted',
SECRET = 'redacted';
CREATE EXTERNAL DATA SOURCE MyElasticDBQueryDataSrc WITH
(TYPE = RDBMS,
LOCATION = 'redactedazure.database.windows.net',
DATABASE_NAME = 'Redacted_Staging',
CREDENTIAL = ElasticDBQueryCred,
);
CREATE EXTERNAL TABLE [dbo].[CustomerInformation]
( [CustomerID] [int] NOT NULL,
[CustomerName] [varchar](50) NOT NULL,
[Company] [varchar](50) NOT NULL)
WITH
( DATA_SOURCE = MyElasticDBQueryDataSrc)
SELECT OrderInformation.CustomerID, OrderInformation.OrderId, CustomerInformation.CustomerName, CustomerInformation.Company
FROM OrderInformation
INNER JOIN CustomerInformation
ON CustomerInformation.CustomerID = OrderInformation.CustomerID;
However, I am getting the following errors:
Msg 46823, Level 16, State 1, Line 1
Error retrieving data from redactedazure.database.windows.net.Redacted_Staging. The underlying error message received was: 'Error retrieving data from redactedazure.database.windows.net.Redacted_Staging. The underlying error message received was: 'Error retrieving data from... (repeats)
Any idea why?
Did you ensure you possess ALTER ANY EXTERNAL DATA SOURCE permission?
ALTER ANY EXTERNAL DATA SOURCE permissions are needed to refer to the underlying data source.
You can see: Vertical partitioning - cross-database queries
Hope this can helps you.
Related
This query used to work in Azure Data Factory pipeline but stopped working few days ago. Nothing changed in case of file names/ formats etc in Azure Blob storage. Getting error in this line:
SELECT * FROM OPENROWSET (
BULK
'/filepath.csv#snapshot=*', FORMAT = 'CSV'
)
The error says .csv#snapshot=* has URL suffix which is not allowed.
Full code:
-- CREATE OR REPLACE VIEW clean.barriers AS
IF EXISTS (SELECT * FROM sys.tables t
JOIN sys.schemas s ON (t.schema_id = s.schema_id)
WHERE s.name = 'clean' AND t.name = 'barriers')
EXEC('DROP EXTERNAL TABLE [clean].[barriers]')
CREATE EXTERNAL TABLE [clean].[barriers]
WITH
(
LOCATION = 'clean/synapse/barriers',
DATA_SOURCE = "",
FILE_FORMAT = [SynapseParquetFormat]
)
AS
SELECT * FROM OPENROWSET (
BULK
'/filepath.csv#snapshot=*', FORMAT = 'CSV'
)
WITH(
-- Schema adjusted to what we have in clean/barriers in Bigquery
mshp_id INT,
prog_name NVARCHAR(256),
barrier_name NVARCHAR(256),
days INT
) AS load_clean_data
As per the Official Documentation, you should have a Data source for the source file also from which you are trying to copy the data.
So, try to create a data source for the source CSV file and check, it may work.
Also, as you are executing the above script using ADF, first try to execute it without ADF and if the error occurs then problem can be with the script not ADF. If not try to change the activity of ADF and check.
You can try this trouble shoot also in your BULK path. As you want the data from that csv files folder give the path like below and check.
/folder/*.csv
We have a scenario where the source csv files are isolated by Customer i.e., each Customer will have a Container in the Azure Storage.
When creating External Table in SQL Synapse, is it possible to pass the Container name as parameter that way there are not multiple External Data Tables in SQL Synapse DB?
CREATE EXTERNAL DATA SOURCE AzureBlobStorage with (
TYPE = HADOOP,
LOCATION ='wasbs://<**container100**>#<accountname>.blob.core.windows.net',
CREDENTIAL = AzureStorageCredential
);
CREATE EXTERNAL TABLE [dbo].[res1_Data] (
[ID] INT,
[UniqueId] VARCHAR(50),
[Status] VARCHAR(50) NULL,
[JoinedDate] DATE
)
WITH (LOCATION='<**container2**>/<folder>/<file>.csv',
DATA_SOURCE = AzureBlobStorage,
FILE_FORMAT = CEFormat
);
Unfortunately you can't use variables within DDL commands. However, you can build dynamic statements and then execute with sp_executesql to do this.
More information here.
Is there any option to query the Azure SQL Data warehouse (Cloud) from the Azure SQL Server database (Cloud)?
We have a central warehouse hosted in cloud with all the domains needed for the Application. It would be great if we can use those tables as external tables using the approach described at https://azure.microsoft.com/en-us/blog/querying-remote-databases-in-azure-sql-db.
Querying Data Warehouse from SQL DB is currently unsupported.
You can access Azure SQL Data Warehouse from Azure SQL using external tables
Here's the article: https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-table-transact-sql
Example:
CREATE MASTER KEY
CREATE DATABASE SCOPED CREDENTIAL dwcredential WITH IDENTITY = '<username>',
SECRET = '<password>';
CREATE EXTERNAL DATA SOURCE sqldw WITH
(
TYPE = RDBMS,
LOCATION = N'<dwserver>.database.windows.net',
DATABASE_NAME = N'<dwdatabasename>',
CREDENTIAL = dwcredential
)
GO
CREATE EXTERNAL TABLE [<azuresqltablename>](
[col1] [int] NOT NULL,
[col2] [int] NOT NULL
)
WITH
(
DATA_SOURCE = sqldw,
SCHEMA_NAME = 'dbo', --schema name of remote table
OBJECT_NAME = '<dwtablename>' --table name of remote table
);
I have created a table in Azure SQL Data Warehouse as below:
CREATE TABLE dbo.test_lob_type
(
id VARCHAR(80) NOT NULL,
mime_type VARCHAR(80) NOT NULL,
binary_lob VARBINARY(MAX) NULL
)
WITH
(
DISTRIBUTION = HASH ( id ),
CLUSTERED INDEX ( id ASC )
);
I want to insert a BLOB object into this table. I tried to achieve this using the OPENROWSET command as pointed in the link How to insert a blob into a database using sql server management studio
But unfortunately this command does not work with Azure SQL DW. Can anyone provide any input on how to insert any BLOB object into a SQL DW table from the command line?
bcp is supported for this scenario. Here is a simple example using SQL Authentication and char format:
REM Example using SQL Authentication and character file
bcp dbo.test_lob_type in test_lob_type.bcp -S yourDWServer.database.windows.net -d yourDWName -U yourLogin -P yourPassword -c
If your file only contains the blob, consider loading to a staging table before inserting into the main table.
I am a newbie to ssis and am having issues with the loading the results of a stored procedure that includes NULL values into a table with ssis. What I have is follows:
Step 1
Execute Stored Procedure on Database1 will return a Full Result Set and put into ADO object User::CallResults
Step 2
Then I Loop through the results of User::CallResults mapping 2 variables:
Variable Index ADO object Type Nullable
User::ID 0 Object NO
User::Result 1 Object Yes
Step 3
Then in the Insert Row Into Database2 takes each row and executes "insert into dbo.myTable id, result values (?,?)"
I map ID as int and Result as long respectively.
When I execute I get the error:
failed with the following error: "An error occurred while extracting the
result into a variable of type (DBTYPE_I4)". Possible failure reasons:
Problems with the query, "ResultSet" property not set correctly,
parameters not set correctly, or connection not established correctly.
Seems like this it errors when there is a null in the Result. Any suggestions to make ssis allow nulls?
You can achieve the process that you have described in the question using Data Flow task. Here is a step by step description of fetching data using stored procedure and then inserting into a table, all done inside Data Flow task. This example is just to give an idea of how this can be done and it uses only a single instance of SQL Server.
Step-by-step process:
Create two tables named dbo.Source and dbo.Destination and populate the table dbo.Source with data as shown in screenshot #1. Create table scripts are provided under Scripts section.
Create a stored procedure named dbo.GetData using the script provided under Scripts section.
On the SSIS package, create a variable named StoredProcedure as shown in screenshot #2. This variable will contain the stored procedure execution statement.
Create an OLE DB Connection in the Connection manager to connect to the SQL Server instance.
On the Control Flow tab of the SSIS package, place a Data Flow task as shown in screenshot #3.
Double-click on the Data flow task to navigate to the Data Flow tab. Inside the Data Flow tab, place an OLE DB Source and an OLE DB Destination as shown in screenshot #4.
Configure the OLE DB Source as shown in screenshots #5 and #6. Notice that the source is using the variable that was created in step #3. The data returned by the stored procedure will be the source input.
Configure the 'OLE DB Destination` as shown in screenshots #7 and #8. This will insert the data into the destination table.
Screenshot #9 displays sample package execution.
Screenshot #10 shows the data in the tables after the package execution. Note that the destination table contains NULL values. This is possible because the column Qty can accept NULL values. However, if we had passed NULL values to the ItemNumber column, the package would have failed because the column is non-nullable.
Hope that helps.
Scripts:
.
CREATE TABLE [dbo].[Destination](
[Id] [int] IDENTITY(1,1) NOT NULL,
[ItemNumber] [varchar](50) NOT NULL,
[Qty] [int] NULL,
CONSTRAINT [PK_Destination] PRIMARY KEY CLUSTERED ([Id] ASC)) ON [PRIMARY]
GO
CREATE TABLE [dbo].[Source](
[Id] [int] IDENTITY(1,1) NOT NULL,
[ItemNumber] [varchar](50) NOT NULL,
[Qty] [int] NULL,
CONSTRAINT [PK_Source] PRIMARY KEY CLUSTERED ([Id] ASC)) ON [PRIMARY]
GO
CREATE PROCEDURE [dbo].[GetData]
AS
BEGIN
SET NOCOUNT ON;
SELECT Id
, ItemNumber
, Qty
FROM dbo.Source
END
GO
Screenshot #1:
Screenshot #2:
Screenshot #3:
Screenshot #4:
Screenshot #5:
Screenshot #6:
Screenshot #7:
Screenshot #8:
Screenshot #9:
Screenshot #10:
Not really an answer, but since code can't be formatted nicely in a comment I'm putting it here.
Do you realize, that if your databases are on the same server, you can do this:
INSERT INTO
database1.dbo.Results
EXEC
database2.dbo.SampleStoredProcedure #param1, #param2, #param3
Put this logic into a Data Flow. After that is accomplished, just do a bulk insert and it should allow the nulls through.
SSIS - Unable to insert NULL for Blank fields in BULK INSERT