I am working on Azure Synapse. I am able to build the Azure Synapse project successfully using the Azure CI pipeline's MS Build task.
But as I am trying to deploy Azure Synapse using the Azure CD pipeline, I am getting the following error.
Internal Error. The database platform service with type Microsoft.Data.Tools.Schema.Sql.SqlDwDatabaseSchemaProvider is not valid. You must make sure the service is loaded, or you must provide the full type name of a valid database platform service.
I am deploying the DacPac using the following task. I hope this should not be any concern.
https://github.com/DrJohnT/AzureDevOpsExtensionsForSqlServer/tree/master/extensions/PublishDacPac
This is a weird error because a couple of days ago same deployment was done successfully.
Please help!
I have resolved the issue. I have deeply investigated and found that this issue is related to Agent which I am using in the Azure DevOps pipeline.
One more thing, I am deploying SQL Database and Azure Synapse using different pipelines but using the same agent.
Below screenshot of the Azure SQL database pipeline agent. SQL Database is deploying successfully with agent specification vs2017-win2016
Below screenshot of the Azure Synapse deployment pipeline agent. Azure Synapse is deploying successfully with agent specification windows-2019
The gist is that one agent is being used but the specification is different for the Azure Synapse and SQL database while both are building on VS2019.
Related
We're trying to set-up Azure Devops Build and Release Pipelines for our Power BI Paginated Reports. We need the release pipeline to support deployment to our Test and two Prod environments.
We able to retrieve RDLs from our Git Repo and publish to the target workspaces. Then we need to modify the connection strings for each target environment. Using Azure PowerShell, we're able to authenticate as a Power BI Service Principle and change the connection string Server and Database names.
In order for this to work, we need to take over each report:
"groups/$workspaceId/reports/$reportId/Default.TakeOver"
and then
"groups/$workspaceId/reports/$reportId/Default.UpdateDatasources".
The updated connection can be verified through
"groups/$workspaceId/reports/$reportId/Datasources".
However when we try to run the report it fails with a "Could not connect to datasource" error. After going to the report management page and choosing "Take over" the report works fine.
Has anyone run into this before?
TIA
Requirement: I wanted to copy data from a specific table/view residing on a on-premise SQL Server to Azure SQL DB.
Infrastructure: As depicted in below picture. Essentially, the Azure network is directly connected with corporate network over Express Route. Thus it's a pure private network connection; as good as the corporate network itself.
Issue/Question: I know there are multiple approaches present to get this operation done and I am not restricted to use ADF copy Data tool only. BUT, for all of these I see some cavets or extra steps needed to be done as below:
ADF Copy Data Tool: Needs a SH-IR and a small MSI package needs to be installed on on-premise machine which hosts the SQL server for registration purpose.
Logic Apps: Needs a Virtual Gateway (OR) ASE
App Service: If the operation is wrapped in a C# application and I choose to deploy to a Azure Web Apps. Then in-order to connect to on-premise SQL Server we need to setup hybrid connection manager and as in #1 we need to install something in on-premise machine.
For my case, none of these extra steps can be done. essentially, the on-premise SQL Server comes under a different BU and thus I don't have any permission there; except they have given grant to a table/view. Thus, none of these extra shitty steps can be done.
Moreover, as mentioned above; since it's connected over express route as direct connection, As can be seen in above picture, both the on-premise and azure SQL are essentially inside the same corporate network. THUS, I should be able to access them directly without configuring any of these extra steps as mentioned above.
Please confirm on these and provide a suggestion.
Thank You.
You can still go with the ADF scenario without a SHIR by creating ADF in a Managed VNET using Private Endpoint. As you already have an ER circuit and have the flexibility to configure the Azure side, can you do this with Azure IR: Access on-premises SQL Server from Data Factory Managed VNet using Private Endpoint - Azure Data Factory | Microsoft Docs
There are 2 solutions which could work for your scenario but even for them to work ,you would need access to on prem SQL server machine access to some extent atleast for one time config and Azure SQL db should be accessible via SSMS installed on on-prem machine.
Using linked server
You can create a linked server ( process explained here https://www.sqlshack.com/create-linked-server-azure-sql-database/ ) on on-prem server and create a agent server job to insert data to azure SQL db table.
Via Python Script
This would need Python installation on on-prem machine. Once installed you can write script to transfer data between on-prem SQL server and Azure SQL db. You can schedule this script again by using an agent server job.
I have a Microsoft Azure SQL Database project. I also have a Python3.9 project that uses unittest to unit test this database project. I have an Azure DevOps build pipeline defined in YAML that runs the unit test against the development-integration environment.
I do not want to publish changes to the development-integration environment before running the tests. If you think this is the wrong approach, I will consider your arguments.
I want to 'virtually' test the changes. I want to deploy the new objects to a temporary ad-hoc database instance. It must be equivalent to Azure Database Instance. When the tests have been executed I want to clear everything away. I do not want to deploy a database in Azure for this purpose due to billing, although if I were to use a serverless instance this would not be a problem.
Any ideas?
If you are on cloud and you need to test you need to test that on the cloud too.
You cannot "virtually" test, there is nothing equivalent to Azure SQL database on-prem.
Go with the serverless instance as you said.
Can nkdagility be used to convert Microsoft Test Manager Test to Azure DevOps? If not what is the suggested path for conversion?
Yes: the Azure DevOps Migration Tools support this with many caviats.
I have hard time with Azure Functions on Azure Government. I need to create a C# trigger bases process on Azure Storage. The goal is to automate the process of the loading the files into Azure SQL DB when a file is dropped into Azure Storage.
Since Azure Functions in Azure Government are not fully comparable to Azure Function in regular Azure and not all UIs are the same, I can't deploy the function to trigger on a storage file.
I was able to build the process in regular Azure Cloud following instructions from https://github.com/yorek/AzureFunctionUploadToSQL but since Azure Government is missing the UI for Azure Functions I'm having hard time to replicating the process in Azure Government.
Portal UI support is not yet available in Azure Government, but it is coming soon. Additionally, Azure Government currently supports "App Service plan" ("Consumption plan" coming soon).
In the meantime, you can do everything you need. First, provision your Azure Function in Azure Gov via the Azure CLI by following this Quickstart example for Functions on Azure Gov. That same link also shows you how you can use Visual Studio to set up your triggers (in your case, a Blob trigger).
Once complete, deploy your Function to Azure Gov with Visual Studio.