What the extent of dbt cloud availability on Azure?
Don't find it in the maretplace on Azure but on AWS
Can i run DBT independent of ADF on Azure?
Am we limited to running dbt only in command line for Azure
Can we leverage the scheduling capabilities of dbt cloud on Azure
Related
I am working on Azure Synapse. I am able to build the Azure Synapse project successfully using the Azure CI pipeline's MS Build task.
But as I am trying to deploy Azure Synapse using the Azure CD pipeline, I am getting the following error.
Internal Error. The database platform service with type Microsoft.Data.Tools.Schema.Sql.SqlDwDatabaseSchemaProvider is not valid. You must make sure the service is loaded, or you must provide the full type name of a valid database platform service.
I am deploying the DacPac using the following task. I hope this should not be any concern.
https://github.com/DrJohnT/AzureDevOpsExtensionsForSqlServer/tree/master/extensions/PublishDacPac
This is a weird error because a couple of days ago same deployment was done successfully.
Please help!
I have resolved the issue. I have deeply investigated and found that this issue is related to Agent which I am using in the Azure DevOps pipeline.
One more thing, I am deploying SQL Database and Azure Synapse using different pipelines but using the same agent.
Below screenshot of the Azure SQL database pipeline agent. SQL Database is deploying successfully with agent specification vs2017-win2016
Below screenshot of the Azure Synapse deployment pipeline agent. Azure Synapse is deploying successfully with agent specification windows-2019
The gist is that one agent is being used but the specification is different for the Azure Synapse and SQL database while both are building on VS2019.
What is the best method to sync medical images between my client PCs and my Azure Blob storage through a cloud-based web application? I tried to use MS Azure Blob SDK v18, but it is not that fast. I'm looking for something like dropbox, fast, resumable and efficient parallel uploading.
Solution 1:
AzCopy is a command-line tool for copying data to or from Azure Blob storage, Azure Files, and Azure Table storage, by using simple commands. The commands are designed for optimal performance. Using AzCopy, you can either copy data between a file system and a storage account, or between storage accounts. AzCopy may be used to copy data from local (on-premises) data to a storage account.
And also You can create a scheduled task or cron job that runs an AzCopy command script. The script identifies and uploads new on-premises data to cloud storage at a specific time interval.
Fore more details refer this document
Solution 2:
Azure Data Factory is a fully managed, cloud-based, data-integration ETL service that automates the movement and transformation of data.
By using Azure Data Factory, you can create data-driven workflows to move data between on-premises and cloud data stores. And you can process and transform data with Data Flows. ADF also supports external compute engines for hand-coded transformations by using compute services such as Azure HDInsight, Azure Databricks, and the SQL Server Integration Services (SSIS) integration runtime.
Create an Azure Data Factory pipeline to transfer files between an on-premises machine and Azure Blob Storage.
For more details refer this thread
Why does external I.P. need access to on-prem sql database when copying data with ADF to Azure SQL?
It looks like on-prem sql makes a direct connection to Azure SQL (bypassing ADF). Is this by design or do I follow the wrong workflow?
Data Factory use the integration runtime to help us create the connection to the Source/Sink dataset. Azure integration runtime for cloud dataset and Self-host integration runtime for on-premise source/sink dataset.
The integration runtime (IR) is the compute infrastructure that Azure
Data Factory uses to provide data-integration capabilities across
different network environments. For details about IR, see Integration
runtime overview.
A self-hosted integration runtime can run copy activities between a
cloud data store and a data store in a private network. It also can
dispatch transform activities against compute resources in an
on-premises network or an Azure virtual network. The installation of
a self-hosted integration runtime needs an on-premises machine or a
virtual machine inside a private network.
Azure integration runtime is provides by ADF in default. The self-host integration runtime must be created manually.
That means Data Factory can not access the on-prem SQL database directly. It need the self-host integration runtime to help us connect to the on-prem SQL database.
It means that the on-prem sql does not make a direct connection to Azure SQL(bypassing ADF. That why external I.P. need access to on-prem sql database when copying data with ADF to Azure SQL.
HTH.
I have hard time with Azure Functions on Azure Government. I need to create a C# trigger bases process on Azure Storage. The goal is to automate the process of the loading the files into Azure SQL DB when a file is dropped into Azure Storage.
Since Azure Functions in Azure Government are not fully comparable to Azure Function in regular Azure and not all UIs are the same, I can't deploy the function to trigger on a storage file.
I was able to build the process in regular Azure Cloud following instructions from https://github.com/yorek/AzureFunctionUploadToSQL but since Azure Government is missing the UI for Azure Functions I'm having hard time to replicating the process in Azure Government.
Portal UI support is not yet available in Azure Government, but it is coming soon. Additionally, Azure Government currently supports "App Service plan" ("Consumption plan" coming soon).
In the meantime, you can do everything you need. First, provision your Azure Function in Azure Gov via the Azure CLI by following this Quickstart example for Functions on Azure Gov. That same link also shows you how you can use Visual Studio to set up your triggers (in your case, a Blob trigger).
Once complete, deploy your Function to Azure Gov with Visual Studio.
we can run U-SQL Query using Azure data factory.
can we run U-SQL query using oozie workflow in HDINSIGHT Spark cluster.
(Azure Data Lake team here)
Currently the U-SQL language is only supported running in a Data Lake Analytics account. HDInsight doesn't support running U-SQL in a cluster.