we can run U-SQL Query using Azure data factory.
can we run U-SQL query using oozie workflow in HDINSIGHT Spark cluster.
(Azure Data Lake team here)
Currently the U-SQL language is only supported running in a Data Lake Analytics account. HDInsight doesn't support running U-SQL in a cluster.
Related
What the extent of dbt cloud availability on Azure?
Don't find it in the maretplace on Azure but on AWS
Can i run DBT independent of ADF on Azure?
Am we limited to running dbt only in command line for Azure
Can we leverage the scheduling capabilities of dbt cloud on Azure
I am working on Azure Synapse. I am able to build the Azure Synapse project successfully using the Azure CI pipeline's MS Build task.
But as I am trying to deploy Azure Synapse using the Azure CD pipeline, I am getting the following error.
Internal Error. The database platform service with type Microsoft.Data.Tools.Schema.Sql.SqlDwDatabaseSchemaProvider is not valid. You must make sure the service is loaded, or you must provide the full type name of a valid database platform service.
I am deploying the DacPac using the following task. I hope this should not be any concern.
https://github.com/DrJohnT/AzureDevOpsExtensionsForSqlServer/tree/master/extensions/PublishDacPac
This is a weird error because a couple of days ago same deployment was done successfully.
Please help!
I have resolved the issue. I have deeply investigated and found that this issue is related to Agent which I am using in the Azure DevOps pipeline.
One more thing, I am deploying SQL Database and Azure Synapse using different pipelines but using the same agent.
Below screenshot of the Azure SQL database pipeline agent. SQL Database is deploying successfully with agent specification vs2017-win2016
Below screenshot of the Azure Synapse deployment pipeline agent. Azure Synapse is deploying successfully with agent specification windows-2019
The gist is that one agent is being used but the specification is different for the Azure Synapse and SQL database while both are building on VS2019.
I have a MySQL DB on AWS.
I want to run a few simple SQL statements that select data from MySQL and insert to Azure DB.
Something like
select *
into Azure_Table
from
MySQL_Table
I also want to schedule this on a daily basis.
How can I do it directly from Azure SQL without having to use Data Factory / SSIS
Thank you
You can use Data Ingestion in ADF.
You can select the source and sink. Then schedule as per your need.
Note: Since you have the Source as MySQL on AWS i.e. outside of Azure Cloud, you would have to setup Self-hosted integration runtime for the linked service at source. Follow official MS doc for Setting up a self-hosted integration runtime using UI.
You can Migrate Amazon RDS for MySQL to Azure Database for MySQL using MySQL Workbench.
You can refer to below official documentation where you can get step by step explanation:
Migrate Amazon for MySQL to Azure Database for MySQL using MySQL Workbench.
Workaround – There is no direct way to query third-party database from Azure. But, you can migrate it to Azure and then perform operations.
What is the best method to sync medical images between my client PCs and my Azure Blob storage through a cloud-based web application? I tried to use MS Azure Blob SDK v18, but it is not that fast. I'm looking for something like dropbox, fast, resumable and efficient parallel uploading.
Solution 1:
AzCopy is a command-line tool for copying data to or from Azure Blob storage, Azure Files, and Azure Table storage, by using simple commands. The commands are designed for optimal performance. Using AzCopy, you can either copy data between a file system and a storage account, or between storage accounts. AzCopy may be used to copy data from local (on-premises) data to a storage account.
And also You can create a scheduled task or cron job that runs an AzCopy command script. The script identifies and uploads new on-premises data to cloud storage at a specific time interval.
Fore more details refer this document
Solution 2:
Azure Data Factory is a fully managed, cloud-based, data-integration ETL service that automates the movement and transformation of data.
By using Azure Data Factory, you can create data-driven workflows to move data between on-premises and cloud data stores. And you can process and transform data with Data Flows. ADF also supports external compute engines for hand-coded transformations by using compute services such as Azure HDInsight, Azure Databricks, and the SQL Server Integration Services (SSIS) integration runtime.
Create an Azure Data Factory pipeline to transfer files between an on-premises machine and Azure Blob Storage.
For more details refer this thread
Since I've starting using Azure Synapse Analytics, I created a Spark Pool Cluster, then on the Spark Pool cluster I created databases and tables using Pyspark on top of parquet files in Azure Data Lake Store Gen2.
I use to be able to access my spark Database/ parquet tables through SSMS using the Serverless SQL endpoint but now I can no longer see my spark Databases through the Severless SQL Endpoint in SSMS. My spark databases are still accessible through Azure Data Studio but not through SSMS. Nothing has been deployed or alter on my side. Can you help resolve the issue? I would like to be able to access my spark databases through SSMS.
Sql Serverless Endpoint
Azure Synapse Database
If your Spark DB is built on top of Parquet files, as you said, databases should sync to external tables in Serverless SQL pool just fine and you should be able to see synced SQL external tables in SSMS as well. Check this link for more info about metadata synchronization.
If everything mentioned above is checked, then I'd suggest you to navigate to Help + Support in Azure Portal and fill in a support ticket request with details of your problem so engineering team can take a look and see whether there is some issue with your workspace or not.