Since I've starting using Azure Synapse Analytics, I created a Spark Pool Cluster, then on the Spark Pool cluster I created databases and tables using Pyspark on top of parquet files in Azure Data Lake Store Gen2.
I use to be able to access my spark Database/ parquet tables through SSMS using the Serverless SQL endpoint but now I can no longer see my spark Databases through the Severless SQL Endpoint in SSMS. My spark databases are still accessible through Azure Data Studio but not through SSMS. Nothing has been deployed or alter on my side. Can you help resolve the issue? I would like to be able to access my spark databases through SSMS.
Sql Serverless Endpoint
Azure Synapse Database
If your Spark DB is built on top of Parquet files, as you said, databases should sync to external tables in Serverless SQL pool just fine and you should be able to see synced SQL external tables in SSMS as well. Check this link for more info about metadata synchronization.
If everything mentioned above is checked, then I'd suggest you to navigate to Help + Support in Azure Portal and fill in a support ticket request with details of your problem so engineering team can take a look and see whether there is some issue with your workspace or not.
Related
Is there a possibility to connect to the Azure Synapse Spark pool from "outside". Meaning with Rstudio or Sparksql?
Unfortunately, R support isn't available for Synapse yet. So connecting it to Rstudio isn't useful.
Spark pools in Azure Synapse include the following components that are
available on the pools by default. Spark Core. Includes Spark Core,
Spark SQL, GraphX, and MLlib.
As per the above statement from this official document, Spark SQL is already by default available in Azure Synapse. There is no such need to connect to outside.
Apart from all this, you can consider #wBob's inputs shared in comment section based on your requirement.
I have a MySQL DB on AWS.
I want to run a few simple SQL statements that select data from MySQL and insert to Azure DB.
Something like
select *
into Azure_Table
from
MySQL_Table
I also want to schedule this on a daily basis.
How can I do it directly from Azure SQL without having to use Data Factory / SSIS
Thank you
You can use Data Ingestion in ADF.
You can select the source and sink. Then schedule as per your need.
Note: Since you have the Source as MySQL on AWS i.e. outside of Azure Cloud, you would have to setup Self-hosted integration runtime for the linked service at source. Follow official MS doc for Setting up a self-hosted integration runtime using UI.
You can Migrate Amazon RDS for MySQL to Azure Database for MySQL using MySQL Workbench.
You can refer to below official documentation where you can get step by step explanation:
Migrate Amazon for MySQL to Azure Database for MySQL using MySQL Workbench.
Workaround – There is no direct way to query third-party database from Azure. But, you can migrate it to Azure and then perform operations.
I am receiving multiple .trn files on daily basis & I am resorting those file in on-premises sql database. Now How we can migrate those .trn files daily on azure.
Connect on-premises SQL database to Azure data factory using Self hosted IR. A self-hosted integration runtime can run copy activities between a cloud data store and a data store in a private network.
Create pipeline and use copy activity.
Select on-premises SQL database as Source.
Select Azure service of your choice as Sink.
Now use trigger to copy data from on premise to Azure periodically.
Refer – Copy activity link
Also refer – Trigger link
We are supporting a legacy system for our organisation. In the current scenario, we receive a SQL Server backup (.bak files) from the application vendor on an FTP location. For every weekend on Sunday it is a Full backup and for every other day its the differential one.
On our side, we have a SQL server instance running which has custom stored procedures written and scheduled to check the location every morning and then restore the backups every day. These restored backups are then used by the organisation for internal reporting purposes. There are 100s of other stored procedures written for different reports in different DBs on the same instance.
Since SQL Server 2008 is now out of support and for cost-saving purposes of running on-premise system, my team has been given a task to look into migrating this whole system to Azure SQL database.
My question is what is the most effective way in which we can move this workflow to the cloud? I have an azure trial account set up for me to try but haven't been successful in restoring the .bak files on Azure SQL instance.
Thanks.
You essentially have two options for Azure, either perform a fairly linear Lift and Shift to SQL Server on an Azure VM or go with a more advanced Azure PaaS offering in Azure SQL Database Managed Instance. The specific deployment Azure SQL Database (Single Instance) will not support your current solution requires with regard to the .bak file support, and I have detailed that below. For further details between the difference between Azure SQL Database Single Instance versus Managed Instance, please see: Features comparison: Azure SQL Database and Azure SQL Managed Instance
The second option, is to leverage the Azure Enterprise Ready Analytics Architecture (AERAA) (link) of Azure (PaaS) Analytics services. With Azure SQL Database (PaaS) services, as opposed to on-premise SQL Server or SQL Server on an Azure VM, there is no Integration Runtime or Analysis Services as a bundled service component. These services are separate PaaS offerings and with the help of the linked AERAA blog, you can gain a better understanding of the Azure Analytics services.
The .bak versus .bacpac support dilemma:
Since the main requirement for your solution is support of .bak files, you need to understand where .bak and where .bacpac files are supported. The term Azure SQL Database applies to both a specific deployment option for an Azure SQL database (PaaS) service and as a general term for Azure SQL cloud databases. As for the specific deployment option, Azure SQL Database (Single Instance nor Elastic Pools) will support your scenario with .bak files. This deployment option will support export/import functionality via .bacpac file format. It will not support full/partial restore functionality. The backup/restore functionality although configurable, is only in scope for the specific database hosted by an Azure SQL (logical) Server instance. Basically, you can not restore an external file. You can import, which is always a full copy. So, for that reason, for an Azure PaaS database service you will need Azure SQL Database Managed Instance for .bak file support or deploy an SQL Server VM image to an Azure VM, and migrate your objects via Azure Database Migration Service.
Regards,
Mike
How to load my SQL database created in MySQL Workbench on Azure cloud?
I created a database which consists of some tables - for now, there is now data in them, it's just a small script created by MySQL Workbench. I also created a database on Azure cloud, created login & password and when I want to use 'automated export' option (I have Storage account, I enter valid login with password) I have error:
'Could not find any bacpac files in the specified storage account.'
I tried google this phrase but I completely do not understand the idea behind these bacpac files and I do not know what to do with it. Can anyone describe me step-by-step how to put my database on Azure cloud?
I want to connect to this DB on Azure in the future because I would like to do a webapplication and android app which will use a remote DB available online.
Azure SQL Database is a custom SQL Server, so if you want to use MySQL you should create a Clear DB (which is a Microsoft partner that offers MySQL on azure). Other option, you can create a Virtual Machine and install by yourself a MySQL.
After that, you can import your tables / records.