I'm trying to find documentation on this, but I don't see a definitive answer. Is it possible to have a Synapse Notebook in Azure to connect directly to an On-Prem SQL Server to pull data into a dataframe using JDBC connections maybe? Perhaps some other method?
I spoke with a guy from Microsoft who works on the Synapse team, and he confirmed that as of right now, it's not possible to directly connect to your on-prem data sources directly through Synapse notebooks.
He suggested that the best way right now, is to bring your data into a Data Lake for example using pipelines, and then do your Synapse processing.
Related
I have created a new Azure Lake Database using the following procedure
The Lake Database name is called TestLakeDB.
However, when I check the list of databases available in Use database TestLakeDB doesn't appear.
Any thoughts?
Thanks for the valuable discussion. Posting your conversation as answer to help other community members who faces similar issues.
When we create Lake database after connecting to the github, it won't reflect in the Use Database because it is created in github mode.
To reflect the the Lake Database, create the database in the synapse live mode and connect to the github. Now we can see it reflects our database named Lake_Database1 which is created in synapse live mode in the Use Database.
I try to use Serverless Sql Pool Integration dataset in Azure Analitycs DataFlow as a source but I can't. SQL Pool is unavailable as a Source in DataFlow, but I don't know why?
What is a problem? I use SQL Pool datasets in Azure Synapse Pipelines and it works. Is it problem with my licence, version or maybe I do something wrong?
Have you tried selecting Synapse Dataset instead of SQL
You will need a Synapse Linked Service as well
There is an option to connect a Cloud mySQL instance from BigQuery. I just wanted to know how we can connect a Cloud SQL Server instance to BigQuery.
SQL Server:
There are a bunch of third-party extensions/tools that provide this service. One of them is SSIS Data Flow Source & Destination for Google BigQuery, which is Visual Studio extension that connects SQL Server with Google BigQuery data through SSIS Workflows.:
https://www.cdata.com/drivers/bigquery/ssis/
https://marketplace.visualstudio.com/items?itemName=CDATASOFTWARE.SSISDataFlowSourceDestinationforGoogleBigQuery
In regards to using SQL Server Integration Services to load the data from the on-premises SQL Server to BigQuery, you can take a look for this site. You can also perform ETL from a relational database into BigQuery using Cloud Dataflow, the official documentation details how it can be done, you might need to use Cloud Storage as an intermediate data sink.
Cloud SQL:
BigQuery allows to query data from Cloud SQL by using federated query. The connection must be created within the same project where your Cloud SQL instance is located. If you want to query your data stored in your Cloud SQL instance from BigQuery located in another project, please follow the steps listed below:
Enable the BigQuery API and the BigQuery connection API within your project.
Create a connection to your Cloud SQL instance within the project by following this documentation.
Once you have created the connection, please locate and select it within BigQuery.
Click on the SHARE CONNECTION button and grant permissions to the users that will be use that connection. Please note that the BigQuery Connection User role is the only needed to use a shared connection.
Additionally, please notice that the "Cloud SQL federated queries" feature is in a Beta stage and might change or have limited support (is no available for certain regions, in which case, it is required to use one the supported options mentioned here). Please remember, that to use Cloud SQL Federated queries in BigQuery, the intances need to have a public IP.
If you are limited e.g. by region, one good option might be exporting the data from CloudSQL to Storage as a CSV, and then load it into BigQuery. If you need, it is possible to automate this process using Cloud Composer, refer to this article.
Other approach is to extract information from Cloud SQL (with exports) and import it into BigQuery through load jobs, or streaming inserts.
I hope you find the above pieces of information useful.
It is possible, but be warned the feature is currently Beta
https://cloud.google.com/bigquery/docs/cloud-sql-federated-queries
I want to copy data from sharepoint to microsoft azure sql DW using azure datafactory or alternative service. Can I do this. Please anyone help me with this.
You can do this by setting up a data pipeline using Azure Data Factory to Azure blob storage. Afterwards you can use Azure's fast PolyBase technology to load the data from blob to your SQL Data Warehouse instance.
Can I ask how much data you intend on loading into the DW? Azure Data Warehouse is intended for use with at least terabyte level data up to petabyte compute and storage. I only ask because each SharePoint list or Excel file has a maximum of 2GB per file.
Is there any direct way within the Azure MSSQL ecosystem to export SQL returned data set into the Azure table storage?
Something like BCP but with the Table Storage connection string on the -Output end?
There is a service named Azure Data Factory which can directly copy data from Azure SQL Database to Azure Table Storage, even between other supported data stores, please see the section Supported data stores of the article "Data movement and the Copy Activity: migrating data to the cloud and between cloud stores" to know, but it is for Web, not like BCP command tool.
You can refer to the tutorial Build your first Azure data factory using Azure Portal/Data Factory Editor to know how to use it.
And as references, you can refer to the articles Move data to and from Azure SQL Database using Azure Data Factory & Move data to and from Azure Table using Azure Data Factory to know how it works.