Migrate ADF - Datasets which are linked with Linked Services and Pipelines to Synapse Analytics - azure-data-factory-2

We Need to migrate Datasets from ADF which are linked with Linked Services and Pipelines only to Synapse Analytics.
The GITHUB solution (from previous posts https://learn.microsoft.com/en-us/answers/questions/533505/import-bulk-pipelines-from-azure-data-factory-to-a.html)
migrates entire all datasets, pipelines, linked services from ADF to Synapse Analytics.
But we need to migrate Datasets, linked services and pipelines which are linked each other and don't need to migrate which were not linked.

Unfortunately, there is no direct way to exclude the unwanted objects from Azure Data Factory when migrating to other service (Synapse Analytics in your case).
As a workaround, you can make a copy of the existing factory, remove the objects you do not wish to migrate, and use that new factory as your source.
Please follow the below steps to copy the existing data factory objects to new data factory.
Go to your existing ADF Workspace. Follow the path: Manage -> ARM templates -> Export ARM template.
Extract the downloaded file. Open the arm_template.json file in Notepad++ or any other editor. On line number 8, for parameter defaultValue, give the name of your new data factory where you will copy the objects.
Create a new Azure Data Factory with the same name which you have provided in the above step.
Go the Workspace of this newly created data factory. Follow the path: Manage -> ARM template -> Import ARM template. This will open a separate Custom deployment tab.
Select Build your own template in the editor option.
Delete the existing content on whitespace. Click on Load file option to upload the arm_template.json file which you downloaded and edited earlier. Click on Save.
In the final step, you need to give Subscription, Resource Group, Region and Name of newly created data factory where all your objects will be copied. Along with that, you need to provide the connection string of all the linked services which will be copied in the new factory. See the below image for reference. Once done, click on Review and Create and this will copy all your objects to new Data Factory.
Now, in your new factory, you can delete all the objects which you don't want to migrate. Once done, follow the same GitHub link which is mentioned in Microsoft Q&A answer to migrate the objects to Synapse Analytics.
Note: You can later delete the resources (data factory which is used for migration).

Related

Load multiple files using Azure Data factory or Synapse

I am moving from SSIS to Azure.
we have 100's of files and MSSQL tables that we want to push into a Gen2 data lake
using 3 zones then SQL Data Lake
Zones being Raw, Staging & Presentation (Change names as you wish)
What is the best process to automate this as much as possible
for example build a table with files / folders / tables to bring into Raw zone
then have Synapse bring these objects either full or incremental load
then process the them into the next 2 zones I guess more custom code as we progress.
Your requirement can be accomplished using multiple activities in Azure Data Factory.
To migrate SSIS packages, you need to use SSIS Integrated Runtime (IR). ADF supports SSIS Integration which can be configured by creating a new SSIS Integration runtime. To create the same, click on the Configure SSIS Integration, provide the basic details and create a new runtime.
Refer below image to create new SSIS IR.
Refer this third-party tutorial by SQLShack to Move local SSIS packages to Azure Data Factory.
Now, to copy the data to different zones using copy activity. You can make as much copy of your data as your requirement using copy activity. Refer Copy data between Azure data stores using Azure Data Factory.
ADF also supports Incrementally load data using Change Data Capture (CDC).
Note: Both Azure SQL MI and SQL Server support the Change Data Capture technology.
Tumbling window trigger and CDC window parameters need to be configured to make the incremental load automated. Check this official tutorial.
The last part:
then process them into the next 2 zones
This you need to manage programmatically as there is no such feature available in ADF which can update the other copies of the data based on CDC. You need to either create a separate CDC for those zones or do it logically.

how to use Azure Synapse database templates programmatically

I can create an Azure data lake database with pre-built tables using Azure Synapse database templates from the Synapse Studio UI, but is there a way to use these templates programmatically? So far from my research I have not found a command, API, or SDK for this. Perhaps I could create the database and tables via the UI, then generate the associated spark sql creation scripts, but don't see a way how to do that either. Does anyone have any ideas on how to do either of the prior?
You can create the data lake storage, tables and data insertion programmatically using Azure SDKs. But these templates have been made available to overcome these series of manual tasks. Using these templates save your time and efforts to create an environment and sample data for your development.
Therefore, asking to deploy these templates programmatically challenging the complete concept of templates. If you want to deploy these resources manually, you can use Azure SDKs.

Is sharing dataset in the Bigquery is migration?

We need to migrate the data from the old GCP instance to new instance( with new organization node). I am using the "share dataset" option to move the data. It is very convenient approach. Do you think this is a good way to migrate data or should we create new tables and then load the data into the tables?
Thanks in advance!
It's depend on what you want to achieve. The share dataset feature allow other to access the data because you have granted the permission.
However, the data doesn't move and still belong to the old GCP project. If you remove the project, you remove the data. In addition, it's still the old project that pay for the data storage, the new one only for the data processing.
If you plan to shut down the old project, you have to copy the data. Automatically with the data transfert service, or by querying them if you want to filter/transform the existing data before storing them in the new project.

Excel into Azure Data Factory into SQL

I read a few threads on this but noticed most are outdated, with excel becoming an integration in 2020.
I have a few excel files stored in Drobox, I would like to automate the extraction of that data into azure data factory, perform some ETL functions with data coming from other sources, and finally push the final, complete table to Azure SQL.
I would like to ask what is the most efficient way of doing so?
Would it be on the basis of automating a logic app to extract the xlsx files into Azure Blob, use data factory for ETL, join with other SQL tables, and finally push the final table to Azure SQL?
Appreciate it!
Before using Logic app to extract excel file Know Issues and Limitations with respect to excel connectors.
If you are importing large files using logic app depending on size of files you are importing consider this thread once - logic apps vs azure functions for large files
Just to summarize approach, I have mentioned below steps:
Step1: Use Azure Logic app to upload excel files from Dropbox to blob storage
Step2: Create data factory pipeline with copy data activity
Step3: Use blob storage service as a source dataset.
Step4: Create SQL database with required schema.
Step5: Do schema mapping
Step6: Finally Use SQL database table as sink

Exporting data sources between environments in pentaho

I'm new with Pentaho and I'm trying to set up an automatic deployment process for the pentaho business analytics platform repository, but I'm having troubles to find out how to proceed with the data sources.
I would like to do export/import all the data sources, the same that here is explained with the repository (Reporting, Analyzer, Dashboards, Solution Files...) but with the data connections, mondrian files, schemas....
I know there's way to backup and restore the entire repository (explained here), but that's not the way I want to proceed, since the entire repository could contain undesired changes for production.
This would need to be with command line or rest system or some other thing that be triggered by Jenkins.
Did you try import-export with the -ds(DataSource) qualifier ? This will include the data connection, mondrian schema and metadata models.
Otherwise, you can import everything, unzip, filter according a certain logic (to be defined by the guy in charge of the deployment), zip again and export it to prod. A half day project with the Pentaho Data Integrator.