Is there a way to sync NoSQL and SQL databases?
My problem is: We have a software that uses MSSQL. We also have a mobile application that uses MongoDB. We want to sync data (on create/update) between those databases. Mostly from MongoDB to MSSQL.
It is not a problem for us (if we have to) to use different NoSQL DBMs, but we can't find clear instructions of how to sync those two the way i described.
Can anyone help? Thanks.
Have you looked at CDC (Change Data Capture) tools that allow you to capture events from your source database (in this case MSSQL) and deal with the event to update/create/delete data in the NoSQL database.
I invite you to look at https://debezium.io/ and the MSSQL Connector & MongoDB connector:
MS SQL Server Connector
MongoDB Connector
Related
We are wanting to use Azure servers to run our Power Apps applications, however we have local SQL servers which contains our data warehouse we want only certain tables to be on Azure and want to create data feeds between the two with information going from one to the other.
Does anyone have any insight into how I can achieve this?
I have googled but there doesn't appear to be a wealth of information on this topic.
It depends on how fast after a change in your source (the on premise SQL Server) you need that change reflected in your Sink (Azure SQL).
If you have some minutes or even only need to update it every day I would suggest a basic Data Factory Pipeline (search on google for data factory upsert). Here it depends on your data on how you can achieve this.
If you need it faster or it is impossible to extract an incremental update from your source you would need to either use triggers and write the changes from one database to the other or get a program that does change data capture that does that.
It looks like you just want to sync the data in some table between local SQL Server and Azure SQL database.
You can use the Azure SQL Data Sync.
Summary：
SQL Data Sync is a service built on Azure SQL Database that lets you synchronize the data you select bi-directionally across multiple SQL databases and SQL Server instances.
With Data Sync, you can keep data synchronized between your on-premises databases and Azure SQL databases to enable hybrid applications.
A Sync Group has the following properties:
The Sync Schema describes which data is being synchronized.
The Sync Direction can be bi-directional or can flow in only one
direction. That is, the Sync Direction can be Hub to Member, or
Member to Hub, or both.
The Sync Interval describes how often synchronization occurs.
The Conflict Resolution Policy is a group level policy, which can be
Hub wins or Member wins.
Next step, you need to learn how to configure the Data Sync. Please reference this Azure document:Tutorial: Set up SQL Data Sync between Azure SQL Database and SQL Server on-premises.
In this tutorial, you learn how to set up Azure SQL Data Sync by creating a sync group that contains both Azure SQL Database and SQL Server instances. The sync group is custom configured and synchronizes on the schedule you set.
Hope this helps.
The most robust solution here is Transactional Replication. You can also use SSIS or Azure Data Factory for copying tables to/from Azure SQL Database. And Azure SQL Data Sync also exists.
I have a database connected with website, data from website is inserting in that Database, i need to transfer data from that database to another Primary Database (SQL) on another server in real time (minimum latency).
I can not use transactional replication in this case. What are the other alternates to achieve this? Can i integrate DataStreams like Apache kafka etc with SQL server?
Without more detail it's hard to give a full answer. There's what's technically possible, and there's architecturally what actually makes sense :)
Yes you can stream from RDBMS to Kafka, and from Kafka to RDBMS. You can use the Kafka Connect JDBC source and sink. There are also CDC tools (e.g. Attunity, GoldenGate, etc) that support integration with MS SQL and other RDBMS)
BUT…it depends why you want the data in the second database. Do you need an exact replica of the first? If so DB-DB replication may be a better option. Kafka's a great option if you want to process the data elsewhere and/or persist it in another store. But if you just want MS SQL-MS SQL…Kafka itself may be overkill.
The use case is Distributed deployment of Web Application on Azure using PaaS. I read the Azure documentation on SQL Azure database geo-replication, and it seems none of the services tiers best fit this need. The other option is SQL Sync, which is in preview and cannot be used in production.It seems Microsoft Azure does not have any way for a redundant database centric application using PaaS model.
Please help how to resolve the issue or any alternate solution.
Akanksha
Both SQL Data Sync and Geo-replication are for database redundant using. But we need to know you detailed scenarios, so that we can say which one is more fit. Basically Geo-replication is DB level data synchronization used for DR. SQL Data Sync is Table level data synchronization used for reference data replication for both Azure DB and On-prem DB.
I have a local SQL Server database with many tables. I need to transfer data to a remote SQL Server database using vb.net, and task should be run every 2 hr from application. How is it possible to do this? Please suggest the best way for this.
I think the better way is replication. i don't know the possibilities in SQL Server but it works great in mysql. if you enables the replication in two database(Master- Slave) what ever be the changes you have made in the master will simultaneously in the slave. there is possibilities for Bi-Directional Replication in MySQL.
SQL Server provides various types of replication to extend the possibilities. this article helps you to establish replication in Sql Server
I have a scenario as explained below and I need to implement the best Data Sync method.
I have a centralized SQL Azure database (master Database)
There are about 20 (this will increase in future) on-premises SQL Server Databases. These database are not necessarily always connected to the internet.
All master and on-premises DB's will have the same schema/table structures.
I would like to do bidirectional data sync between all on-premises databases with SQL Azure and vice-versa.
Data Sync frequency will be once in a day.
Each on-premises DB size is reasonable(not too big and not too small).
These below options I have explored:
SQL Azure Data Sync
Microsoft Sync Framework
SQL Server 2008 Change Data Capture
SQL Server Change Tracking
I would like to know the best possible method to achieve this.
I have been working with SQl azure data sync, Microsift sync framework and Sql server change tracking. I have no idea about change data capture.
Sql azure data sync.
This is the easiest way to implement data sync. It is a matter of configuration. But unfortunately still in preview and Microsoft no recommended for production yet. We have been using to sync 20 databases spread around different geographical location and so far works good. No coding required. But you may have to pay in future when you are using this service. At the moment it is free.
Microsoft Sync Framework
Microsoft sync framework is for developers. Developers can use Sync framework as an API and develop sync application. Sql azure data sync use sync framework internally. To implement data sync with azure you need to implement N-Tier architecture with WCF. And you need to host your WCF service in azure web site or virtual machine. Considerable development time required and see the following link for sample implementation from Microsoft. Once you develop you can easily configure and use for sync multiple databases.
Database Sync:SQL Server and SQL Express N-Tier with WCF
SQL Server Change Tracking
You need to manually programme the each table for data syn and you need to have link server setup between each sql server. To setup link server with azure database you need to open some specific port.
items #3 and #4 in your list are not really synchronization solutions, just part of it. Both SQL CDC and SQL CT simply allows you to track the changes. you have to put in extra code to grab those changes and apply/sync to another database.
SQL Data Sync service will be your best option if you don't want to write code. Note that up until today (despite the fact its in preview for so long), Data Sync is still in Preview Mode.
If you're find writing code, Sync Fx is a good option as well (SQL Data Sync internally uses Sync Framework).
Azure SQL Data Sync has now reached general availability (GA) as shown on the following Microsoft Article.
Announcing the general availability of Azure SQL Data Sync