I am looking for the best way to achieve the dynamic sync between server and clients. I have a central server and n-number of clients. I have designed a sync framework which will sync the data from and to between server and each client.
Here each client data is segregated by 2 filter columns. If I hard code these values for each client, then it will create one set of Sp' for each client (ex: selectchanges sp).
How can I achieve this dynamically? Please provide some sample code.
if by dynamic, you meant dynamic filtering, it's not supported.
however, if you're concern is having one selectchanges SP for each client, you can use Filter templates so all the clients share the same SP but use different parameter values.
Related
I would like to setup a datawarehouse which can be used by our companies Qlik application. The applications where I would like to retrieve data from a mostly running on-premises and all have a SQL server database as a source. Application which doesn't have an accessible SQL server as a source can be accessed via a REST API and/or Webservice.
This is how the setup looks like:
Data warehouse is a SQL-server (Standard, no Express version)
SQL datasources are running on 3 different SQL-servers (2 are Standard, 1 is Express)
Other sources are Webservice and/or REST API accessibel (SaaS application).
The SQL-servers are all on-premises are located within our network. The SaaS application is running in a data center (cloud).
Preferable I would like to have data as live as possible and the load on the server as small as possible. To do so I was wondering if there are ETL-tools which work with SQL Change Tracking to keep track of changes on table level (so that changes are PUSH based on not PULL). If this is the case I can let them sync sequential and set per table if it has to be synced based on Change Tracking or only full sync a day. As soon as I have the data in my datawarehouse I can create some data transformations in the ETL-tool or with T-SQL, that doesn't matter.
Hopefully there are some people around here who can tell me which ETL tool to use. There is a lot of information on the internet, but not much who go into the subject of SQL Change Tracking.
Many thanks in advance!
We have a multi-tenant, single db application where some customers have expressed the desire to get direct access to their own data.
I have been suggested looking into Azure Data Sync to achieve a setup where each of the customers get their own Azure SQL instance to which we setup a one-way synchronization of their data from the master database.
I managed to find some documentation on this, but one I got around to try it out in a lab setup, it looks like the ability to filter rows in the sync job has been removed in a later iteration of the Azure Data Sync service.
Am I wrong or is that feature really gone? If so, what would be your suggestions to achieve something similar on Azure?
You cannot filter rows using Azure SQL Data Sync. However, you can build a custom solution based on Sync Framework as explained here.
One way communication, from app to db is well apparent. Is there a way that my db also communicate back to app/ middle tier, or multiple instances of middle tier or apps.
Can triggers be used for this purpose in conventional rdbms (sql server/postgres)?
If you have a .NET based application, you can get notification back from SQL Server easily.
SQL Server service broker infrastructure, query notifications enables notification from database back to the application.
You can implement this in 3 ways.
1- Using SqlNotificationRequest class. Example
2- Using SqlDependency. Example
3- Using SqlCacheDependency Example
We are at the onset of developing a solution to handle collection and storage of scientific field data.
The solution should handle multiple Thick Windows PC field-clients attached to vehicles (trucks, boats, etc.) connected through cellular-network to a central SQL server.
The clients provide the central server, with data collected from equipment as well as manual input. The clients consume semi-static data from the central server e.g. personnel lists, and predefined data relevant to the specific task.
Connection to the server is erratic and hence the clients should be able to operate fully without connection to the central server for up to 3 hrs.
We are looking at MSMQ and Microsoft Sync Framework as options to handle client/server communication. Any insights you can provide will be much appreciated.
Implement the sync with sync framework over WCF. This will allow you to (a.o.) compress the data with WCf behaviors. And you won't have to expose your sql server to the internets.
http://code.msdn.microsoft.com/Database-Sync-SQL-Server-7e88adab
and http://code.msdn.microsoft.com/Database-SyncSQL-Server-e97d1208
If you can have collisions (update data on multiple clients or both on client and server), implement a command pattern to send data to the server from the clients. Change the data locally on the clients and at the same time create a message to send to the server that does not use sync framework, but can be processed by the server with the same results. This gives you more control and flexibility.
I don't know about msmq. You can have reliable messaging over WCF and as long as the messages you send from the clients are idempotent and the data you send to the clients from the server is considered as the overriding truth, I don't see the need for msmq.
If you can use sql express on the clients, I very much prefer the sync fx 2.0 approach with sql server change tracking, but that's a Microsoft unsupported scenario.
Otherwise, the sync fx 2.1 approach with metadata tables is ok, as long as you don't have more thann, say 50 tables.
If you have more specific questions, I might know more.
I have a scenario as explained below and I need to implement the best Data Sync method.
I have a centralized SQL Azure database (master Database)
There are about 20 (this will increase in future) on-premises SQL Server Databases. These database are not necessarily always connected to the internet.
All master and on-premises DB's will have the same schema/table structures.
I would like to do bidirectional data sync between all on-premises databases with SQL Azure and vice-versa.
Data Sync frequency will be once in a day.
Each on-premises DB size is reasonable(not too big and not too small).
These below options I have explored:
SQL Azure Data Sync
Microsoft Sync Framework
SQL Server 2008 Change Data Capture
SQL Server Change Tracking
I would like to know the best possible method to achieve this.
I have been working with SQl azure data sync, Microsift sync framework and Sql server change tracking. I have no idea about change data capture.
Sql azure data sync.
This is the easiest way to implement data sync. It is a matter of configuration. But unfortunately still in preview and Microsoft no recommended for production yet. We have been using to sync 20 databases spread around different geographical location and so far works good. No coding required. But you may have to pay in future when you are using this service. At the moment it is free.
Microsoft Sync Framework
Microsoft sync framework is for developers. Developers can use Sync framework as an API and develop sync application. Sql azure data sync use sync framework internally. To implement data sync with azure you need to implement N-Tier architecture with WCF. And you need to host your WCF service in azure web site or virtual machine. Considerable development time required and see the following link for sample implementation from Microsoft. Once you develop you can easily configure and use for sync multiple databases.
Database Sync:SQL Server and SQL Express N-Tier with WCF
SQL Server Change Tracking
You need to manually programme the each table for data syn and you need to have link server setup between each sql server. To setup link server with azure database you need to open some specific port.
items #3 and #4 in your list are not really synchronization solutions, just part of it. Both SQL CDC and SQL CT simply allows you to track the changes. you have to put in extra code to grab those changes and apply/sync to another database.
SQL Data Sync service will be your best option if you don't want to write code. Note that up until today (despite the fact its in preview for so long), Data Sync is still in Preview Mode.
If you're find writing code, Sync Fx is a good option as well (SQL Data Sync internally uses Sync Framework).
Azure SQL Data Sync has now reached general availability (GA) as shown on the following Microsoft Article.
Announcing the general availability of Azure SQL Data Sync