We are at the onset of developing a solution to handle collection and storage of scientific field data.
The solution should handle multiple Thick Windows PC field-clients attached to vehicles (trucks, boats, etc.) connected through cellular-network to a central SQL server.
The clients provide the central server, with data collected from equipment as well as manual input. The clients consume semi-static data from the central server e.g. personnel lists, and predefined data relevant to the specific task.
Connection to the server is erratic and hence the clients should be able to operate fully without connection to the central server for up to 3 hrs.
We are looking at MSMQ and Microsoft Sync Framework as options to handle client/server communication. Any insights you can provide will be much appreciated.
Implement the sync with sync framework over WCF. This will allow you to (a.o.) compress the data with WCf behaviors. And you won't have to expose your sql server to the internets.
http://code.msdn.microsoft.com/Database-Sync-SQL-Server-7e88adab
and http://code.msdn.microsoft.com/Database-SyncSQL-Server-e97d1208
If you can have collisions (update data on multiple clients or both on client and server), implement a command pattern to send data to the server from the clients. Change the data locally on the clients and at the same time create a message to send to the server that does not use sync framework, but can be processed by the server with the same results. This gives you more control and flexibility.
I don't know about msmq. You can have reliable messaging over WCF and as long as the messages you send from the clients are idempotent and the data you send to the clients from the server is considered as the overriding truth, I don't see the need for msmq.
If you can use sql express on the clients, I very much prefer the sync fx 2.0 approach with sql server change tracking, but that's a Microsoft unsupported scenario.
Otherwise, the sync fx 2.1 approach with metadata tables is ok, as long as you don't have more thann, say 50 tables.
If you have more specific questions, I might know more.
Related
One way communication, from app to db is well apparent. Is there a way that my db also communicate back to app/ middle tier, or multiple instances of middle tier or apps.
Can triggers be used for this purpose in conventional rdbms (sql server/postgres)?
If you have a .NET based application, you can get notification back from SQL Server easily.
SQL Server service broker infrastructure, query notifications enables notification from database back to the application.
You can implement this in 3 ways.
1- Using SqlNotificationRequest class. Example
2- Using SqlDependency. Example
3- Using SqlCacheDependency Example
I am looking for the best way to achieve the dynamic sync between server and clients. I have a central server and n-number of clients. I have designed a sync framework which will sync the data from and to between server and each client.
Here each client data is segregated by 2 filter columns. If I hard code these values for each client, then it will create one set of Sp' for each client (ex: selectchanges sp).
How can I achieve this dynamically? Please provide some sample code.
if by dynamic, you meant dynamic filtering, it's not supported.
however, if you're concern is having one selectchanges SP for each client, you can use Filter templates so all the clients share the same SP but use different parameter values.
I have a scenario as explained below and I need to implement the best Data Sync method.
I have a centralized SQL Azure database (master Database)
There are about 20 (this will increase in future) on-premises SQL Server Databases. These database are not necessarily always connected to the internet.
All master and on-premises DB's will have the same schema/table structures.
I would like to do bidirectional data sync between all on-premises databases with SQL Azure and vice-versa.
Data Sync frequency will be once in a day.
Each on-premises DB size is reasonable(not too big and not too small).
These below options I have explored:
SQL Azure Data Sync
Microsoft Sync Framework
SQL Server 2008 Change Data Capture
SQL Server Change Tracking
I would like to know the best possible method to achieve this.
I have been working with SQl azure data sync, Microsift sync framework and Sql server change tracking. I have no idea about change data capture.
Sql azure data sync.
This is the easiest way to implement data sync. It is a matter of configuration. But unfortunately still in preview and Microsoft no recommended for production yet. We have been using to sync 20 databases spread around different geographical location and so far works good. No coding required. But you may have to pay in future when you are using this service. At the moment it is free.
Microsoft Sync Framework
Microsoft sync framework is for developers. Developers can use Sync framework as an API and develop sync application. Sql azure data sync use sync framework internally. To implement data sync with azure you need to implement N-Tier architecture with WCF. And you need to host your WCF service in azure web site or virtual machine. Considerable development time required and see the following link for sample implementation from Microsoft. Once you develop you can easily configure and use for sync multiple databases.
Database Sync:SQL Server and SQL Express N-Tier with WCF
SQL Server Change Tracking
You need to manually programme the each table for data syn and you need to have link server setup between each sql server. To setup link server with azure database you need to open some specific port.
items #3 and #4 in your list are not really synchronization solutions, just part of it. Both SQL CDC and SQL CT simply allows you to track the changes. you have to put in extra code to grab those changes and apply/sync to another database.
SQL Data Sync service will be your best option if you don't want to write code. Note that up until today (despite the fact its in preview for so long), Data Sync is still in Preview Mode.
If you're find writing code, Sync Fx is a good option as well (SQL Data Sync internally uses Sync Framework).
Azure SQL Data Sync has now reached general availability (GA) as shown on the following Microsoft Article.
Announcing the general availability of Azure SQL Data Sync
I'm developing application WinForms .net 4.0 using C# and the backend is SQL Server 2008
the nature of the data for this app is to be displayed to the user in real time manner, whenever the data is changed or new data was added the UI has to reflect that in real time.
I'm trying to find out the best way to get the data from SQL without constantly pooling from the server, I came to a few options:
Create background thread to update the data. (I don't like pooling from the server)
Use SQLDependency class to receive notification from the server.
What do you recommend, or if you have a better method it will be great if you can share it.
If you only have a few clients then a SQLDependency *might be an OK solution. However here is the Microsoft recommended approach for a full blown client/server application.
http://msdn.microsoft.com/en-us/library/ms187528.aspx
This approach is good for many clients but less frequent changes.
The last time I had this type of requirement for more frequent changes with a bunch of clients (i.e. thousands) we built a middleware service that we installed on the server which in turn broadcast the running changes from the database via socket.
We want to distribute / synchronize data from our Datawarehouse (MS SQL Server) to external customers (also MS SQL Server). The connection has to be secure, because we are dealing with trusted data. Transmission of data from our system to external client system must be via the http/https
In addition it is possible that the clients still run their systems with an older database schema, so already existing tables and columns should be transmitted and non existing ones should be ignored.
Its most likely that we will have large database updates and the updates have to arrive in almost real-time.
And it is definitely necessary that the data is stored in a client side datawarehouse / SQL database.
The whole process should also include good monitoring possibilities in case something goes wrong.
We started to develop our own .NET solution but I thought it should be almost a common problem to exchange data between different systems.
Does anybody know about an existing solution which we can adapt to our scenario?
Any help is appreciated!
The problem is so common that it has a dedicated component in SQL Server: Service Broker. Rather than start your own .Net thing and take care of the many problems (how are you gonna handle down time? Retries? duplicates? out of order delivery? authentication of non-domain joined computers? routing for machines that change names? service upgrades? transactional consistency, rollbacks? are you gonna use dtc?). You can look at the demo I gave to SQL connections to see how you can easily scale SSB to a throughput of well over 1000 msgs/sec (1k payload) on commodity hardware.
the only requirement is that all partitcipants must be at least SQL Server 2005 (no SSB in 2000).
Just use regular SQL connections over a secure VPN or an SSH tunnel. Should be very easy to setup for your networking guys.
For example, you can create a linked server. Then a SQL scheduled job could move the data:
truncate table targetserver.dbname.dbo.tablename
insert into targetserver.dbname.dbo.tablename
select a, b, c
from dbname.dbo.sourcetable
Since the linked server talks to your server over a VPN or SSH tunnel, all data is send encrypted over the internet.