I want to use rabbitMQ to store new transactions(records) that are happening in the local database into the remote similar database.
Any suggestions, ideas on this?
Can you share the reason of wanting to use RabbitMQ as the store and forward mechanism to the remote database. I believe most databases already have mechanisms to replicate/backup to a remote database installation. These are tuned to the database and tend to work very reliably. RabbitMQ will not help you shard database operations across multiple remove database instances either.
Why RabbitMQ?
Related
I have a database connected with website, data from website is inserting in that Database, i need to transfer data from that database to another Primary Database (SQL) on another server in real time (minimum latency).
I can not use transactional replication in this case. What are the other alternates to achieve this? Can i integrate DataStreams like Apache kafka etc with SQL server?
Without more detail it's hard to give a full answer. There's what's technically possible, and there's architecturally what actually makes sense :)
Yes you can stream from RDBMS to Kafka, and from Kafka to RDBMS. You can use the Kafka Connect JDBC source and sink. There are also CDC tools (e.g. Attunity, GoldenGate, etc) that support integration with MS SQL and other RDBMS)
BUT…it depends why you want the data in the second database. Do you need an exact replica of the first? If so DB-DB replication may be a better option. Kafka's a great option if you want to process the data elsewhere and/or persist it in another store. But if you just want MS SQL-MS SQL…Kafka itself may be overkill.
We have achieved persistent mechanism with activemq as it has built in support to store messages in SQL database.
But our requirement to store message in SQL using rabbitmq but after some research we are unable to find any strong support or help for this.
Any suggestion would be great help to us.
RabbitMQ uses Mnesia DB to store the definitions about queues/exchanges etc. and it uses a custom DB to store the messages.
Right now, with RabbitMQ is not possible to use an external database to store the messages as ActiveMQ
maybe in the future.
According to my system i have maintained two databases in LAN and online db.But i want to synchronize these two databases. I hope to do this things using microsoft sync frame work.
.http://msdn.microsoft.com/en-us/library/ee819079.aspx
Can i do sync local and online sql db using this? or any suitable method for do this.thank you
Sync Framework is designed for occasionally connected systems, eg. a laptop that can access the corporate network every other day and update its database, but needs to work when it has no corpnet access too. The pairing of Sync Framework is usually a central DB (SQL Server) and local embedded SQL Server Compact or SQL Express on the devices (laptops, phones, tablets etc).
IF the databases are always connected (eg. two DBs in two servers, with 24x7 connectivity between them, even if over Internet) then the appropriate technology is replication. Either Merge or Transactional. Theoretically replication also works when disconnect periods are expected, but Sync Framework is much better at it, and most importantly Sync Framework is not strongly dependent on DNS names as replication is (very important for occasionally connected systems).
Synchronizing the database is a vague term, you have to consider if you want a Master-Slave replication shcme or a Master-Master (the later being very difficult to achieve) and you have to consider what do you want replicated from the database. You also need to consider if more partners will be later added (more databases to 'synchronize'). And you have to be way more careful now about schema changes.
Im looking at developing a simple ecommerce platform and need to replicate product and customer data to the web host over the internet so the website can run disconnected. The two options i can think of at present are using enterprise messaging and database replication.
Im leaning towards database replication over enterprise messaging as enterprise messaging would require additional developer resource to write all the plumbing code. Anyone have any success using sql server one way replication over unreliable wan links through the internet?
I'm sorry I missed this... NitroAccelerator from Nitrosphere.com is built exactly to speed up replication over the internet. It compresses the TDS packets very efficiently and results in 80-90% improvement in replication times.
In the last company I worked for we had full merge replication for some of our customers.
There were 2 scenarios
Merge Replication for hanadheld devices
Some of our customers had PDAs and they subscribed to some published tables of our main database. They were disconnected for large periods and merge replication worked fine and updated changes on both sides when the connection was restored
Full site to site Merge Replications
This was used for customers that had remote offices but required a fully synchronized local database for performance reasons. In most cases the VPN was extremely poor and we did have some instances of the VPN being down for a week and on restoration replication synchronized both database without an issue.
In both cases replication seems to be very fault tolerant and performed very well.
In your case its one way replication so there should not be no merge conflicts to deal with making the situation easier.
There is a learning curve with replication but as a technology it works very well I found even over poor connections.
Liam
We want to distribute / synchronize data from our Datawarehouse (MS SQL Server) to external customers (also MS SQL Server). The connection has to be secure, because we are dealing with trusted data. Transmission of data from our system to external client system must be via the http/https
In addition it is possible that the clients still run their systems with an older database schema, so already existing tables and columns should be transmitted and non existing ones should be ignored.
Its most likely that we will have large database updates and the updates have to arrive in almost real-time.
And it is definitely necessary that the data is stored in a client side datawarehouse / SQL database.
The whole process should also include good monitoring possibilities in case something goes wrong.
We started to develop our own .NET solution but I thought it should be almost a common problem to exchange data between different systems.
Does anybody know about an existing solution which we can adapt to our scenario?
Any help is appreciated!
The problem is so common that it has a dedicated component in SQL Server: Service Broker. Rather than start your own .Net thing and take care of the many problems (how are you gonna handle down time? Retries? duplicates? out of order delivery? authentication of non-domain joined computers? routing for machines that change names? service upgrades? transactional consistency, rollbacks? are you gonna use dtc?). You can look at the demo I gave to SQL connections to see how you can easily scale SSB to a throughput of well over 1000 msgs/sec (1k payload) on commodity hardware.
the only requirement is that all partitcipants must be at least SQL Server 2005 (no SSB in 2000).
Just use regular SQL connections over a secure VPN or an SSH tunnel. Should be very easy to setup for your networking guys.
For example, you can create a linked server. Then a SQL scheduled job could move the data:
truncate table targetserver.dbname.dbo.tablename
insert into targetserver.dbname.dbo.tablename
select a, b, c
from dbname.dbo.sourcetable
Since the linked server talks to your server over a VPN or SSH tunnel, all data is send encrypted over the internet.