We want to distribute / synchronize data from our Datawarehouse (MS SQL Server) to external customers (also MS SQL Server). The connection has to be secure, because we are dealing with trusted data. Transmission of data from our system to external client system must be via the http/https
In addition it is possible that the clients still run their systems with an older database schema, so already existing tables and columns should be transmitted and non existing ones should be ignored.
Its most likely that we will have large database updates and the updates have to arrive in almost real-time.
And it is definitely necessary that the data is stored in a client side datawarehouse / SQL database.
The whole process should also include good monitoring possibilities in case something goes wrong.
We started to develop our own .NET solution but I thought it should be almost a common problem to exchange data between different systems.
Does anybody know about an existing solution which we can adapt to our scenario?
Any help is appreciated!
The problem is so common that it has a dedicated component in SQL Server: Service Broker. Rather than start your own .Net thing and take care of the many problems (how are you gonna handle down time? Retries? duplicates? out of order delivery? authentication of non-domain joined computers? routing for machines that change names? service upgrades? transactional consistency, rollbacks? are you gonna use dtc?). You can look at the demo I gave to SQL connections to see how you can easily scale SSB to a throughput of well over 1000 msgs/sec (1k payload) on commodity hardware.
the only requirement is that all partitcipants must be at least SQL Server 2005 (no SSB in 2000).
Just use regular SQL connections over a secure VPN or an SSH tunnel. Should be very easy to setup for your networking guys.
For example, you can create a linked server. Then a SQL scheduled job could move the data:
truncate table targetserver.dbname.dbo.tablename
insert into targetserver.dbname.dbo.tablename
select a, b, c
from dbname.dbo.sourcetable
Since the linked server talks to your server over a VPN or SSH tunnel, all data is send encrypted over the internet.
Related
I am running a MS Access app using a few tables hosted on our native SQL server to facilitate easy integration with PowerBI and other apps. I am an intermediate MS Access user and new the SQL Server. I don't store anything critically sensitive in any tables and am happy with our normal security for data at rest. However, when my app or PowerBI requests data from the server, how is that data protected? I don't want the increased complexity that comes w/ certificate management and processing time associated with encrypting data. However, I don't want to be low hanging fruit for attacks when I request data from the server (i.e. attacks in transit).
Thanks!
First of all - security is about three things. Confidentiality, integrity and availability. Confidentiality means, that people are not supposed to see data that is not meant for them. Integrity means, that the data is what you expect it to be (no third party is able to manipulate it). And availability means, that the server is always up and running.
This is a simplification, but just to make the point that security is more than many of us believe. If your server is down, then the security is equal to zero.
Having said that, you wanted to know how to protect the communication channel. Sorry to say that, but the best thing we have right now is TLS, which requires certificate management. Look into this page to understand, how this can be configured: http://dba-datascience.com/ssl-or-tls-encryption-on-sql-server/
If you want to know how to protect your SQL Server even more (beyond the communication channel), look into the CIS Benchmarks available here: https://www.cisecurity.org/cis-benchmarks/. They are very technical (which is good), but may be also confusing a bit when you see it for the first time.
I have a database connected with website, data from website is inserting in that Database, i need to transfer data from that database to another Primary Database (SQL) on another server in real time (minimum latency).
I can not use transactional replication in this case. What are the other alternates to achieve this? Can i integrate DataStreams like Apache kafka etc with SQL server?
Without more detail it's hard to give a full answer. There's what's technically possible, and there's architecturally what actually makes sense :)
Yes you can stream from RDBMS to Kafka, and from Kafka to RDBMS. You can use the Kafka Connect JDBC source and sink. There are also CDC tools (e.g. Attunity, GoldenGate, etc) that support integration with MS SQL and other RDBMS)
BUT…it depends why you want the data in the second database. Do you need an exact replica of the first? If so DB-DB replication may be a better option. Kafka's a great option if you want to process the data elsewhere and/or persist it in another store. But if you just want MS SQL-MS SQL…Kafka itself may be overkill.
According to my system i have maintained two databases in LAN and online db.But i want to synchronize these two databases. I hope to do this things using microsoft sync frame work.
.http://msdn.microsoft.com/en-us/library/ee819079.aspx
Can i do sync local and online sql db using this? or any suitable method for do this.thank you
Sync Framework is designed for occasionally connected systems, eg. a laptop that can access the corporate network every other day and update its database, but needs to work when it has no corpnet access too. The pairing of Sync Framework is usually a central DB (SQL Server) and local embedded SQL Server Compact or SQL Express on the devices (laptops, phones, tablets etc).
IF the databases are always connected (eg. two DBs in two servers, with 24x7 connectivity between them, even if over Internet) then the appropriate technology is replication. Either Merge or Transactional. Theoretically replication also works when disconnect periods are expected, but Sync Framework is much better at it, and most importantly Sync Framework is not strongly dependent on DNS names as replication is (very important for occasionally connected systems).
Synchronizing the database is a vague term, you have to consider if you want a Master-Slave replication shcme or a Master-Master (the later being very difficult to achieve) and you have to consider what do you want replicated from the database. You also need to consider if more partners will be later added (more databases to 'synchronize'). And you have to be way more careful now about schema changes.
Im looking at developing a simple ecommerce platform and need to replicate product and customer data to the web host over the internet so the website can run disconnected. The two options i can think of at present are using enterprise messaging and database replication.
Im leaning towards database replication over enterprise messaging as enterprise messaging would require additional developer resource to write all the plumbing code. Anyone have any success using sql server one way replication over unreliable wan links through the internet?
I'm sorry I missed this... NitroAccelerator from Nitrosphere.com is built exactly to speed up replication over the internet. It compresses the TDS packets very efficiently and results in 80-90% improvement in replication times.
In the last company I worked for we had full merge replication for some of our customers.
There were 2 scenarios
Merge Replication for hanadheld devices
Some of our customers had PDAs and they subscribed to some published tables of our main database. They were disconnected for large periods and merge replication worked fine and updated changes on both sides when the connection was restored
Full site to site Merge Replications
This was used for customers that had remote offices but required a fully synchronized local database for performance reasons. In most cases the VPN was extremely poor and we did have some instances of the VPN being down for a week and on restoration replication synchronized both database without an issue.
In both cases replication seems to be very fault tolerant and performed very well.
In your case its one way replication so there should not be no merge conflicts to deal with making the situation easier.
There is a learning curve with replication but as a technology it works very well I found even over poor connections.
Liam
I'm currently trying to build a high-availability and load-balanced web application with SQL Server Replication Services technologies. Automatic fail-over is built into the application logic. Basically, there are two groups of application servers running the same, each with its own SQL Server instance. They are set to use the other instance in case of failure. Data is continously replicated between the two SQL Server instances via transactional replication. (A few seconds lag exists, that's okay.)
I set up both servers in a way that Distribution agents run on the Distributor (= Publisher). My idea is, that as long as Server A (publisher) is working, it 'collects' the transactions and forwards them to Server B (subscriber) as soon as it's available. The same for Server B. The default option (distributor on the subscriber) would 'lose' changes while the subscriber is offline. Am I right with that? UPDATE to this first question: transactions (waiting to be delivered) are stored in the distribution database, so they are "safe" anyway. This leads be to another question: if the Distributor Agent is on the subscriber, how will it know about new transactions to be delivered? Via frequent polling?
The two databases have the same schema (except identity seed and increment). Each read-write table is cross-replicated. Why don't I see a loop, when I insert a row into a table? When I read about bi-directional replication in documents or blogs, is this scenario what they mean or rather updateable transactional replication?
What do you think about this scenario in general? This is my first time with replication, and I fear the risks. Therefore any comments are very welcome.