I have a online Database which will be updated Daily from various Sources.
I need to have a local Database with some tables from Server Database which have to check for any changes or new rows in tables in server and update the local Database for particular Intervals of Time. How can I Achieve this???
You may want to look into SQL Server Replication.
Replication will manage the data synchronization between the two copies of your database. You can configure replication for any tables in the database, including all tables. Replication will take care of checking for updates, adds and deletes from the Server Database and transfer the changes to the local database.
You can setup replication to update the local database at near-real-time or you can schedule periodic updates.
Replication is a high-maintenance solution. It's designed to maintain two copies of the same database with significant reliability. This makes replication a good solution when you must avoid data problems or recover from problems with little to no data loss.
If you don't require the high-maintenance solution, then SQL Server Integration Services (SSIS) may be a good alternative. With SSIS, you develop the data transfer and data management solution. Along with managing data problems, you design the solution to identify data adds, deletes and updates.
Related
I'm building a monitoring tool which analyzes information in some sql tables and creates some charts and alerts based on some configurable criterias. However the underlying application is now getting some errors. I think it's because my queries are rather intensive on the tables which causes them to be locked for some amount of time and my idea of a work around is to synchronize the tables to a monitoring database and do my operations there.
Do you have any other ideas? And if I do the sync, whats the best way of syncing tables in SQL server? I prefer if the sync is as close to real-time as possible.
If you are running SQL Server 2008 R2 or above, Transactional Replication is usually a good fit for this type of scenarios and can support near real time synchronization. Here are few links to get familiar with Replication
Overview of replication
Use of replication for data warehousing and reporting applications.
The other solution is to Log ship the transactional database to reporting database.But
Log shipping is asynchronous operation, so the state of data in reporting database will be behind that of the data in transactional database.
You need to log ship the entire database even if you end up using only couple of tables.
The reporting database is not available when it is restoring from the transactional database.
so that would not match to your requirements.
1.I am new in azure, I want to know can we have same replication mechanism provided by on premise sql on azure sql db?
2 .Issue we are facing is, few of the tables are growing fast, daily insert around 10k records, so we are planning to keep only few months say 6 data on main DB and copy all data to other DB using replication (not sure if feasible).
We need to read data from backup as well in application for some reports.
Please suggest on this if replication will work or any other solution.
Geo-replication uses a version of AlwaysOn with async replicas under the hood. It is very similar to a distributed Availability Group in SQL 2016, but you cannot control it, you can only turn it on or off.
Replication will work for that, but it would replicate all the data in the DB, not just the tables you want.
Link to Azure Documentation: https://azure.microsoft.com/en-us/documentation/articles/sql-database-geo-replication-overview/
I want to understand the best approach for SQL Server architecture on production environment.
Here is my problem:
I have database which has on average around 20,000 records being inserted every second in various tables.
We have reports also implemented for the same, now what's happening is whenever reports is searched by user, performance of other application steeps down.
We have implemented
Table Partitioning
Indexing
And all other required things.
My question is: can anyone suggest an architecture that have different SQL Server databases for reports and application, and they can sync themselves online every time when new data is entered in master SQL Server?
Some what like Master and Slave Architecture. I understand Master and Slave architecture, however need to get more idea around it.
Our main tables are having around 40 millions rows (table partitioning done)
In SQL Server 2008R2 you have database mirroring and replication available, which will keep two databases in sync.
A schema which is efficient for OLTP is unlikely to be efficient for large volume reporting. The 'live' and 'reporting' databases should have different schema with an ETL process moving data from one to the other. I'd would like to negotiate with the business just how synchronised the reporting database needs to be. If the reports are processing large amounts of data they will take some time to run so a lag in data replication will not be noticed, I would suggest. In extremis you could construct a solution using Service Broker to move the data and processing on the reporting server to distribute it amonst the reporting tables.
The numbers you quote (20,000 inserts per second, 40 millions rows in largest table) suggests a record doesn't reside in the DB for long. You would have a significant load performing DELETEs. Optimising these out of peak hours could be sufficient to solve your problems.
I am new to sql server .I have a sql server and I have 2nd sql server(backupserver).I want to copy data from sql server to 2nd one and all my transactions create update delete must be reflected in the 2nd server immediately in real time.I have very large data in my table assuming million of rows.How can I achieve this.I dont have previalge to use third party tools.
You specify that you need data to be reflected on the second server immediately, in real time.
This requirement will add latency to every transaction that occurs on your primary server, since every action will need to be committed on the secondary server before it can be committed on your primary server.
In addition to the latency, this requirement will also likely reduce your availability. If the secondary server is no longer able to successfully commit transactions from the primary, then the primary can no longer commit transactions either, and your system is down.
For more information about these constraints, refer to the extensive discussion around this topic(CAP theorem).
If you're OK with these restrictions, you might consider using Synchronous Database Mirroring (High-Safety Mode).
If you're not OK with these restrictions, please adjust / clarify your requirements.
You may try to get some help from these two references:
Replication in MS SQL Server
SQL Server Replication
On a side note an important thing you should need to plan before doing replication is the Replication model which you will use for your replication.
There is a list of Replication model which you can use:-
Peer-to-peer model
Central publisher model
Central publisher with remote distributor model
Central subscriber model
Publishing subscriber model
Each one of the above has its own advantages. Check them as per your need.
Also to add to it there are three types of Replication:-
Transactional replication
Merge replication
Snapshot replication
Check out this tutorial on SQL Server 2008 replication:
Tutorial: Replicating Data Between Continuously Connected Servers
Since you said "... reflected in the 2nd server immediately in real time", that means you want to use Transactional Replication. You still may only choose to replicate certain tables.
Does the "millions of rows" represent some kind of history? If so, consider the risks that Michael mentioned... and whether you need all the history in the 2nd server, or just current / recent activity. If it's just current/recent, it may be safer and less of a system drain, to write something in T-SQL or SSIS, for a job to execute that loops, reads, and copies the data.
That could be done with linked servers and triggers... but the risks Michael mentions, about preventing the primary server from committing transactions, are as much or more a concern with triggers... that you can avoid with your own T-SQL/SSIS + job.
Hope that helps...
Is there a way to backup certain tables in a SQL Database? I know I can move certain tables into different filegroups and preform a backup on these filegroup. The only issue with this is I believe you need a backup of all the filegroups and transaction logs to restore the database on a different server.
The reason why I need to restore the backup on a different server is these are backups of customers database. For example we may have a remote customer and need to get a copy of they 4GB database. 90% of this space is taken up by two tables, we don’t need these tables as they only store images. Currently we have to take a copy of the database and upload it to a FTP site…With larger databases this can take a lot of the time and we need to reduce the database size.
The other way I can think of doing this would be to take a full backup of the DB and restore it on the clients SQL server. Then connect to the new temp DB and drop the two tables. Once this is done we could take a backup of the DB. The only issue with this solution is that it could use a lot of system restores at the time of running the query so its less than ideal.
So my idea was to use two filegroups. The primary filegroup would host all of the tables except the two tables which would be in the second filegroup. Then when we need a copy of the database we just take a backup of the primary filegroup.
I have done some testing but have been unable to get it working. Any suggestions? Thanks
Basically your approach using 2 filegroups seems reasonable.
I suppose you're working with SQL Server on both ends, but you should clarify for each which whether that is truly the case as well as which editions (enterprise, standard, express etc.), and which releases 2000, 2005, 2008, (2012 ? ).
Table backup in SQL Server is here a dead horse that still gets a good whippin' now and again. Basically, that's not a feature in the built-in backup feature-set. As you rightly point out, the partial backup feature can be used as a workaround. Also, if you just want to transfer a snapshot from a subset of tables to another server, using ftp you might try working with the bcp utility as suggested by one of the answers in the above linked post, or the export/import data wizards. To round out the list of table backup solutions and workarounds for SQL Server, there is this (and possibly other ? ) third party software that claims to allow individual recovery of table objects, but unfortunately doesn't seem to offer individual object backup, "Object Level Recovery Native" by Red Gate". (I have no affiliation or experience using this particular tool).
As per your more specific concern about restore from partial database backups :
I believe you need a backup of all the filegroups and transaction logs
to restore the database on a different server
1) You might have some difficulties your first time trying to get it to work, but you can perform restores from partial backups as far back as SQL Server 2000, (as a reference see here
2) From 2005 and onward you have the option of partially restoring today, and if you need to you can later restore the remainder of your database. You don't need to include all filegroups-you always include the primary filegroup and if your database is simple recovery mode you need to add all read-write filegroups.
3) You need to apply log backups only if your db is in bulk or full recovery mode and you are restoring changes to a readonly filegroup that since last restore has become read-write. Since you are expecting changes to these tables you will likely not be concerned about read only filegroups, and so not concerned about shipping and applying log backups
You might also investigate some time whether any of the other SQL Server features, merge replication, or those mentioned above (bcp, import/export wizards) might provide a solution that is more simple or more adequately meets your needs.