I am trying to work out the best practice for recovering specific records that get corrupted in a live SQL Azure DB.
I have got the automatic BacPac reports working.
Currently I would import the relevant BacPac into a recovery DB instance and then compare this instance with the live instance using MS SQL Data Tools or Red Gate Data Compare, and then create a sync script for the required records. Not sure if this is the best practice approach though.
This is also part of a slightly bigger question which is how to do deal with certain backup situations ie
1) Complete DB corruption
Recover using complete BacPac
2) Significant number of records damaged
difficult to answer!!
3) Small number of corrupt records to mend, using user specific records.
Use Data compare tools such as MS SQL Data Tools or RedGate Data Compare.
A colleague also mentioned to me about "periodic transaction log backups". Not sure if SQL Azure does these?
Thoughts?
Thanks in advance.
Related
what is the best way to transfer database copy as backed up file for outside maintenance on the application? But that copy should not have any sensitive data and it can only have dummy data. What is the efficient and best practice to erase all data in the tables and populate with dummy data? ( sql server 2019)
This is not a trivial task. A 3rd party solution would probably be easiest.
There are several answers available here that discuss copying objects in SQL Server Management Studio. Example: Backup SQL Schema Only?.
If you have access to SQL Server Integration Services, you can copy selected objects using the Transfer SQL Server Objects Task. I have tried this once a long time ago, so I have very little experience to describe how it works.
Another option is to create a job that runs a copy-only backup, restores the database, and then runs a manual series of SQL queries to clear or mask sensitive data.
Hello folks first post in stack, btw wonderful community and helps out a lot.
like mentioned in the title what is the best way to copy such a large database? we got an ~ 500 GB Database and im currently moving this database from managed instance to a azure single database using smss:smss copy via deploy to microsoft azure sql database and it takes me right now 22 hours. i feel like im back in early 20s.
it's all in the same subscription and also in the same network configuration. afaik the process of that is that smss creates a bacpac file and then import it back to the single database. but 16 hours is just too long. so do you know any better option to do this quicker because i've a hell of more and partly larger databases to copy.
Did you think about using ETL tools, such as Azure Data Factory? It has good performance to migrate the big data. Ref this performance table:
It supports SQL database and Azure SQL MI. Ref these tutorial:
Copy and transform data in Azure SQL Database by using Azure Data Factory
Copy and transform data in Azure SQL Managed Instance by using Azure
Data Factory
It may takes some money but save much time. As we all know, time is money.
HTH.
Currently we have 100+ databases, some about 10GB in size with millions of records and they are growing at an alarming rate. We need to evaluate our archiving strategy.
Does anyone have any suggestions and sample scripts that go through all the tables and archives the data into an ARCHIVED database - with everything being audited (in regards to number of records imported etc..) and in case of failure it rolls back everything?
Regards
Partitioning can help a lot to archive within single database. Sliding window scenario is a particular tool.
Let me suggest setting up an Admin database. It will handle all setting and information about archiving.
There may be 2 SQL Server instances: Current Server and Archive Server. They will have same structure.
Process copies data from remote server to archive server using settings from Admin DB. There may be need in writing Dynamic SQL. Check Sp_MSForEachDB.
+1 to partitioning idea. To add - I think you can use it also if you have Developers edition
I've been searching the internet for this question:
What are ways to transfer data and tables on a daily basis from an Oracle's Hyperion to SQL Server 2000?
I am an intern at a company and trying to figure out possible ways to do this. Any help or point in the right direction is greatly appreciated
This is going to depend a lot on specifics. Here are just a few possible solutions:
DTS
DTS is packaged with SQL 2000 and is made for this kind of a task. If written correctly, your DTS package can have good error-handling and be rerunnable/reusable.
SSIS
SSIS is actually packaged with SQL 2005 and above, but you can connect it to other databases. It's basically a better version of DTS. (technically it's radically different than DTS, but has a lot of the same functionality)
Linked Servers
From SQL 2000 you should be able to connect directly to your Oracle database as a linked server. In the pros column this kind of direct access can be easy to work with if you don't have any other technical skills such as DTS or SSIS, but it can be complex to get the initial set-up right and there may be security concerns/issues.
Build Your Own
Depending on what other technologies you use you can build your own application to do the ETL (Extract/Transform/Load, which is what you're doing). This could be in .NET, Java, etc. In the pros column you can use something with which you're familiar but there's a big downside here in that most of the low level type of work is already out there in tools like DTS/SSIS, so why reinvent the wheel?
BCP
You can simply extract the data from Oracle as .csv files (or some other format) and then import them back in using SQL Server's Bulk Copy Process. This can be fast, but there aren't many bells and whistles to go with this. If this is a one-time thing with just a few tables though then this is probably the easiest and fastest way to do it.
Third Party Applications
There are a slew of ETL applications already written out there (Data Import, Data Slave, etc.). They will usually provide wizards and one-click solutions (maybe a few more than one click), but they are also going to cost a bit of extra money.
EDIT:
Given your latest comment, I would probably go with a DTS package that's scheduled in SQL Agent to run daily. You can add in error-handling and have the system email/text/call someone if there's ever an issue (or do positive case reporting - ie. send a message when it's successful so that someone knows that there's a problem if they don't get a message each day.
In our company we use ADO.Net for the same task.
We created a source to Oracle , taking all data and then creating it in SQL server
You could write DTS packages to copy the data, and schedule them to run within Sql Server Agent.
See DTS Overview for information on DTS packages.
Here's a tutorial on creating a DTS package: Creating DTS Packages With SQL Server 2000
Oracle Hyperion is a suite of products, largely unrelated to Oracle's database product. I expect you are referring to a product such as Hyperion Financial Management or Hyperion Strategic Finance. These products have APIs that can be consumed using COM Interop or web services. The data can be extracted from the internal multidimensional database by analyzing the database metadata, creating dimension trees, and then using the information to create selections, that represent subcubes within the database; allowing you to get or set cell data.
I don't know what your level of knowledge of multidimensional databases is, but unless it is substantial you may find the task pretty hard. You also need to get a handle on the particular product API.
My company specializes in these kinds of activities, and we have components for this kind of thing. Drop me a line on my blog if you need further advice.
danielvaughan.org
Cheers,
Daniel
I don't know anything about Hyperion, but SQL Server 2000 is very old and may not have a driver to be able to pull data from Hyperion if the version of that is newer than the year 2000. You may need to look to see if there is a way to push the data from Hyperion rather than pull it into SQL Server 2000. One way i have done this is the past is to create pipe delimited text file from the data base that orginally has the data and palce it in a processing directory. I do know that DTS will process a pipe-delimited text file. So if you can't find a driver to process this data directly, consider if you can push it out to file and then process. You wil have to schedule a time gap between the job on Hyperion that creates the file and the DTS package job. But if you are only doing it once a day, that's prbably not a problme.
I have notice, that when i am trying to restore DB, it is restoring DATA + Stored Procedures. I want to restore only data from my existing database in sql server 2008, how can i achieve this.
Scenario is
I have Production DB and Development DB, while developing i have made several changes to SPs and Table Structure. My file which i am using to track those changes is lost and now i want all Table Structure + SP's change in DB, and should also have latest data from production DB.
How can i achieve this?
You can't do a selective restore.
You have to restore the backup to another "work" database and then migrate the bits you want to recover into the target database. After that, you're free to drop the work database.
You can use the following two tools, one to sync your objects and the other one to sync your data:
ApexSQL Diff – a SQL Server database comparison and synchronization tool which detects differences between database objects and resolves them without errors. It generates comprehensive reports on the found differences and can automate the synchronization process between live and versioned databases, backups, snapshots and script folders
ApexSQL Data Diff - a SQL Server data comparison and synchronization tool which detects data differences and resolves them without errors. It can compare and sync live databases and native or natively compressed database backups and generate comprehensive reports on the detected differences
Disclaimer: I work as a Product Support Engineer at ApexSQL