I have a table with 2 terabyte worth of data and I need to transfer it between two databases on the same server.
The data is in prod so when I use ETL the production gets affected.
It's 40 days of data and my manager suggested to send daily data starting from the last 40 days.
What is the best solution?
Related
I have a SSIS package with an OLE DB source and an OLE DB destination. Both are connected to the same database. The source is a query from a staging table, some left join and conditions. We truncate the destination table every time and load approximately 30 0000 000 rows.
The issue is the SSIS package is now running more than 12h in the production server, but approximately 45 minutes in the dev and test server.
The last modification we did when this issue occurred was added a left join, on the same staging table as the from, a new column and a condition in the where. To add a little more context, the staging have two rows for a sale, one for the sale and optionally one for the refund. So from now, the destination table have only one row for the sale and a new column for the refund number, instead of two rows for the same sale. The left join on the staging is to find the refund number. We added an index in the staging table to help the left join.
It has the same code in three environments and we even restore the databases in dev and test from a backup from prod, so it have the same data. The resources are better in the production server. All the same version of SQL Server.
The DBA verify the indexes and statistics and rebuild some, but it didn't fix the issue.
I also tried to changed the SSIS to a stored procedure, but it took even longer. With the stored procedure, we saw that the first 8 000 000 rows are fast (maybe 30 minutes), but after it takes like 1h30 to insert 1 000 000 rows. Only the log file is limited but to 2TB. Disks have plenty of free space.
Anybody have any idea why it's taking more than 12h longer in the production server?
Thank you
In our application, we have a master database 'X'. For each new client, we will create a new database copy of master database 'X'.
I am using the following SQL command which will be executed against Azure SQL server.
CREATE DATABASE [NEW NAME] AS COPY OF [MASTER DB]
We are using a custom queue tier so that we can create more than one client at a time parallelly.
I am facing issues in following scenario.
I am trying to create 70 clients. Once 25 clients got created I am getting below error.
Database copy limit per database reached. The database 'BlankDBClient' cannot have more than 10 concurrent database copies
Can you please share your thoughts on this?
SQL Azure has logic to do various operations online/automatically for you (backups, upgrades, etc). There are IOs required to do each copy, so there are limits in place because the machine does not have infinite iops. (Those limits may change a bit over time as we work to improve the service, get newer hardware, etc).
In terms of what options you have, you could:
Restore N databases from a database backup (which would still have IO limits but they may be higher for you depending on your reservation size)
Consider models to copy in parallel using a single source to hierarchically create what you need (copy 2 from one, then copy 2 from each of the ones you just copied, etc)
Stage out the copies over time based on the limits you get back from the system.
Try a larger reservation size for the source and target during the copy to get more IOPS and lower the time to perform the operations.
In addition to Connor answer, you can consider to a have a dacpac or bacpac of that master database stored on Azure Storage and once you have submitted 25 concurrent database copies you can start restoring the dacpac from Azure Storage.
You can also monitor how many database copies are showing COPYING on the state_desc column of the following queries, after sending the first batch of 25 copies, and when those queries return less than 25 rows, start sending more copies until reaching the 25 limit. Keep doing this until finishing the queue of copies required.
Select
[sys].[databases].[name],
[sys].[databases].[state_desc],
[sys].[dm_database_copies].[start_date],
[sys].[dm_database_copies].[modify_date],
[sys].[dm_database_copies].[percent_complete],
[sys].[dm_database_copies].[error_code],
[sys].[dm_database_copies].[error_desc],
[sys].[dm_database_copies].[error_severity],
[sys].[dm_database_copies].[error_state]
From
[sys].[databases]
Left
Outer
Join
[sys].[dm_database_copies]
On
[sys].[databases].[database_id] = [sys].[dm_database_copies].[database_id]
Where
[sys].[databases].[state_desc] = 'COPYING'
SELECT state_desc, *
FROM sys.databases
WHERE [state_desc] = 'COPYING'
I am currently working on a project that is based on:
Azure EventHub1-->Stream Analytics1-->SQL Server DB
Azure EventHub1-->Stream Analytics2-->Document DB
Both SQL Server and DocumentDB have their respective Stream job, but share the same EventHub stream.
DocumentDB is an archive sink and SQL Server DB is a reporting base that should only houses 3 days of data. This is per reporting and query efficiency requirements.
Daily we receive around 30K messages through EventHub, that are pushed through Stream job (basic SELECT query, no manipulation) to a SQL Server table.
To preserve 3 days of data, we had designed a Logic App that calls a SQL SP that deletes any data, based on date, that is more than 3 days old. Runs every day at 12am.
Also, there is another business rule Logic App that READs from the SQL table to perform business logic checks. Runs every 5 mins.
We noticed that for some strange reason the Logic App for data deletion isn't working and data through months has stacked up to 3 Million rows. The SP can be run manually, as tested in Dev setup.
The Logic App shows Succeeded status, but SP execute step shows an Amber check sign, which when expanded says 3 tries occurred.
I am not sure why the SP doesn't delete old data. My understanding is that because Stream job keep pushing data, the Delete operation in SP can't get a Delete Lock and time out.
Try using Azure Automation instead. Create a runbook that runs the stored procedure. Here you will find an example and step-by-step procedure.
I have 2 different databases (A and B) in a SQL Server with different environment server.
Database A has around 100 tables out of which I want to synchronize 15 tables from database A to database B.
So if any add / delete / update happens on those 15 tables, then those entries
should be updated simultaneously in database B.
Same way I need to keep track of update and delete.
Can anybody suggest me what will be the best solution?
I am new to SSIS, i got a task to archive the data from production to Archive DB and then delete the data from production keeping 13 months of data in production. There is around 300+ tables of these i have to archive around 50 tables. Out of these 50 table 6 tables have size around 1 TB.
Listing out the the 2 methods which we are planning.
1. Using 50 data flow tasks in a sequence container.
2. Using SELECT * FROM...INSERT INTO.. where table name and column name can be stored in some configuration table and through loop we can archive the data.
Which will be the better option?
Is there any other better method so please let me know.
What precautions(Performanec tips) i have to take while doing the archive process so that it should not affect the Production server?
Please give your suggestion
Thanks