I have a database with 200 million rows. I run a delete query for deleting 20 million rows.
It takes about 8 hours to delete. In the mean time after running 6 hours transaction fails due to some reason.
Now it takes 5-6 hours to roll back.
How to delete data so that if transaction fails due to some reason, it can not be roll backed?
Related
Last year, I wrote a powershell script that has a DMV query that pulls results into a variable and then inserts those results using bulk copy into another table i had created to save this data then export it to a csv file. We use AutoSys for scheduling jobs so in this case, we set 5 minutes increments for this script to run everytime to stay up to date with usage data. so every minute that table is getting updated with about 5-10 records technically.
the next day, we exported a csv file with all data in that table for that prior day, then truncated the table.
we received a requirement to retain usage information for up to 2 years (long story).
so we altered the truncate statement in the script to delete records only if they are older than 2 years. we tested it of course and it worked fine in Dev/Test (with a shorter time span)
ever since, the table now has over 5 million records after 2 months, and we started seeing the Autosys job failing/restarting often recently.
So the question is, is it possible that because the table is now growing fast, with millions of records, that this is is causing a performance issue by the bulk copy query? does the query "get tired" maybe because there is a bunch of transactions at once?
as far as i know, a table can have millions of records and there wouldnt be an issues with continous inserts nonstop. but maybe bulk copy is different from a regular insert? im thinking of increasing the job scheduling from 5 mins to 10 mins, but im afraid we might lose some usage data here between that extra gap. and it might not even be a solution to the issue...
I have a DB table which size is around 100,000 rows. I Need to run an UPDATE query on that table (affecting every row) which runs for approximately 25 minutes. During this 25 minutes, there will be approximately 300 INSERTs to the same table.
I'm wondering whether those INSERTs will run? Will they be blocked for the time the UPDATE is running, and then executed, or will they just never be executed?
I'm using Postgres database.
Yes, those inserts will run.
An UPDATE, even when changing all rows, does not block the INSERTs
If the update starts before the insert commits, then 100,000 existing records will be updated, the 300 new ones will not be. They will not block each other, unless there is something else going on like attempts to violate a constraint or elaborate triggers.
Seeing slowness in insert/select after truncating a table that has over 6 million rows.
I daily insert 5 to 6 millions records into a table and I was able to insert/select data without any issue for some 7 or 8 days but when the table size went above 10 GB / 30 million rows, there were a few timeout issues.
So I thought of truncating the table daily before data uploading since 1 day's data is enough for me. Now I am seeing extreme slowness in insert/select till i rebuild index in middle of upload. last 2 days it took 5 hours to insert 1 million rows , after a index rebuild remaining 3.5 to 4 million rows went into table in less than 15 mins. I dont prefer to rebuild index in middle of my upload process.
i don't do a .NET Bulk insert , i insert rows in batches using Stored proc , since i do some validation.
I am using SQL Server 2008.
After enabling auto stats update , every thing went normal. from now After truncating big tables i will makes sure table stats get updated if DB auto stats update is disabled.. thanks every one for all your valuable suggestions.
Make sure you update your statistics on all your indexes... and it probably wouldn't hurt to shrink your log file. There are articles on how to do both all over the internet. But that's my best guess at what's going on.
I suggest you to drop indexes as well, when you truncated the table and once the data is loaded into the table then go ahead and create the indexes, this will at least solve the issue to some extent.
With the given information I am not sure of what utility you are using to load the rows into the table,
if you are using BCP, then please mention batch size to a appropriate value, as 6 million records going into one single transaction will create a very big transaction, instead of that you can create a batch size of 50,000 and if any batch fails then truncate the whole table before loading again.
For reading more about batch size, please go to through page (the -b option)
http://msdn.microsoft.com/en-us/library/ms162802.aspx
I have 2 tables which need to get refreshed every one hour. One table is a truncate load and other one is an incremental load. Total process takes around 30 seconds to complete. There are couple of applications hitting these tables on a continuous basis. I can't have applications with blank data at any moment. Any idea what could be done so that operations on these table doesn't affect the output on UI (including truncate/load)? I am thinking of creating a MV on these tables, but any better approach?
Convert the TRUNCATE to a DELETE and make the whole process one transaction. If the current process only takes 30 seconds the extra overhead of deleting and conventional inserts shouldn't be too bad.
I have a SP written in C# which makes calculation on around 2 million rows. Calculations takes about 3 minutes. For each row result is generated in the form of three numbers.
Those results are inserted into temporary table which later is somehow processed.
Results are added in chunks and inserting takes sometimes over 200 minutes (yes, over 3 hours!). Sometimes it takes "only" 50 minutes.
I have modified it so results are kept in memory till the end and then whole 2 millions are dumped in one loop inside one transaction. Still - it takes around 20 minutes.
Similar loop written in SQL with transaction begin/commit takes less than 30 seconds.
Anyone has an idea where is the problem?
Processing 2 millions (so selecting them, etc) takes 3 minutes, inserting results in the best solution 20 minutes.
UPDATE: this table has one clustered index on identity column (to assure that rows are being physically appended at the end), no triggers, no other indexes, no other process is accessing it.
As long as we are all being vague, here is a vague answer. If inserting 2mil rows takes that long, I would check four problems in this order:
Validating foreign key references or uniqueness constraints. You shouldn't need any of these on your temporary table. Do the validation in your CLR before the record gets to the insert step.
Complicated triggers. Please tell me you don't have any triggers on the temporary table. Finish your inserts and then do more processing after everything is in.
Trying to recalculate indexes after each insert. Try dropping the indexes before the insert step and recreating after.
If these aren't it, you might be dealing with record locking. Do you have other processes that hit the temporary table that might be getting in the way? Can you stop them during your insert?