I run a query in which I wanted to update more then 130 mln of records. After few hours I got an error:
The transaction log for database 'MGR' is full due to 'ACTIVE_TRANSACTION'.
now I ve got 70 MB free on my C disk drive.
I supose that the problem was with to little disc space and thats why query failed but how can I now regain the lost disc space from before query ?
Im using sql server 2008 R2
Thanks for any hints
The problem has to do with how sql logs all the changes during an active transaction. While a transaction is active, the log cannot be flushed, so if you have a huge active transaction the log keeps growing until it reaches a point where it can exceed its capacity. The amount of logging depends on many factors: the recovery mode (full recovery mode is the one that generates more logging activity). Also, you can breakdown the transaction in small chunks to enable log flushing in between. Also look into table hint TABLOCK. The lost amount of disk must possibly have gone to the log file. Check that out.
Related
I'd like to understand, what causes the size of a SQL Server 12 database. The mdf has 21.5 GB. Using the "Disk Usage by Top Tables" report in SQL Server Management Studio, I can see that 15.4 GB are used by the "Data" of one table. This table has 1,691 rows of 4 columns (int, varchar(512), varchar(512), image). I assume the image column is responsible for most of the consumption. But
Select (sum(datalength(<col1>)) + ... )/1024.0/1024.0 as MB From <Table>
only gives 328.9 MB.
What might be the reason behind this huge discrepancy?
Additional information:
For some rows the image column is updated regularly.
This is a screenshot of the report:
If we can trust it, indices or unused space should not be the cause.
Maybe you are using a lot of indexes per table, these will all add up. Maybe your auto-grow settings are wrong.
The reason was a long running transaction on another unrelated database (!) on the same SQL Server instance. The read committed snapshot isolation level filled the version store. Disconnecting the other application reduced the memory usage to a sensible amount.
The Issue
I've been running a particularly large query, generating millions of records to be inserted into a table. Each time I run the query I get an error reporting that the transaction log file is full.
I've managed to get a test query to run with a reduced set of results and by using SELECT INTO instead of INSERT into as pre built table. This reduced set of results generated a 20 gb table, 838,978,560 rows.
When trying to INSERT into the pre built table I've also tried using it with and without a Cluster index. Both failed.
Server Settings
The server is running SQL Server 2005 (Full not Express).
The dbase being used is set to SIMPLE for recovery and there is space available (around 100 gb) on the drive that the file is sitting on.
The transaction log file setting is for File Growth of 250 mb and to a maximum of 2,097,152 mb.
The log file appears to grow as expected till it gets to 4729 mb.
When the issue first appeared the file grow to a lower value however i've reduced the size of other log files on the same server and this appears to allow this transaction log file grow further by the same amount as the reduction on the other files.
I've now run out of ideas of how to solve this. If anyone has any suggestion or insight into what to do it would be much appreciated.
First, you want to avoid auto-growth whenever possible; auto-growth events are HUGE performance killers. If you have 100GB available why not change the log file size to something like 20GB (just temporarily while you troubleshoot this). My policy has always been to use 90%+ of the disk space allocated for a specific MDF/NDF/LDF file. There's no reason not to.
If you are using SIMPLE recovery SQL Server is supposed manage the task of returning unused space but sometimes SQL Server does not do a great job. Before running your query check the available free log space. You can do this by:
right-click the DB > go to Tasks > Shrink > Files.
change the type to "Log"
This will help you understand how much unused space you have. You can set "Reorganize pages before releasing unused space > Shrink File" to 0. Moving forward you can also release unused space using CHECKPOINT; this may be something to include as a first step before your query runs.
I have 5 SSIS jobs running in sql server job agent and some of them are pulling transactional data into our database over the interval of 4 hours frequently. The problem is log file of our database is growing rapidly which means in a day, it eats up 160GB of disk space. Since our requirement dont need In-point recovery, so I set the recovery model to SIMPLE, eventhough I set it to SIMPLE, the log data consumes more than 160GB in a day. Because of disk full, the scheduled jobs getting failed often.Temporarily I am doing DETACH approach to cleanup the log.
FYI: All the SSIS packages in the job is using Transaction on some tasks. for eg. Sequence Cointainer
I want a permanent solution to keep log file in a particular memory limit and as I said earlier I dont want my log data for future In-Point recovery, so no need to take log backup at all.
And one more problem is that in our database,the transactional table has 10 million records in it and some master tables have over 1000 records on them but our mdf file size is about 50 GB now.I dont believe that this 10 million records should make it to 50GB memory consumption.Whats the problem here?
Help me on these issues. Thanks in advance.
I have a database that's taking up nearly 7 gigs. If I look at the table usage, it should be much less than that, like 40 megs. There was a large log table that I deleted yesterday, but my database still says it's very large.
Here are the stats:
database_name database_size unallocated space
Umbraco_Indoorpower 6911.56 MB 859.59 MB
reserved data index_size unused
31144 KB 26272 KB 3240 KB 1632 KB
I ran this:
DBCC SHRINKDATABASE (umbraco_indoorpower, 99);
And that got my database down to 2.3 gigs. Still though, way too large.
database_name database_size unallocated space
Umbraco_Indoorpower 2302.44 MB 1.63 MB
reserved data index_size unused
30016 KB 26200 KB 3240 KB 576 KB
I'm guessing I'm not freeing up all the space from that log table that I deleted yesterday. I actual ran delete from tblLog. Maybe that was the wrong way to go about it.
Does anyone know how I can free up some more space?
How big is the log file? What is your recovery model? It's quite possible that the database_size number above is nearly 7 GB of log and very little data. Find the files on your hard drive - you can locate the paths using:
EXEC umbraco_indoorpower..sp_helpfile;
I am going to bet that the LDF is HUGE and the MDF is actually small. In which case you are probably in FULL recovery model and have never taken a log backup. If this is true then you can do this:
USE umbraco_indoorpower;
GO
BACKUP LOG umbraco_indoorpower TO DISK = 'C:\some_path\umbraco.trn';
GO
DBCC SHRINKFILE(umbraco_indoorpower_log, 20); -- guessing on target MB size here
(If you are in simple recovery model, the above will fail, but there will be some other explanation why the log file is large - e.g. a long-running or uncommitted transaction, did your delete commit?)
Then you will want to either (a) set up proper maintenance, including full/diff/log backups, which will help make optimal reuse of the log file, or (b) switch to simple recovery, in which case the log will manage itself.
In most cases simple recovery does not provide enough protection in the event of a disaster, but that is for you to decide.
In the meantime, you can shrink the file all you want, but if you keep your recovery model and transaction handling the way it is, you'll just be seeing your file grow again and you'll be back tomorrow running the shrink command. This is absolutely horrible for your files. This is why I object to answers like "Run a shrink operation." I talk about why here:
Oh, the horror! Please stop telling people they should shrink their log files!
http://www.brentozar.com/archive/2009/08/stop-shrinking-your-database-files-seriously-now/
In any case, you have data and log, and for the log to be shrunk, you would have to have made a backup.
Edit: Everything Aaron said
The existing answers are already pretty good. I have one additional solution: Script the database including data (the SSMS UI allows you to do this easily) and execute the script in a fresh database.
You maybe want to switch to simple log model, too (if you don't have a special need for using the full logging model). One thing is for sure: You can't run in full mode and not have proper transaction log management.
Another thing that can take up more space in SQL Server is Service Broker queues. In my case I have 6 million rows in queues taking up 17GB...
I am running SQL Server 2005.
My db backup scheme is:
Recovery model: FULL
Backup Type: Full
Backup component: Database
Backup set will expire: after 0 days
Overwrite media: Back up to the existing media set, Append to the existing backup set
The db is writing to 250GB drive (232GB actual).
My _Data.mdf file is over 55GB and my _Log.ldf is over 148GB.
We ran into a situation where our drive was filled today. I moved our ab_Full.bak and ab_Log.bak files to another drive to make space - about 45GB. Five hours later, free space is at 37GB.
I'm new to managing SQL server; so, I have some basic questions about my backups.
I know I need to update the db to start managing the transaction log size to help prevent this problem in the future. So, assuming I have enough free space, I:
1. right click the db and choose Backup
2. set 'Backup Type' to 'Transaction Log'
3. change 'Backup set will expire' after to 30 days
4. click 'ok'
My understanding is this will move 'closed' transactions from the transaction log to a backup and truncate the transaction log.
Is this plan sound? Will I need to manually resize the log file afterwards?
Thanks for your time.
Are you backing up the transaction log at any time at all?
If you are using the FULL recovery model, then you need to back up the transaction log in addition to backing up the main database, or if you don't want to back up the log (why would you then use the FULL recovery model?) then at least truncate the log at some regular interval.
You should back up the transaction log before every full backup (and keep it as long as you keep the previous full backup) so you can restore to any point in time since the first full backup you've kept. Also, it might be worth backing up the transaction log more often (the total size is the same) in case something bad happens between two full backups.
The best procedure is to regularly backup your log file. In the mean-time, for 'catastrofic' scenarios like the one you described, you may use this snippet to reduce the size of your log:
http://www.snip2code.com/Snippet/12913/How-to-correctly-Shrink-Log-File-for-SQL