The issue I'm trying to solve is index rebuild that keeps consuming the entire disc space (and then fails the operation).
The command in use is (while iterating over all tables in the DB):
ALTER INDEX ALL ON MyTable REBUILD
SQL Server 2014
Total disk space is 465 GB
The database size is ~320GB
During index rebuild the disc is filled entirely. And when the operation fails, the space is not deallocated. That means no free space is left on the disk - at all.
Any suggestions regarding what can be done differently?
Thank you.
Any suggestions regarding what can be done differently?
Don't rebuild all your indexes. Why are you even doing that? Try just updating statistics instead.
Rebuild the indexes one-by one.
If a single index is too large to rebuild, consider disabling it first, which is like dropping the index but retaining the index definition. Warning, though, disabling the primary key index (or a unique index referenced by a foreign key) or a clustered index has side effects. Foreign keys on other tables will be disabled (NOCKECK), and if the clustered index is disabled all other indexes on the table will be disabled too.
And when the operation fails, the space is not deallocated.
When a database is set to auto-grow, it grows. There is no "auto shrink" setting for a database. To reclaim space allocated to a database after a on-time operation you can shrink it.
And if you absolutely have to rebuild all indexes for some reason, then get enough drive space to do so with room to spare.
Do not shrink constantly (e.g., as part of a maintenance plan).
It's OK to shrink in rare/emergency situations when you have no option, but ironically, can fragment the very indexes your were trying to rebuild.
Related
In a PostgreSQL database, assuming there is enough disc space, can I create an index in a live production database without risk of downtime? In other words, are there locks or possible crash or data loss possible or something else with the creation of an index.
To be more precise, it's an index on a JSONB sub-property in a 1Gb table:
create index on foo((bar->>'region'));
I'm currently hesitating in running this create index (it works without a problem on my machine with a database sample).
CREATE INDEX will block writes to the table for the duration of the CREATE INDEX statement, but CREATE INDEX CONCURRENTLY will not. The CONCURRENTLY modifier does require more work, but in this case, it should be negligible (re-reading 1GB shortly after it is read for the initial scan).
Reads are not affected by either form of CREATE INDEX, other than competing for I/O at the operating system level.
Does inserting/ updating or deleting a record in DB auto update the indexes?
I've seen rebuild commands for indexes and I just can't understand if I have to do all these manually, then what's the purpose of indexes. This is so inconvenient that I spend time rebuilding the indexes every week or so.
When index fragmentation becomes too high, indexes will be very inefficient. There is need to rebuild index.
Let SQL Server Tell You Which Indexes to Rebuild
Rebuilding an index drops and re-creates the index. This removes fragmentation, reclaims disk space by compacting the pages based on the existing fill factor setting, and reorders the index rows in contiguous pages.
Reorganizing and Rebuilding Indexes
Yes, when you insert, update, or delete a row, any indexes on that table automatically reflect the change.
So a pretty big problem just came up with a few databases we're using. For some reason, people's databases have grown to a ridiculous file size without any change in the table's data. While I have no idea what is causing this all of a sudden, I'm more concerned right now with clearing the disk space its using. I've run sp_spaceused and have tracked the main culprit down to 1 of 2 tables (depending on the database). For each database, one of these tables is allocating over half a GB to reserved spacewhile the data is only something like 50 MB. It shows that the index_size is at ~113 MB. The table has no clustered index and has about 15 columns, all of relatively small length except for 2 columns that are of type nvarchar with length 255 (these are usually null or empty in the table).
I've tried running DBCC shrinkdatabase and truncate table but it didn't do anything. I've researched this a bit and some others have had this problem, but if shrinkdatabase didn't fix it, then no solution was found for them either.
Let me know if there's anything else you guys need to know about the table or database set up. I don't know what else to try and this is a significant issue for us as people's databases are all of a sudden taking up 10 times the space they were before.
EDIT:
After trying to run DBCC DBREINDEX and trying to change to a clustered index through Enterprise Manager, I get an error message saying:
Could not allocate new page for database 'DB'. There are no more pages available in filegroup PRIMARY. Space can be created by dropping objects, adding additional files, or allowing file growth.
I've tried deleting rows from this table too and it has no effect on the table's size. The log file increases, as expected, but that's the only change to the table's size.
These are heaps (no clustered indexes)? Why?
If the users have done a lot of updates or deletes, I would try rebuilding the tables. Shrinkdatabase is not the way to do this. It will not fix fragmentation, unrecovered space from deletes or altering the width of a column, or rows wasted as forwarding pointers. I bet the tables would fare much better with a rebuild and/or a clustered index. In SQL Server 2000 you would do this by either:
(a) adding a clustered index. (I can't give you exact syntax for adding a clustered index (or changing one of your existing indexes, or the non-clustered primary key to be clustered), because I don't have enough information about your table.)
(b) DBCC DBREINDEX('dbo.tablename');
This will rebuild the non-clustered indexes on your heaps, but that may do you no good depending on where the wasted space is happening. I still suggest you should create a clustered index. If you can share some details about your table (e.g. the indexes that exist, what kind of queries are typically run, how you currently uniquely identify a row and the nature of data changes/additions to this table) we might be able to give more detailed advice.
I understand that indexes should get updated automatically but when that does not happen we need to reindex.
My question is (1) Why this automatic udate fails or why an index become bad?
(2) How do I prgramatically know which table/index needs re-indexing at a point of time?
Indexes' statistics may be updated automatically. I do not believe that the indexes themselves would be rebuilt automatically when needed (although there may be some administrative feature that allows such a thing to take place).
Indexes associated with tables which receive a lot of changes (new rows, updated rows and deleted rows) may have their indexes that become fragmented, and less efficient. Rebuilding the index then "repacks" the index in a contiguous section of storage space, a bit akin to the way defragmentation of the file system makes file access faster...
Furthermore the Indexes (on several DBMS) have a FILL_FACTOR parameter, which determine how much extra space should be left in each node for growth. For example if you expect a given table to grow by 20% next year, by declaring the fill factor around 80%, the amount of fragmentation of the index should be minimal during the first year (there may be some if these 20% of growth are not evenly distributed,..)
In SQL Server, It is possible to query properties of the index that indicate its level of fragmentation, and hence it possible need for maintenance. This can be done by way of the interactive management console. It is also possible to do this programatically, by way of sys.dm_db_index_physical_stats in MSSQL 2005 and above (maybe even older versions?)
What is the difference between Index Rebuilding and Index Reorganizing?
Think about how the index is implemented. It's generally some kind of tree, like a B+ Tree or B- Tree. The index itself is created by looking at the keys in the data, and building the tree so the table can be searched efficiently.
When you reorganize the index, you go through the existing index, cleaning up blocks for deleted records etc. This could be done (and is in some databases) when you make a deletion, but that imposes some performance penalty. instead, you do it separately in order to do it more or less batch mode.
When you rebuild the index, you delete the existing tree and read all the records, building a new tree directly from the data. That gives you a new, and hopefully optimized, tree that may be better than the results of reorganizing the table; it also lets you regenerate the tree if it somehow has been corrupted.
REBUILD locks the table for the whole operation period (which may be hours and days if the table is large).
REORGANIZE doesn't lock the table.
Well. actually, it places some temporary locks on the pages it works with right now, but they are removed as soon as the operation is complete (which is fractions of second for any given lock).
As #Andomar noted, there is an option to REBUILD an index online, which creates the new index, and when the operation is complete, just replaces the old index with the new one.
This of course means you should have enough space to keep both the old and the new copy of the index.
REBUILD is also a DML operation which changes the system tables, affects statistics, enables disabled indexes etc.
REORGANIZE is a pure cleanup operation which leaves all system state as is.
There are a number of differences. Basically, rebuilding is a total rebuild of an index - it will build a new index, then drop the existing one, whereas reorganising it will simply, well... it will reorganise it.
This blog entry I came across a while back will explain it much better than I can. :)
Rebuild it droping the current indexes and recreating new ones.
Reorganizing is like putting the house in order with what u already have.
it is a good practice to use 30% fragmentation to determine rebuild vs reorganize.
<30% reorganize vs. >30% rebuild
"Reorganize index" is a process of cleaning, organizing, and defragmenting of "leaf level" of the B-tree (really, data pages).
Rebuilding of the index is changing the whole B-tree, recreating the index.
It’s recommended that index should be reorganized when index fragmentation is from 10% to 40%; if index fragmentation is great than 40%, it’s better to rebuild it.
Rebuilding of an index takes more resources, produce locks and slowing performance (if you choose to keep table online). So, you need to find right time for that process.
In addition to the differences above (basically rebuild will create the index anew, and then "swap it in" for the existing one, rather than trying to fix the existing one), an important consideration is that a rebuild - even an Enterprise ONLINE rebuild - will interfere with snapshot isolation transactions.
TX1 starts snapshot transaction
TX1 reads from T
TX2 rebuilds index on T
TX2 rebuild complete
TX1 read from T again:
Error 3961, Snapshot isolation transaction failed in database because the object accessed by the statement has been modified by a DDL statement in another concurrent transaction since the start of this transaction. It is disallowed because the metadata is not versioned. A concurrent update to metadata can lead to inconsistency if mixed with snapshot isolation.
Rebuild index - rebuilds one or more indexes for a table in the specified database.
Reorganise index - Defragments clustered and secondary indexes of the specified table