Unable get 600,000 deletion stubs ro purge from database - lotus-domino

Question

Update: I have issued a: show database mycleandb.nsf and I can see there are still 835,000 deletion stubs. I can't seem to remove them.
Summary:
We have a large database that we are now ready to start trimming down to a more manageable size.
I deleted the documents, but I am left with a database it appears I can't copy or replicate, receiving error: Unable to extend an ID table - insufficient memory.
Detail:
Current live database has roughly 1,400,000 docs in it, we can immediately reduce it to 650,000. With some further design\ architectural work we can reduce this to 300,000. This web application is clustered on 5 servers, 1 being the application "hub", with 3 HTTP\SSO servers behind a reverse proxy and the 5th server is the access point for external services. The HUB initiates a PUSH\PULL replication (with no document restrictions) every 30mins, logs show no replication errors. Transaction logging is set for Run Time performance on servers, back up is performed on another back server not in cluster. We run daily archiving that archives roughly 1,500 docs a day.
So far, nearly 2 years on, application has run fine...apart from a few nightmares here and there :-). But, we are finally at point we can trim the databases down.
Prior to any work I OS backed up the live DB, and copied it to a test server. On a test server I did a: CL copy mylivedb.nsf mycleandb.nsf
In mycleandb.nsf I then deleted docs to reduce the DB to 650,000 documents.
All I'm after is a new copy, not a replica, see replica issue below, so I issued another command...to give me a new copy without deletion stubs, it's my understanding that a COPY does not copy del' stubs.:
CL copy mycleandb.nsf mycleandbNEW.nsf
...this also ensures that if any rogue replica is out on the network, we don't get allt he docs back in (not all our servers are 8.5.3, so we can't use the Database Property to set a cut off date for deletion stubs)
But I got a: Unable to extend an ID table - insufficient memory.
Tried this:
CL copy mycleandb.nsf mycleandbNEW.nsf REPLICA
...same thing.
Tried steps found here to my mycleandb.nsf:
http://www-01.ibm.com/support/docview.wss?uid=swg21220384
...same thing.
Changed purge interval to 0 days as described here, also set purge date into future:
http://www-01.ibm.com/support/docview.wss?uid=swg21095683
...same thing.
Then ran a:
load compact mycleandb.nsf -B
...same thing.
I have seen this thread here, which is a similar situation, except, we don't have any issues with clustering\ replication...yet!
http://www-10.lotus.com/ldd/nd85forum.nsf/DateAllFlatWeb/74d3e0f5467f843885257aaf0081abe5?OpenDocument
So, I have a DB I have deleted docs from, I can open it, it appears to function, but I can't copy\ replicate it.
Replication\ Cluster Issue:
I have seen this error before: Unable to extend an ID table - insufficient memory...and what I have done, is drop one of our primary servers, OS copy the DB to server with issue, restart.
I have always created new replicas using Admin Client\ Dom console, but in this case, whenever I do it floods the database with documents, but purge intervals are correct on all servers. The new replicas have been created from the HUB server, which initiates PUSH\PULL replication with all the cluster mates every 30mins, so if this was a deletion stub issue, it would manifest itself all the time?
I will NotesPeek it tomorrow to see if any deletion stubs are still there, but I am not sure how to proceed.
Any comments or suggestions gratefully received.

Answer 1

In my experience, the code in Domino that actually purges deletion stubs cannot work when the total number of modified docs + stubs is too high. I think it's probably a limitation that's pretty deep in the Notes API internals, combined with the algorithm that has to be used -- which is to look at all notes modified before the purge interval date -- a potentially very large number.
Your best option may be to make a local non-replica copy and re-deploy.
I researched this a while back. I found some LotusScript code that uses Notes C API calls to purge deletion stubs posted on various blogs. (I think the original code may have come from Rod Whitely, but I'm not sure.) My version code is below.
When run on a database where the total of docs + stubs is somewhere between 2 and 3 million, it gets the "Unable to extend an ID table - insufficient memory". I never did contact IBM support about this, as it was really just a side project for me. I ended up just making non-replica copies of the production databases and then setting the purge interval low enough so that the number of stubs never got too high again.
Declare Private Sub IDDestroyTable Lib wAPIModule Alias "IDDestroyTable" _
( ByVal hT As Long)
Declare Private Function IDScan Lib wAPIModule Alias "IDScan" _
( ByVal hT As Long, ByVal F As Integer, ID As Long) As Integer
Declare Private Function NSFDbOpen Lib wAPIModule Alias "NSFDbOpen" _
( ByVal P As String, hDB As Long) As Integer
Declare Private Function NSFDbClose Lib wAPIModule Alias "NSFDbClose" _
( ByVal hDB As Long) As Integer
Declare Private Function NSFDbGetModifiedNoteTable Lib wAPIModule Alias "NSFDbGetModifiedNoteTable" _
( ByVal hDB As Long, ByVal C As Integer, ByVal S As Currency, U As Currency, hT As Long) As Integer
Declare Private Function NSFNoteDelete Lib wAPIModule Alias "NSFNoteDelete" _
( ByVal hDB As Long, ByVal N As Long, ByVal F As Integer) As Integer
Declare Private Function OSPathNetConstruct Lib wAPIModule Alias "OSPathNetConstruct" _
( ByVal NullPort As Long, ByVal Server As String, ByVal FIle As String, ByVal PathNet As String) As Integer
Declare Private Sub TimeConstant Lib wAPIModule Alias "TimeConstant" _
( ByVal C As Integer, T As Currency)
Function countAndDeleteStubsByOpenDatabase(db As NotesDatabase, deleteFlag As boolean) As Long
If db Is Nothing GoTo bail
Dim forever As Currency
Dim last As Currency
Dim hT As Long
Dim RRV As Long
Dim hDB As Long
Dim path As String
Dim nStubs As Long
Dim ret As integer
On Error GoTo bail
path = Space(1024)
Call OSPathNetConstruct(0, db.Server, db.FilePath, path)
Call NSFDbOpen(path, hDB)
Call TimeConstant(2, forever)
Call NSFDbGetModifiedNoteTable(hDB, &H7FFF, forever, last, hT)
nStubs = 0
ret = IDScan(hT, True, RRV)
While Not (ret = 0)
If RRV < 0 Then
If (deleteFlag = true) Then
NSFNoteDelete hDB, RRV And &H7FFFFFFF, &H0201
End If
nStubs = nStubs + 1
End If
ret = IDScan(hT, False, RRV)
Wend
IDDestroyTable hT
NSFDbClose hDB
If deleteFlag = True Then
Print "Deleted " + CStr(nStubs) + " stubs for " + db.Filepath + " on " + db.Server
Else
Print "Counted " + CStr(nStubs) + " stubs for " + db.Filepath + " on " + db.Server
End If
countAndDeleteStubsByOpenDatabase = nStubs
On Error GoTo 0
Exit Function
bail:
Print "Unexpected error in countAndDeleteStubsByOpenDatabase. Line " + CStr(Erl) + " " + CStr(Err()) + " " + Error(Err())
countAndDeleteStubsByOpenDatabase = 0
End Function
I never tried it, but it also occurred to me that it might be possible to modify this code to just search for the most recent one day's worth of stubs, delete them, then go back one more day and get the most recent stubs, delete those, etc. It might not be all that easy to do in LotusScript, though, given that you have to deal with the C TIMEDATE struc as a Currency field. You might have to do it in C. Of course, deleting the newest stubs first is the exact reverse of what you want to do in practice, but this strategy might work around the limitation on idtables.

Answer 2

http://www.ytria.com/ has a tool ScanEZ that has a function to delete deletion stubs comfortably. I'm not sure if it would help if you're hitting against internal Notes limits as Richard suggested, but it might be worth a try.