I have a problem insert mongodb using vb.net.
Public Sub runMongoDBQuery(ByVal nameDatabase As String, ByVal nameCollection As String, ByVal queryBson As BsonDocument, ByVal serverString As String)
Dim mongo1 As MongoServer = MongoServer.Create("mongodb://localhost/?safe=true")
mongo1.Connect()
Dim db = mongo1.GetDatabase(nameDatabase)
Dim collection1 = db(nameCollection)
collection1.Save(queryBson)// this is hang
mongo1.Disconnect()
End Sub
When I user this function, it run smoothly and suddenly hang.
It doesn't throw exception but hang.
The problems run very rarely.
All the function did is saving queryBson
I will:
Check content of queryBson
Check log of mongodb server
Update the question once I got more info
I managed to solve this problem by restarting mongodb and deleting the log. Thanks to Cirrus's comment.
The log is 3GB and I strongly suspected it's the culprit.
Such a high log is useless because text editor can't open it anyway. Deleting the log without restarting mongodb causes a side effect where logs are no longer created.
I will soon ask another question how to limit log for mongodb, maybe on the database forum.
This question has a sequel:
https://dba.stackexchange.com/questions/25658/how-to-keep-mongodb-log-small
Related
Objective : Create a simple VB.NET application to scan the GlobalCatalog with a basic filter, limited to predefined properties and write the results to a text file.
Method : Existing code below - this "works" but occasionally throws an exception : "System.DirectoryServices.SearchResultCollection.ResultsEnumerator.MoveNext() : More data is available"
Some browsing leads me to think (open to correction) that the issue is caused by attempting to retrieve large volumes of records (in my case roughly 400k) via the DirectorySearcher, despite the results being paginated, and that the solution may be to switch the existing System.DirectoryServices method for something utilising System.DirectoryServices.Protocols. See this SO thread leading to this article.
However, all the responses I've found, both the links above and others from extensive searching, only provide code snippets in C#, and only seem to query a single record (e.g. retrieve properties based on a specific distinguishedName or login)
I need to retrieve a tonne of records, as quickly and efficiently as possible, using VB.NET. I like the DirectoryServices method because it gives me an easy handle on the GlobalCatalog without having to provide domains or passwords - I can jump straight into the searcher and start specifying filters and properties. And it generally works - but I need it to work every time.
Can anybody advise how I might adapt this code to circumvent the occasional exception and pull back all the data I need in the best possible way?
Imports System.DirectoryServices
Public Sub ScanGlobalCatalog()
Dim searcher As DirectorySearcher = ActiveDirectory.Forest.GetCurrentForest.FindGlobalCatalog.GetDirectorySearcher
Try
With searcher
.Filter = "(&(|(objectClass=user)(objectClass=group))(proxyAddresses=*))"
.PageSize = 1000
.SearchScope = SearchScope.Subtree
.CacheResults = False
.PropertiesToLoad.Add("sAMAccountName")
.PropertiesToLoad.Add("distinguishedName")
.PropertiesToLoad.Add("displayName")
.PropertiesToLoad.Add("proxyAddresses")
End With
For Each result As SearchResult In searcher.FindAll()
Dim properties As ResultPropertyCollection = result.Properties
Dim sAMAccountName As ResultPropertyValueCollection = properties("sAMAccountName")
Dim distinguishedName As ResultPropertyValueCollection = properties("distinguishedName")
Dim displayName As ResultPropertyValueCollection = properties("displayName")
Dim proxyAddresses As ResultPropertyValueCollection = properties("proxyAddresses")
' Check / process / write each property to the output file...
Next
Catch ex As Exception
' Do something...
End Try
End Sub
Thanks Vesper!
Added as shown and it doesn't seem to be happening any more (I believe setting .SizeLimit to 0 equates to 'limitless' but again, open to correction from those with greater knowledge than I...)
With searcher
.Filter = "(&(|(objectClass=user)(objectClass=group))(proxyAddresses=*))"
.PageSize = 1000
.SizeLimit = 0
.SearchScope = SearchScope.Subtree
.CacheResults = False
.PropertiesToLoad.Add("sAMAccountName")
.PropertiesToLoad.Add("distinguishedName")
.PropertiesToLoad.Add("displayName")
.PropertiesToLoad.Add("proxyAddresses")
End With
Have been running the script as a service at 15 minute intervals for the last 20 hours or so and I can see 5 or 6 'fails' in the event log - however, previously this would have caused a fatal termination (the service would simply discontinue); now it simply reports the exception and tries again on the next iteration.
The fails are clumped together (consecutive 'runs' within the same hour / hour and a half) and the same service has run uninterrupted and error-free for around 15 hours now, leading me to suspect those fails may have coincided with some kind of maintenance being carried out on the server or some kind of record update mid-read which was throwing the exception. Again, would welcome any insight or opinions on this behaviour.
But I can live with 'occasional' exceptions like this as long as the script generally runs okay and doesn't fall over permanently if/when they arise.
Thanks again for the advice Vesper!
I use the IBM Host Access Class Library for COM Automation as a way to communicate with an IBM AS400 (aka iSeries, IBM i, green screen, 5250) through a terminal emulator. I notice that when you issue a "SendKeys" instruction, control returns to your application before the IBM emulator finishes with the command. This can lead to timing problems because you might then send another "SendKeys" instruction before the system is ready to accept it.
For example:
Imports AutPSTypeLibrary
Imports AutConnListTypeLibrary
Imports AutSessTypeLibrary
Sub Example
Dim connections As New AutConnList
connections.Refresh()
If connections.Count < 1 Then Throw New InvalidOperationException("No AS400 screen can currently be found.")
Dim connection As IAutConnInfo = DirectCast(connections(1), IAutConnInfo)
_Session = New AutSess2
_Session.SetConnectionByHandle(connection.Handle)
Dim _Presentation As AutPS = DirectCast(_Session.autECLPS, AutPS)
_Presentation.SendKeys("PM70[enter]", 22, 8)
_Presentation.SendKeys("ND71221AD[enter]", 22, 20)
End Sub
would work correctly when stepping through code in a debugger, but would fail when running normally because the second instruction was sent too soon.
One way to work with this is to put a timer or loop after each command to slow the calling program down. I consider this less than ideal because the length of time is not always predictable, you will often be waiting longer than necessary to accommodate an occasional hiccup. This slows down the run time of the entire process.
Another way to work around this is to wait until there is a testable condition on the screen as a result of your sent command. This will work sometimes, but some commands do not cause a screen change to test and if you are looking to abstract your command calling into a class or subroutine, you would have to pass in what screen condition to be watching for.
What I would like to find is one of the "Wait" methods that will work in the general case. Options like the autECLScreenDesc class seem like they have to be tailored to very specific conditions.
The autECLPS (aka AutPS) class has a number of Wait methods (Wait, WaitForCursor, WaitWhileCursor, WaitForString, WaitWhileString, WaitForStringInRect, WaitWhileStringInRect, WaitForAttrib, WaitWhileAttrib, WaitForScreen, WaitWhileScreen) but they also seem to be waiting for specific conditions and do not work for the general case. The general case it important to me because I am actually trying to write a general purpose field update subroutine that can be called from many places inside and outside of my .dll.
This example is written in VB.NET, but I would expect the same behavior from C#, C++, VB6, Java; really anything that uses IBM's Personal Communications for Windows, Version 6.0
Host Access Class Library.
The "Operator Information Area" class seems to provide a solution for this problem.
My general case seems to be working correctly with this implementation:
Friend Sub PutTextWithEnter(ByVal field As FieldDefinition, ByVal value As String)
If IsNothing(field) Then Throw New ArgumentNullException("field")
If IsNothing(value) Then Throw New ArgumentNullException("value")
_Presentation.SendKeys(Mid(value.Trim, 1, field.Length).PadRight(field.Length) & "[enter]", field.Row, field.Column)
WaitForEmulator(_Session.Handle)
End Sub
Private Sub WaitForEmulator(ByVal EmulatorHandle As Integer)
Dim Oia As New AutOIATypeLibrary.AutOIA
Oia.SetConnectionByHandle(EmulatorHandle)
Oia.WaitForInputReady()
Oia.WaitForAppAvailable()
End Sub
I give thanks to a user named "khieyzer" on this message board for pointing our this clean and general-purpose solution.
Edit:
After a few weeks debugging and working through timing and resource release issues, this method now reads like:
Private Sub WaitForEmulator(ByRef NeededReset As Boolean)
Dim Oia As New AutOIA
Oia.SetConnectionByHandle(_Presentation.Handle)
Dim inhibit As InhibitReason = Oia.InputInhibited
If inhibit = InhibitReason.pcOtherInhibit Then
_Presentation.SendKeys("[reset]")
NeededReset = True
WaitForEmulator(NeededReset)
Marshal.ReleaseComObject(Oia)
Exit Sub
End If
If Not Oia.WaitForInputReady(6000) Then
If Oia.InputInhibited = InhibitReason.pcOtherInhibit Then
_Presentation.SendKeys("[reset]")
NeededReset = True
WaitForEmulator(NeededReset)
Marshal.ReleaseComObject(Oia)
Exit Sub
Else
Marshal.ReleaseComObject(Oia)
Throw New InvalidOperationException("The system has stopped responding.")
End If
End If
Oia.WaitForInputReady()
Oia.WaitForAppAvailable()
Marshal.ReleaseComObject(Oia)
End Sub
I have the following code which I am using to populate a ImageList from a SQLite database with images stored as blobs.
Public Sub populateImagesStyles()
ShoeImages1.Images.Clear()
StyleImagesLView.Items.Clear()
Dim s As SQLiteDataReader
Dim rcount As Integer = 0
dbLocalQuery = New SQLiteCommand("SELECT id, image FROM tblImages", dbLocal)
s = dbLocalQuery.ExecuteReader()
While s.Read()
rcount += 1
ShoeImages1.Images.Add(CStr(s("id")), byte2img(s("image")))
StyleImagesLView.Items.Add(CStr(s("id")), CStr(s("id")))
End While
s.Close()
Here is the byte2img function...
Public Function byte2img(ByVal imgByte As Byte()) As Image
Dim imgMemoryStream As System.IO.MemoryStream = New System.IO.MemoryStream(imgByte)
byte2img = Drawing.Image.FromStream(imgMemoryStream)
End Function
The database contains over 250 images and this process is completed twice on load to populate two different ImageList, because I need the images displayed at two different sizes.
When the process runs on loading the form, it causes the process to consume between 800MB and 1GB of system memory, unless I manually run the process again from an form control, which seems to trigger garbage collection.
Stepping through the loading process, it is clear that it is the byte2img process that is causing the memory usage to escalate - what is the best way to mitigate this?
Also, if anyone can think of a more efficient process to execute this, i'm all ears. The images have to be stored in the database file because I need to be able to just package the .db file and send it to a remote location at a moments notice, so I can't mess with folders with images.
All help appreciated.
You are creating a lot of memory streams without disposing of them. Try this:
Public Function byte2img(ByVal imgByte As Byte()) As Image
Dim img As Image
Try
Using ms As New MemoryStream(imgbyte)
img = Drawing.Image.FromStream(ms)
End Using ' auto dispose of the MS
Catch ex As Exception
' report possibly bad/missing imgByte()
' resulting in an error in either place
End Try
Return img
End Function
An imprecise way to detect this kind of thing is to watch the HANDLES count in TaskManager.
Ok, I've found a solution/workaround that seems to work - call the PopulateImageStyles sub when a user visits the specific TabPage the ImageList resides on.
For some arbitrary reason, when run this way (as above, when called on the form), the process never proceeds to consume more than 50-60 MB of working memory.
I'll add a Background Worker so that the process can execute without hanging the form.
Update: I have issued a: show database mycleandb.nsf and I can see there are still 835,000 deletion stubs. I can't seem to remove them.
Summary:
We have a large database that we are now ready to start trimming down to a more manageable size.
I deleted the documents, but I am left with a database it appears I can't copy or replicate, receiving error: Unable to extend an ID table - insufficient memory.
Detail:
Current live database has roughly 1,400,000 docs in it, we can immediately reduce it to 650,000. With some further design\ architectural work we can reduce this to 300,000. This web application is clustered on 5 servers, 1 being the application "hub", with 3 HTTP\SSO servers behind a reverse proxy and the 5th server is the access point for external services. The HUB initiates a PUSH\PULL replication (with no document restrictions) every 30mins, logs show no replication errors. Transaction logging is set for Run Time performance on servers, back up is performed on another back server not in cluster. We run daily archiving that archives roughly 1,500 docs a day.
So far, nearly 2 years on, application has run fine...apart from a few nightmares here and there :-). But, we are finally at point we can trim the databases down.
Prior to any work I OS backed up the live DB, and copied it to a test server. On a test server I did a: CL copy mylivedb.nsf mycleandb.nsf
In mycleandb.nsf I then deleted docs to reduce the DB to 650,000 documents.
All I'm after is a new copy, not a replica, see replica issue below, so I issued another command...to give me a new copy without deletion stubs, it's my understanding that a COPY does not copy del' stubs.:
CL copy mycleandb.nsf mycleandbNEW.nsf
...this also ensures that if any rogue replica is out on the network, we don't get allt he docs back in (not all our servers are 8.5.3, so we can't use the Database Property to set a cut off date for deletion stubs)
But I got a: Unable to extend an ID table - insufficient memory.
Tried this:
CL copy mycleandb.nsf mycleandbNEW.nsf REPLICA
...same thing.
Tried steps found here to my mycleandb.nsf:
http://www-01.ibm.com/support/docview.wss?uid=swg21220384
...same thing.
Changed purge interval to 0 days as described here, also set purge date into future:
http://www-01.ibm.com/support/docview.wss?uid=swg21095683
...same thing.
Then ran a:
load compact mycleandb.nsf -B
...same thing.
I have seen this thread here, which is a similar situation, except, we don't have any issues with clustering\ replication...yet!
http://www-10.lotus.com/ldd/nd85forum.nsf/DateAllFlatWeb/74d3e0f5467f843885257aaf0081abe5?OpenDocument
So, I have a DB I have deleted docs from, I can open it, it appears to function, but I can't copy\ replicate it.
Replication\ Cluster Issue:
I have seen this error before: Unable to extend an ID table - insufficient memory...and what I have done, is drop one of our primary servers, OS copy the DB to server with issue, restart.
I have always created new replicas using Admin Client\ Dom console, but in this case, whenever I do it floods the database with documents, but purge intervals are correct on all servers. The new replicas have been created from the HUB server, which initiates PUSH\PULL replication with all the cluster mates every 30mins, so if this was a deletion stub issue, it would manifest itself all the time?
I will NotesPeek it tomorrow to see if any deletion stubs are still there, but I am not sure how to proceed.
Any comments or suggestions gratefully received.
In my experience, the code in Domino that actually purges deletion stubs cannot work when the total number of modified docs + stubs is too high. I think it's probably a limitation that's pretty deep in the Notes API internals, combined with the algorithm that has to be used -- which is to look at all notes modified before the purge interval date -- a potentially very large number.
Your best option may be to make a local non-replica copy and re-deploy.
I researched this a while back. I found some LotusScript code that uses Notes C API calls to purge deletion stubs posted on various blogs. (I think the original code may have come from Rod Whitely, but I'm not sure.) My version code is below.
When run on a database where the total of docs + stubs is somewhere between 2 and 3 million, it gets the "Unable to extend an ID table - insufficient memory". I never did contact IBM support about this, as it was really just a side project for me. I ended up just making non-replica copies of the production databases and then setting the purge interval low enough so that the number of stubs never got too high again.
Declare Private Sub IDDestroyTable Lib wAPIModule Alias "IDDestroyTable" _
( ByVal hT As Long)
Declare Private Function IDScan Lib wAPIModule Alias "IDScan" _
( ByVal hT As Long, ByVal F As Integer, ID As Long) As Integer
Declare Private Function NSFDbOpen Lib wAPIModule Alias "NSFDbOpen" _
( ByVal P As String, hDB As Long) As Integer
Declare Private Function NSFDbClose Lib wAPIModule Alias "NSFDbClose" _
( ByVal hDB As Long) As Integer
Declare Private Function NSFDbGetModifiedNoteTable Lib wAPIModule Alias "NSFDbGetModifiedNoteTable" _
( ByVal hDB As Long, ByVal C As Integer, ByVal S As Currency, U As Currency, hT As Long) As Integer
Declare Private Function NSFNoteDelete Lib wAPIModule Alias "NSFNoteDelete" _
( ByVal hDB As Long, ByVal N As Long, ByVal F As Integer) As Integer
Declare Private Function OSPathNetConstruct Lib wAPIModule Alias "OSPathNetConstruct" _
( ByVal NullPort As Long, ByVal Server As String, ByVal FIle As String, ByVal PathNet As String) As Integer
Declare Private Sub TimeConstant Lib wAPIModule Alias "TimeConstant" _
( ByVal C As Integer, T As Currency)
Function countAndDeleteStubsByOpenDatabase(db As NotesDatabase, deleteFlag As boolean) As Long
If db Is Nothing GoTo bail
Dim forever As Currency
Dim last As Currency
Dim hT As Long
Dim RRV As Long
Dim hDB As Long
Dim path As String
Dim nStubs As Long
Dim ret As integer
On Error GoTo bail
path = Space(1024)
Call OSPathNetConstruct(0, db.Server, db.FilePath, path)
Call NSFDbOpen(path, hDB)
Call TimeConstant(2, forever)
Call NSFDbGetModifiedNoteTable(hDB, &H7FFF, forever, last, hT)
nStubs = 0
ret = IDScan(hT, True, RRV)
While Not (ret = 0)
If RRV < 0 Then
If (deleteFlag = true) Then
NSFNoteDelete hDB, RRV And &H7FFFFFFF, &H0201
End If
nStubs = nStubs + 1
End If
ret = IDScan(hT, False, RRV)
Wend
IDDestroyTable hT
NSFDbClose hDB
If deleteFlag = True Then
Print "Deleted " + CStr(nStubs) + " stubs for " + db.Filepath + " on " + db.Server
Else
Print "Counted " + CStr(nStubs) + " stubs for " + db.Filepath + " on " + db.Server
End If
countAndDeleteStubsByOpenDatabase = nStubs
On Error GoTo 0
Exit Function
bail:
Print "Unexpected error in countAndDeleteStubsByOpenDatabase. Line " + CStr(Erl) + " " + CStr(Err()) + " " + Error(Err())
countAndDeleteStubsByOpenDatabase = 0
End Function
I never tried it, but it also occurred to me that it might be possible to modify this code to just search for the most recent one day's worth of stubs, delete them, then go back one more day and get the most recent stubs, delete those, etc. It might not be all that easy to do in LotusScript, though, given that you have to deal with the C TIMEDATE struc as a Currency field. You might have to do it in C. Of course, deleting the newest stubs first is the exact reverse of what you want to do in practice, but this strategy might work around the limitation on idtables.
http://www.ytria.com/ has a tool ScanEZ that has a function to delete deletion stubs comfortably. I'm not sure if it would help if you're hitting against internal Notes limits as Richard suggested, but it might be worth a try.
I'm having some timing problems in a unit test of some of my vb code.
I set up a test harness by checking for and then deleting records added to the db in the previous testing session, and then I test my recording adding code by adding back the same records.
Interestingly this code works fine when there is a break in the debugger, but fails with a "duplicate key" exception when I let it run with no breaks, which leads me to believe there is some kind of concurrency issue.
The basic metacode is as follows:
DoTest()
dim j as datacontext
dim recs = from myrecs in j.mythings select myrecs where myrecs.key="key1" or
myrecs.key = "key2"
if recs.count > 0
for each rec in myrecs
j.mythings.deleteonsubmit(rec)
next
j.submitchanges()
end if
j.dispose
dim tc as new tablecontroller()
tc.addrecordtomytable("key1","value1")
tc.addrecordtomytable("key2","value2")
end
Class tablecontroller
Sub addrecordstomytable(key as string, value as string)
dim j as new mydatacontext
dim thing as new mything
thing.key = key
thing.value = value
j.mythings.addonsubmit(thing)
j.submitchanges
j.dispose
end sub
end class
I've confirmed that I'm properly deleted the previous added records, and this works fine as does adding the new records when I have a break in the code before I hit the add records step. but without the break, it throws duplicate key exceptions in the "addrecordestomytable" method suggesting that it hasn't grabbed the current version of the table when it creates the new data context in addrecordstomytable, even though the records should have already been deleted.
I've tried refreshing the table, but this doesn't seem to work either.
Note backing database is ms sql server 10
Suggestions?
This is not so much an answer as a trouble-shooting technique. Have you tried using the log property? i.e.
j.log = Console.Out
So that you can see the actual SQL generated to make sure it is what you expect? Other than that is there anything relevant in your test setup or tear down? Is there anything managing a transaction? Are there triggers running? In terms of the latter if there is a trigger that takes some time to run and then the delete is finalized that might explain what you're seeing. I guess LINQ syntax varies between VB and C#, which surprised me, because I don't think your comparison code as written is valid in C#, you would need ==, not =, for a comparison but since it works when you break in the debugger...