I am using a for/each iteration to get the last document in a mongo database. I pull this document every second and retrieve the 2nd element in the last document. However, my database is probably going to get quite large and I worry this is not a very effective method. Does anyone have any better ideas? I'm new to mongo and vb.net and just trying to learn and track my car at the same time.
Dim counter As Integer
For Each item As BsonDocument In drivingData.FindAll()
Next
counter += 1
If counter = drivingData.FindAll().Count - 1 Then
Dim dataString As String = item.GetElement(1).Value.ToString()...
As was commented on, why not sort in reverse and limit your response document to 1.
Considering you are looking for the last document, even without another field such as a time to sort on the _id field will be returned in the order of insertion so you can reverse sort in this and the "last" document will then be the "first" result.
db.collection.find().sort({ _id: -1 }).limit(1)
Through trial and error I got the result. Thanks for your repsonses as they eventually got through my thick skull:
Dim drivingValues As BsonArray
Dim lastDoc As BsonDocument
lastDoc = drivingData.FindAll().SetSortOrder(SortBy.Descending("result")).Last
drivingValues = lastDoc.GetElement(1).Value
Related
so I'm using SQLite in a VB.net project with a populated database. I'm using the Microsoft.Data.Sqlite.Core and System.Data.SQLite NuGet package libraries. So the problem presents when I'm trying to get the result of a query. At first the SQLiteDataReader gets the response and all the elements of the desired table. I know this cause in the debugger I have a breakpoint after the setting the object and when I check the parameters of the SQLiteDataReader object the Result View shows all the elements of my table, but as soon as I remove the mouse from the object and check it again the Result View turns out empty, without even resuming with the next line of code. Does anyone know if its a known bug or something cause Ive used this exact method of querying a table in another project and it works.
The code:
Public Function RunQuery(com As String)
If CheckConnection() Then
command.CommandText = com
Dim response As SQLiteDataReader
response = command.ExecuteReader
Dim len As Integer = response.StepCount
Dim col As Integer = response.FieldCount
Dim resp(len, col) As Object
For i = 0 To col - 1
Using response
response.Read()
For j = 0 To len - 1
resp(i, j) = response.GetValue(j)
Next
End Using
Next
Debugger with populated result view
Debugger with empty result view
edit: added the for loop to show that its not only on the debugger that the result view is empty. When using response.Read() it throws an exception "System.InvalidOperationException: 'No current row'"
As I have told you in the comment, a DataReader derived class is a forward only retrieval object. This means that when you reach the end of the records returned by the query, that object is not capable to restart from the beginning.
So if you force the debugger to enumerate the view the reader reaches the end of the results and a second attempt to show the content of the reader fails.
The other part of your problem is caused by a misunderstanding on how to work on the reader. You should loop over the Read result not using a StepCount property. This property as far as I know, is not standard and other data providers don't support it. Probably it is present in the SQLite Provider because for them it is relatively easy to count the number of records while other providers don't attempt do calculate that value for performance reasons.
However, there are other ways to read from the reader. One of them is to fill a DataTable object with its Load method that conveniently take a DataReader
Dim data As DataTable = New DataTable()
Using response
data.Load(response)
End Using
' Now you have a datatable filled with your data.
' No need to have a dedicated bidimensional array
A DataTable is like an array where you have Rows and Columns instead of indexes to iterate over.
After using Mongoimport to import a CSV file to my database, I want to add a new field or element per document. And, the data per for this new field is the is the index number plus 2.
Dim documents = DB.GetCollection(Of BsonDocument)(collectionName).Find(filterSelectedDocuments).ToListAsync.Result
For Each doc in documents
DB.GetCollection(Of BsonDocument)(collectionName).UpdateOneAsync(
Builders(Of BsonDocument).Filter.Eq(Of ObjectId)("_id", doc.GetValue("_id").AsObjectId),
Builders(Of BsonDocument).Update.Set(Of Integer)("increment.value", documents.IndexOf(doc) + 2).Wait()
Next
If I have over a million of data to import, is there a better way to achieved this like using UpdateManyAsync?
Just as a side note: Since you've got the Wait() and the Result everywhere, the Async methods don't seem to make an awful lot of sense. Also, your logic appears flawed since there is no .Sort() anywhere. So you've got no guarantee about the order of your returned documents. Is it indended that every document just gets a kind of random but unique and increasing number assigned?
Anyway, to make this faster, you'd really want to patch your CSV file and write the increasing "increment.value" field straight into it before the import. This way, you've got your value directly in MongoDB and do not need to query and update the imported data again.
If this is not an option you could optimize your code like this:
Only retrieve the _id of your documents - that's all you need and it will majorly impact your .find() perfomance since a lot less data needs to be transferred/deserialized from MongoDB.
Iterate over the Enumerable of your result instead of using a fully populated list.
Use bulk writes to avoid connecting to MongoDB again and again for every document and use a chunked flushing approach and flush every 1000 documents or so.
Theoretically, you could go further using multithreading or yield semantics for nicer streaming. However, that's getting a little complicated and may not even be needed.
The following should get you going faster already:
' just some cached values
Dim filterDefinitionBuilder = Builders(Of BsonDocument).Filter
Dim updateDefinitionBuilder = Builders(Of BsonDocument).Update
Dim collection = DB.GetCollection(Of BsonDocument)(collectionName)
' load only _id field
Dim documentIds = collection.Find(filterSelectedDocuments).Project(Function(doc) doc.GetValue("_id")).ToEnumerable()
' bulk write buffer (pre-initialized to size 1000 to avoid memory traffic upon array expansion)
Dim updateModelsBuffer = new List(Of UpdateOneModel(Of BsonDocument))(1000)
' starting value for our update counter
Dim i As Long = 2
For Each objectId In documentIds
' for every document we want one update command...
' ...that finds exactly one document identified by its _id field
Dim filterDefinition = filterDefinitionBuilder.Eq(Of ObjectId)("_id", objectId)
' ...and updates the "increment.value" with our running counter
Dim updateDefinition = updateDefinitionBuilder.Set(Of Integer)("increment.value", i)
updateModelsBuffer.Add(New UpdateOneModel(Of BsonDocument)(filterDefinition, updateDefinition))
' every e.g. 1000 documents
If updateModelsBuffer.Count = 1000
' we flush the contents to the database
collection.BulkWrite(updateModelsBuffer)
' and we empty our buffer list
updateModelsBuffer.Clear()
End If
i = i + 1
Next
' flush left over commands that have not been written yet in case we do not have a multiple of 1000 documents
collection.BulkWrite(updateModelsBuffer)
I have a VB.NET application that I use to load various files into a RichTextBox and then scan through the document to find specific words. It's similar to the Find function in Word. The app was running fine until a 5,150 line .sql document run through it and it's taking upwards of 10 minutes to run to completion.
Can anyone recommend a better way of coding it than I have below?
If sqlText.Contains("GRANT") Then
Dim searchstring As String = "GRANT"
Dim count As New List(Of Integer)()
For i As Integer = 0 To rtbFile.Text.Length - 1
If rtbFile.Text.IndexOf(searchstring, i) <> -1 Then
count.Add(rtbFile.Text.IndexOf(searchstring, i))
End If
Next
Try
For i As Integer = 0 To count.Count - 1
rtbFile.Select(count(i), searchstring.Length)
rtbFile.SelectionBackColor = Color.Yellow
rtbFile.SelectionFont = New Font(rtbFile.Font, FontStyle.Bold)
count.RemoveAt(i)
Next
Catch ex As Exception
End Try
rtbFile.Select(rtbFile.Text.Length, 0)
rtbFile.SelectionBackColor = Color.White
rtbFile.SelectionFont = New Font(rtbFile.Font, FontStyle.Regular)
End If
That first loop is killing the performance, you are calling IndexOf for every character in the string. Also the two loops can be merged in to one. Change it to:
rtbFile.SelectionBackColor = Color.Yellow
rtbFile.SelectionFont = New Font(rtbFile.Font, FontStyle.Bold)
For Each m As Match in Regex.Matches(sertbFile.Text, searchstring)
rtbFile.Select(m.Index, searchstring.Length)
Next
This can also be done with a While loop and RichTextBox.Find():
Dim searchstring As String = "GRANT"
Dim index As Integer = rtbFile.Find(searchstring, 0, RichTextBoxFinds.None)
While index <> -1
rtbFile.Select(index, searchstring.Length)
rtbFile.SelectionBackColor = Color.Yellow
rtbFile.SelectionFont = New Font(rtbFile.Font, FontStyle.Bold)
index = rtbFile.Find(searchstring, index + searchstring.Length, RichTextBoxFinds.None)
End While
You've got a few bad things going on here:
First, the following code:
For i As Integer = 0 To rtbFile.Text.Length - 1
If rtbFile.Text.IndexOf(searchstring, i) <> -1 Then
count.Add(rtbFile.Text.IndexOf(searchstring, i))
End If
Next
This is looping through every character in your string, and calling IndexOf on the entire string from that point forward. So your 50,000-character string is running IndexOf 50,000 times, on large strings.
You only need to call IndexOf as many times as you find a string. When your string is found, you increment your start index to that point, and keep searching only from that point.
Next thing, this code:
For i As Integer = 0 To count.Count - 1
...
count.RemoveAt(i)
Next
The RemoveAt line is unnecessary. You're already looping through a list, so you don't need to remove the items as you go along. The way it stands, your loop will skip every other item in your list.
Whoops. I missed a very important point about the IndexOf (and incorrectly assumed it was fed with the end of the last match). See Magnus's answer.
I am not sure where the bottleneck is (and it might very well be from setting the selection itself), but here are my suggestions, roughly in order of priority:
Invoke rtbFile.Text once to avoid any roundtrips to underlying control (perhaps a native Windows control?) and use a variable to store the resulting string. Once the string is obtained in .NET, just keep using it directly unless/until the text may change. If the control is native then a lot of work may be required to simply "get the text".
Use normal item iteration over the count collection (not indexing) and do not remove from the front-of the List when assigning the selections. Removing from the front of a List is "expensive" in that it must shift all items down internally. Also, removing the element is unneeded here and dubious at best: since the collection being modified is also is being iterated which likely leads to incorrect behavior (skipped items), regardless of performance.
Only call IndexOf once per loop and use a variable to avoid a duplicate search. This likely won't have an overall impact, but it does avoid some "extra" work. IndexOf itself is fine and doesn't need to be replaced.
YMMV.
For each connection in an array called ALLconn, I would like to compare it to my sql table. If exist, then add to my listview. Here is my code below, but it does not seem to work:
Dim LoginFilter As Object
Dim SelCurrAllSessions As SqlClient.SqlCommand = New SqlClient.SqlCommand("Select * from CurrAllSessions", LFcnn)
SelCurrAllSessions.CommandType = CommandType.Text
LoginFilter = SelCurrAllSessions.ExecuteReader
For Each conn In AllConn
While LoginFilter.Read()
If conn.UserName.ToString() = LoginFilter.Item(0) Then
ListBox1.Items.Add(LoginFilter.Item(0))
End If
End While
Next
Well you need to change the order of the loops
While LoginFilter.Read()
For Each conn In AllConn
If conn.UserName.ToString() = LoginFilter.Item(0).ToString Then
ListBox1.Items.Add(LoginFilter.Item(0).ToString)
Exit For
End If
Next
End While
This is necessary because in your original code, the internal while run till the end of the data loaded from the database then, when you try to check the next conn, you cannot reposition the reader at the start of the data loaded by the database.
It's the other way around, use Contains to check if the string is contained in the collection, then you can add it to the ListBox:
Using LoginFilter = SelCurrAllSessions.ExecuteReader()
While LoginFilter.Read()
Dim connection = LoginFilter.GetString(0)
If AllConn.Contains(connection) Then
ListBox1.Items.Add(connection)
End If
End While
End Using
Actually, your Question is uncompleted but still as per my understanding your trying to read result of SqlCommand.ExecuteReader(). When you read that Result it will reading from first to last and that is only one time you can read. if you try to read again it will show up an error because no more content to read from an object Loginfilter.
So, Either you can store that reading into an array and continue with your Foreach and while logic with newly created array or you can do reading from LoginFilter and compare into forech connections in AllConn array or List.
Please feel free to response me if you need more explaination, I will send you in C# version.
tl;dr Method of copying contents of an SQL query to an array or a string or a listbox. Single columns, multiple rows.
Working on a small project for some extra course credit.
Developing it currently in Visual Studio 2010.
Essentially is an interactive menu where users select items and can add them to an inbuilt
list and it will calculate total nutritional information and costs etc..
I'm having an issue however. When the user reaches the order builder page they can select the type of item they wish to buy.
E.G.
Beef
Clicking this should then populate a list box with all the related items.
I'm hoping to do this via a database connection.
I currently have an embedded database.
Their where 2 ways I've tried to do this but both proved unsuccessful or perhaps I'm just doing it wrong.
First method.
Dim index As Integer = 0
Dim length As Integer = adapter.productscounter()
' Small query that works out total number of rows.
For index = 0 To length
ListBox1.Items.Add(adapter.SelectBeef(index))
Next
This gives me the error:
There is no row at position 0.
which I seem unable to solve. The query runs upon trial execution and theirs something their.
Index out of range exception
The other method I was attempting was similar code however using an array and then copying the contents of that into the listbox.
Dim index As Integer
Dim test(5)
Dim length As Integer = adapter.productscounter()
Dim counter As Integer
For index = 0 To length
test(index) = adapter.SelectChicken()
counter = counter + 1
Next
For counter = 0 To length
ListBox1.Items.Add(test(index))
Next
Generates:
Argument nullexception
Value cannot be null.
Parameter name: item.
In Visual Basic, the standard is to start a list with element 1, not element 0. You could try:
For index = 1 To length
ListBox1.Items.Add(adapter.SelectBeef(index))
Next
Though of course, I have no insight in what the SelectBeef method does.