I'm scraping twitter tweets, I launch multiple backgroundworkers and they do the following:
For x as Integer = 0 to 5
Dim BGW As New BackgroundWorker
AddHandler BGW.DoWork, AddressOf TweetGrab
BGW.RunWorkerAsync(tweeturl)
Next
Public TemporaryRows As New List(Of DataRow)
Private Sub TweetGrab(tweeturl as String)
'some html stuff here
Dim ImageUrl as String = twitterImage.Attributes("src").Value
Dim ThumbnailUrl As String = ImageUrl & ":small"
Dim DataRowTemporary As DataRow = DataTable1.NewRow()
DataRowTemporary("ImageUrl") = ImageUrl
DataRowTemporary("ThumbnailUrl") = ThumbnailUrl
DataRowTemporary("Checked") = False
'I detect the error even here
TemporaryRows.Add(DataRowTemporary)
End Sub
Later on, I do stuff with the TemporaryRows. I loop over the rows and check if they meet some conditions.
The problem is that DataRowTemporary("Checked") ends being DBNull and DataRowTemporary("ThumbnailUrl") is completely different than ImageUrl even though I specified Dim ThumbnailUrl As String = ImageUrl & ":small"
This happens in about 2/10 cases. I would guess it has something to do with background threads but I don't have any ideas how to solve it. I can reedit the fields after the error occurs, but I would like to prevent the error from occurring in the first place.
The problem was changing the collection when its being accessed by other threads (in parallel).
You can not do add/remove in parallel, the collection must be locked in order to not get strange errors.
SyncLock TemporaryRows
TemporaryRows 'Add/Remove
End SyncLock
A very simple issue im having trying to return a dataset from a VB.NET object function.
The following shows my function that is currently exiting from the function as soon as the SQL query is run and just before the new object connection is created.
The edit form is called here:
edit.Show()
Within the edit form, the following is run to to retrieve the details of the selected data in the database fro a retrieved datatset of the 'editEntry' method based on the ID set at the constructor.
Private Sub edit_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
Dim editDetails As New DBHandler(ID)
Dim returnedDetails As New DataSet
returnedDetails = editDetails.editEntry()
Dim nameReturned As Object = returnedDetails.Tables("editedTable").Rows(0)(1)
Dim firstNameEdit As String = nameReturned.ToString()
TextBox1.Text = firstNameEdit
This is the function where the problem is occuring. Nothing is being returned from the query
Constructor where the ID is set:
Public Sub New(ByVal ID As Integer)
IDofFault = ID
End Sub
The function of the class:
Public Function editEntry() As DataSet
Dim editDataSet As New DataSet
Dim editSql As String = "SELECT * FROM duraGadget WHERE _id = " + IDofFault + ""
'Exiting from the function here
Dim connectionEdit As New OleDbConnection(conString)
Dim editAdapter As New OleDbDataAdapter(editSql, connectionEdit)
connectionEdit.Open()
editAdapter.Fill(editDataSet, "editedTable")
connectionEdit.Close()
Return editDataSet
End Function
There is no error it simply exits from the function and im not sure why.
You could be receiving an exception and your visual studio debug settings are not configured to stop you on those types of exceptions.
Wrap the contents of the EditEntry function in a Try / Catch block, and put a break point inside the catch. See if that triggers and look at the exception details for more info on what occurred.
Very silly error this was guys. I simply stored the ID value as a sting...then tried to pass it as an Integer to the constructor...the result?.....Conversion exception.
I am trying to get a function for a class assignment to work however as the program hits the specific line in question it dies off and nothing after this line will execute. The program does not lock up, just the current execution path dies.
I have tried running debugging but much the same happens. Once I hit the link that should call a function from the object stored in the Arraylist element the break point at the actual function that should be called is not hit and nothing further happens.
Public Structure Appliances
' Create New Appliance object
Public Sub New(name As String, pusage As Double)
aName = name
aPUsage = pusage
End Sub
' Create New Washer Object
Public Sub New(name As String, pusage As Double, wusage As Double)
aName = name
aPUsage = pusage
aWUsage = wusage
End Sub
' Functions
Public Function getAName()
Return aName
End Function
Public Function getAPUsage()
Return aPUsage
End Function
Public Function getAWUsage()
Return aWUsage
End Function
Dim aName As String ' Appliance Name
Dim aPUsage As Double ' Appliane Power Usage
Dim aWUsage As Double ' Appliance Water Usage
End Structure
...
Public Class Form1
...
Dim appList As New ArrayList() ' Create an arraylist appliance objects
Public appTemp As Appliances ' To store appliance objects until they can be added to the arraylist
...
Private Function getAppInfo()
getAppInfo = Nothing
Do While fInStream.Peek() <> -1
s = fInStream.ReadLine() ' Get a line from the file and set s to it
Dim words As String() = s.Split(New Char() {","c}) ' Split the line contents along commas and set those parts into words
words(0) = words(0).Replace("_", " ") ' Reaplce underscores with spaces
If (words.Count = 3) Then ' If words contains the washer appliance
appTemp = New Appliances(words(0), Double.Parse(words(1)), Double.Parse(words(2)))
appList.Add(appTemp)
Else ' For all other appliances
appTemp = New Appliances(words(0), Double.Parse(words(1)))
appList.Add(appTemp)
End If
Loop
End Function
Private Function setUsage(name As String)
setUsage = Nothing
' Find appliance
For i = 0 To appList.Count
If (name = appList(i).getAName()) Then
If (name = "Washer") Then
s = appList(i).getWUsage() ' !!!This is the line where the execution dies at, nothing after this line is processed and the function call is not completed
txtbGPH.Text = s
End If
MsgBox("Test 1")
Exit For
ElseIf (i = appList.Count) Then
MsgBox("Appliance could not be found")
End If
Next
End Function
End Class
Use a List(Of X) instead of ArrayList if you are going to insert only one type:
Dim appList As New List(Of Appliances)
And I recommend you to declare your temp var inside the methods unless is necessary. Anyway, in this case you don't need it, you can add your var in this way:
appList.Add(New Appliances(words(0), Double.Parse(words(1))))
With this use (using lists) you won't need to use arraylistObj.Item(i).Method() and you can simply use the common way:
s = appList(i).getWUsage()
Nevermind, I figured it out just now. I did not know that arraylists are not "arraylists" but a collection. I thought maybe it would act like other collection oriented objects and that you have to use a .item(i) to access the elements, which turns out to be the case.
txtbGPH.text = appList.item(i).getAWusage()
produces the proper behavior and the rest of the code after the problematic line indicated in the OP executes as does the break point set at the called function.
I often need to make a large number of webrequests, without overloading the network
I currently do this by running synchronous requests in parallel, utilizing ThreadPool.SetMinThreads and MaxDegreeOfParallelism to exactly specify how many requests run concurrently
Now this works just fine, but it feels wrong.
I would really like to utilize async methods, but i cant work out how to limit the number of concurrent requests.
A simplified example of my parallel way of doing this( using a webclient and no error handling for brevity):
Private Function SearchSitesForKeywordInParallel(ByVal keyword As String, ByVal sites As String(), ByVal maxConcurrency As Integer) As String()
Dim po As New ParallelOptions
po.MaxDegreeOfParallelism = maxConcurrency
Threading.ThreadPool.SetMinThreads(maxConcurrency, 2)
Dim sitesContainingKeyword As New Concurrent.ConcurrentBag(Of String)
Parallel.For(0, sites.Count, po, Sub(i)
Dim wc As New Net.WebClient
wc.Proxy = Nothing
Dim pageSource As String = wc.DownloadString(sites(i))
If pageSource.Contains(keyword) Then
sitesContainingKeyword.Add(sites(i))
End If
End Sub)
Return sitesContainingKeyword.ToArray
End Function
This is a blocking function, which is what i require.
Now i have tested the webclient.downloadStringAsync method in a regular for loop, and it will fire all the requests pretty much at once, overloading the network.
What i would like to do is initially make X requests, then make new ones as each response comes back.
I am fairly sure tasks is the way to go, and im positive a have read some very nice implementations in c#, but my c# experience is limited, and i have a hard time translating c# lambadas to vb.net.
I am also limited to vs2010 and .net4, so the niceties of .net4.5 async await are not an option for me.
Any help very much appreciated
Not sure, if I understand completey, what exactly you want to achieve, but if you want to use aync methods, you can do it like this:
Dim google As String = "http://www.google.com/#&q="
Dim qsites As New Concurrent.ConcurrentQueue(Of String)
For Each k In {"foo", "bar", "john", "jack", "stackoverflow", "basic", "ship", "car", "42"}
qsites.Enqueue(google & k)
Next
Dim cde As New System.Threading.CountdownEvent(qsites.Count)
Dim strings As New Concurrent.ConcurrentBag(Of String)
Dim completedhandler = Sub(wco As Object, ev As Net.DownloadStringCompletedEventArgs)
Dim wc = DirectCast(wco, Net.WebClient)
Debug.Print("got one!")
strings.Add(ev.Result)
cde.Signal()
Dim s As String = String.Empty
If qsites.TryDequeue(s) Then
Debug.Print("downloading from {0}", s)
wc.DownloadStringAsync(New Uri(s))
End If
End Sub
Dim numthreads As Integer = 4
System.Threading.Tasks.Task.Factory.StartNew(Sub()
For i = 1 To numthreads
Dim s As String = String.Empty
If qsites.TryDequeue(s) Then
Dim wc As New Net.WebClient
wc.Proxy = Nothing
AddHandler wc.DownloadStringCompleted, completedhandler
Debug.Print("downloading from {0}", s)
wc.DownloadStringAsync(New Uri(s))
End If
Next
End Sub)
cde.Wait()
You only need to "start" the async downloads in a different thread/task because (afaik) the WC's downloadcompleted events fire in the UI thread (or currentsync..context) and the cde.wait would then not allow the events to be handled.
I just want to add another answer to this as I have recently solved a similar problem (note the code snippet is in C#, but should give the idea).
I used to have number of parallel http synchronous requests sent to http server on different thread and used to limit the number of requests I sent using semaphore.
Now, I have adapted to new TPL (c# 5.0 - aysn/await - quite handy (basically continuation introduced in TPL sound natural to me - and with async/await it has become much easier to use)), to invoke network I/O asynchronously.
i.e. ideally now I will be using only one thread in caller (unless I really need to get results before continuing), and let .net, os and I/o completion port threads work together to invoke my continuation code in thread pool to complete operation (basically 'callback' in APM, on completed event in event based pattern, 'continuation' in TPL, code after await in C# 5.0 (4.5 .net))
The principle I followed when I have embraced async i/o is simple - don't let thread wait and waste CPU and resources, unless it is really necessary!
You can do this asynchronously in VB.NET using the Wintellect Powerthreading library's AsyncEnumerator class, which you can get from NuGet.
This gives you some of the functionality of Await but works in VS2010 with .Net 2.0 to 4.0 while giving you an upgrade path to the 4.5 async features.
The downside is that the WebClient async methods require an EAP-to-APM shim based on Task<> to be used with AsyncEnumerator, so the code is quite a lot more complicated.
The simplest way to control the number of concurrent requests is to initiate X async operations, then just initiate another every time one completes.
Example code:
Imports System.Collections.Generic
Imports System.Runtime.CompilerServices
Imports System.Threading.Tasks
Imports System.Net
Imports Wintellect.Threading.AsyncProgModel
Module TaskExtension
REM http://msdn.microsoft.com/en-us/library/hh873178.aspx
<Extension()>
Public Function AsApm(Of T1)(ByVal task As Task(Of T1), callback As AsyncCallback, state As Object) As IAsyncResult
If (task Is Nothing) Then
Throw New ArgumentNullException("task")
End If
Dim tcs = New TaskCompletionSource(Of T1)(state)
task.ContinueWith(Sub(t As Task(Of T1))
If (t.IsFaulted) Then
tcs.TrySetException(t.Exception.InnerExceptions)
ElseIf t.IsCanceled Then
tcs.TrySetCanceled()
Else : tcs.TrySetResult(t.Result)
End If
If (Not callback Is Nothing) Then
callback(tcs.Task)
End If
End Sub, TaskScheduler.Default)
Return tcs.Task
End Function
End Module
Module ApmAsyncDownload
Public Function DownloadStringAsync(url As Uri) As Task(Of String)
Dim tcs As New TaskCompletionSource(Of String)
Dim wc As New WebClient()
AddHandler wc.DownloadStringCompleted, Sub(s As Object, e As System.Net.DownloadStringCompletedEventArgs)
If (Not (e.Error Is Nothing)) Then
tcs.TrySetException(e.Error)
ElseIf e.Cancelled Then
tcs.TrySetCanceled()
Else : tcs.TrySetResult(e.Result)
End If
End Sub
wc.DownloadStringAsync(url)
Return tcs.Task
End Function
Public Function BeginDownloadString(url As Uri, callback As AsyncCallback, state As Object) As IAsyncResult
Return DownloadStringAsync(url).AsApm(callback, state)
End Function
Public Function EndDownloadString(asyncResult As IAsyncResult) As String
Dim castToTask As Task(Of String) = asyncResult
Return castToTask.Result
End Function
End Module
Public Class AsyncIterators
Private Shared Iterator Function SearchUrl(ae As AsyncEnumerator(Of Boolean), keyword As String, uri As Uri) As IEnumerator(Of Int32)
ae.Result = False
ApmAsyncDownload.BeginDownloadString(uri, ae.End(0, AddressOf ApmAsyncDownload.EndDownloadString), Nothing)
Yield 1
If (ae.IsCanceled()) Then
Return
End If
Try
Dim page As String = ApmAsyncDownload.EndDownloadString(ae.DequeueAsyncResult)
ae.Result = page.Contains(keyword)
Catch ex As AggregateException
End Try
End Function
Public Shared Iterator Function SearchIterator(ae As AsyncEnumerator(Of List(Of String)), keyword As String, urls As List(Of Uri)) As IEnumerator(Of Int32)
ae.Result = New List(Of String)
'Control how many searches are started asynchonously
Dim startSearches = Math.Min(3, urls.Count)
Dim enumerator = urls.GetEnumerator
Dim toBeCompleted = urls.Count
Do Until (toBeCompleted <= 0)
While (startSearches > 0)
If enumerator.MoveNext Then
Dim subAe = New AsyncEnumerator(Of Boolean)()
subAe.SyncContext = Nothing
subAe.BeginExecute(SearchUrl(subAe, keyword, enumerator.Current), ae.End(0, Function(ar As IAsyncResult) As AsyncEnumerator.EndObjectXxx
subAe.EndExecute(ar)
End Function), enumerator.Current)
End If
startSearches = startSearches - 1
End While
'Wait for first async search to complete
Yield 1
toBeCompleted = toBeCompleted - 1
If (ae.IsCanceled()) Then
Exit Do
End If
'Get result of the search and add to results
Dim result = ae.DequeueAsyncResult()
Dim completedAe = AsyncEnumerator(Of Boolean).FromAsyncResult(result)
If (completedAe.EndExecute(result)) Then
Dim uri As Uri = result.AsyncState
ae.Result.Add(uri.OriginalString)
End If
'Start 1 more search
startSearches = startSearches + 1
Loop
End Function
End Class
Module Module1
Sub Main()
Dim searchAe = New AsyncEnumerator(Of List(Of String))()
searchAe.SyncContext = Nothing
Dim urlStrings = New List(Of String) From {"http://www.google.com", "http://www.yahoo.com", "http://www.dogpile.com"}
Dim uris = urlStrings.Select(Function(urlString As String) As Uri
Return New Uri(urlString)
End Function).ToList()
For Each Str As String In searchAe.EndExecute(searchAe.BeginExecute(AsyncIterators.SearchIterator(searchAe, "search", uris), Nothing, Nothing))
Console.WriteLine(Str)
Next
Console.ReadKey()
End Sub
End Module
And I now see what you mean about translating c# lambdas!
I am trying to return a binary from the DB using linq for display in the browser. The method below using ado.net works but I am trying to ypgrade to linq but the linq version returned the error.
Public Sub ProcessRequest(ByVal context As System.Web.HttpContext)
Dim imageId As String = context.Request.QueryString("id")
Dim ret As DataTable = Nothing
ret = DPGetImageData.GetImageById(Convert.ToInt64(imageId))
For Each dt As DataRow In ret.Rows
For Each c As DataColumn In ret.Columns
If Not (dt(c) Is Nothing) Then
context.Response.Clear()
context.Response.BufferOutput = False
context.Response.OutputStream.Write(CType(dt.Table.Rows(0).Item("imageData"), Byte()), 0, CInt(dt.Table.Rows(0).Item("imageSize")))
context.Response.End()
End If
Next
Next
End Sub
Working Linq Version:
Public Sub ProcessRequest(ByVal context As HttpContext) Implements IHttpHandler.ProcessRequest
If Not String.IsNullOrEmpty(Current.Request.QueryString("Id")) Then
Dim imageId = HttpContext.Current.Request.QueryString("Id")
Dim result = repository.FindById(Convert.ToInt64(imageId)).imageData.ToArray
HttpContext.Current.Response.BinaryWrite(result)
context.Response.End()
End If
End Sub
You should be able to just call Response.BinaryWrite(result.ToArray()). Notice the parantheses, ToArray is a method call.
You should not need the cast as Binary.ToArray, returns a byte array already.
An alternative is to use a byte array directly. You can modify it in the designer.
UPDATE:
I am not fully sure, but it could be that your DataContext has been disposed already. I think the Binary type uses deferred loading.
If you add a stacktrace, we could see if that is the case.