How can I properly implement threaded downloads using WebClient with a BlockingCollection? - vb.net

Question

I'm attempting to make a multi-threaded download manager that has a limit of 4 concurrent downloads. In my research, I came across the following: C# Downloader: should I use Threads, BackgroundWorker or ThreadPool?
[edit] updated code:
Imports System.Net
Imports System.Collections.Concurrent
Imports System.ComponentModel
Imports System.Threading
Public Class Form1
Const MaxClients As Integer = 4
' create a queue that allows the max items
Dim ClientQueue As New BlockingCollection(Of WebClient)(MaxClients)
' queue of urls to be downloaded (unbounded)
Dim UrlQueue As New Queue(Of String)()
Dim downloadThread As Thread
'Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
' create four WebClient instances and put them into the queue
For i As Integer = 0 To MaxClients - 1
Dim cli = New WebClient()
AddHandler cli.DownloadFileCompleted, AddressOf DownloadFileCompleted
AddHandler cli.DownloadProgressChanged, AddressOf DownloadProgressChanged
ClientQueue.Add(cli)
Next
' Fill the UrlQueue here
UrlQueue.Enqueue("http://www.gnu.org/licenses/gpl-1.0.txt")
UrlQueue.Enqueue("http://www.gnu.org/licenses/gpl-2.0.txt")
UrlQueue.Enqueue("http://www.gnu.org/licenses/gpl-3.0.txt")
UrlQueue.Enqueue("http://www.gnu.org/licenses/lgpl-2.1.txt")
UrlQueue.Enqueue("http://www.gnu.org/licenses/lgpl-3.0.txt")
UrlQueue.Enqueue("http://www.gnu.org/licenses/fdl-1.1.txt")
UrlQueue.Enqueue("http://www.gnu.org/licenses/fdl-1.2.txt")
UrlQueue.Enqueue("http://www.gnu.org/licenses/fdl-1.3.txt")
downloadThread = New Thread(AddressOf downloadQueue)
downloadThread.IsBackground = True
downloadThread.Start()
End Sub
Private Sub downloadQueue()
' Now go until the UrlQueue is empty
While UrlQueue.Count > 0
Dim cli As WebClient = ClientQueue.Take() ' blocks if there is no client available
Dim url As String = UrlQueue.Dequeue()
Dim fname As String = CreateOutputFilename(url)
cli.DownloadFileAsync(New Uri(url), fname, New DownloadArgs(url, fname, cli))
AppendText(url & " started" & vbCrLf)
End While
End Sub
Private Sub DownloadProgressChanged(sender As Object, e As DownloadProgressChangedEventArgs)
Dim args As DownloadArgs = DirectCast(e.UserState, DownloadArgs)
' Do status updates for this download
End Sub
Private Sub DownloadFileCompleted(sender As Object, e As AsyncCompletedEventArgs)
Dim args As DownloadArgs = DirectCast(e.UserState, DownloadArgs)
' do whatever UI updates
Dim url As String = "Filename" '<============I'd like to be able to pass the filename or URL but can't figure this out
AppendText(url & " completed" & vbCrLf)
' now put this client back into the queue
ClientQueue.Add(args.Client)
End Sub
Public Function CreateOutputFilename(ByVal url As String) As String
Try
Return url.Substring(url.LastIndexOf("/") + 1)
Catch ex As Exception
Return url
End Try
End Function
Private Delegate Sub SetTextCallback(text As String)
Private Sub AppendText(text As String)
If Me.TextBox1.InvokeRequired Then
TextBox1.Invoke(New Action(Of String)(AddressOf AppendText), text)
Return
End If
Me.TextBox1.AppendText(text)
Me.TextBox1.SelectionStart = TextBox1.TextLength
Me.TextBox1.ScrollToCaret()
End Sub
End Class
Class DownloadArgs
Public ReadOnly Url As String
Public ReadOnly Filename As String
Public ReadOnly Client As WebClient
Public Sub New(u As String, f As String, c As WebClient)
Url = u
Filename = f
Client = c
End Sub
End Class
This will successfully download the first 4 files in the UrlQueue, but it then seems to freeze and no further files download. I'd imagine the problem lies in something minor I missed in the process of converting from C# to vb.net, but I can't seem to figure this out.

Answer 1

ClientQueue.Take() blocks the UI thread. Also, WebClient will want to raise the DownloadFileCompleted event on the UI thread - but it is already blocked by ClientQueue.Take(). You have a deadlock.
To resolve this, you got to move your blocking loop to another background thread.

Answer 2

You are blocking the ability for your async queue to process. Not sure this is the "Correct" way to do this but the changes here make it work:
While UrlQueue.Count > 0
Do While ClientQueue.Count = 0
Application.DoEvents()
Loop
Dim cli As WebClient = ClientQueue.Take() ' blocks if there is no client available
Dim url As String = UrlQueue.Dequeue()
Dim fname As String = CreateOutputFilename(url) ' or however you get the output file name
cli.DownloadFileAsync(New Uri(url), fname, New DownloadArgs(url, fname, cli))
End While