Downloading files code gets unexpected 404 error - vb.net

Question

My code downloads files in loop but after the last file downloads it keeps downloading files that aren't there. Website shows redirect and 404 error.
I'm new with visual basic so I'm asking for help here.
My.Computer.Network.DownloadFile(strFullUrlDownload, strFullSavePath, False, 1000)
404 error
redirect
Public Class Form1
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim strMainUrl As String = "http://jixxer.com/123/"
Dim dt As DateTime = DateTime.Now
Dim dtDate As String = dt.ToString("yyyy-MM-dd")
Dim strSlash As String = "/"
Dim strPdf As String = "pdf"
Dim strDot As String = "."
Dim strPage As String = "page"
Dim strPageNbr As String = 1
Dim intCounter As Integer = 1
Dim strPageCounter As String = String.Format("{0:000}", intCounter)
Dim strSavePath As String = "D:\dls\title1\"
Dim strFullSavePath As String = strSavePath & strPageCounter & strDot & strPdf
Dim strFullUrlDownload As String = strMainUrl & dtDate & strSlash & strPdf & strSlash & strPage & strPageNbr & strDot & strPdf
Do Until strPageCounter = 200
' Downloads the resource with the specified URI to a local file.
My.Computer.Network.DownloadFile(strFullUrlDownload, strFullSavePath, False, 1000)
intCounter = intCounter + 1
strPageNbr = strPageNbr + 1
strPageCounter = String.Format("{0:000}", intCounter)
strFullSavePath = strSavePath & strPageCounter & strDot & strPdf
strFullUrlDownload = strMainUrl & dtDate & strSlash & strPdf & strSlash & strPage & strPageNbr & strDot & strPdf
Loop
End Sub
End Class
Try
'TRY to download the file using https first...
My.Computer.Network.DownloadFile(New Uri("https://" & ServerAddress & WebLogoPath & Convert.ToString(RowArray(0)) & ".png"), Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData) & "\" & AppDataFolder & PCLogoPath & Convert.ToString(RowArray(0)) & ".png", "", "", False, 500, True)
Catch ex_https As Exception
'Unable to locate file or write file
'If the operation timed out...
If (ex_https.Message = "The operation has timed out") Then
'Re-TRY to download the file using http instead, as a time out error may indicate that HTTPS is not supported.
Try
My.Computer.Network.DownloadFile(New Uri("http://" & ServerAddress & WebLogoPath & Convert.ToString(RowArray(0)) & ".png"), Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData) & "\" & AppDataFolder & PCLogoPath & Convert.ToString(RowArray(0)) & ".png", "", "", False, 500, True)
Catch ex_http As Exception
'Most likely, the file doesn't exist on the server. Either way, we cannot obtain the file so we need to perform the same action,
'which is handled outside of this Try block.
End Try
Else
'This is most likely a 404 error. Either way, we cannot obtain the file (and the connection is not timing out) - so
'we need to perform the same action, which is handled outside of this Try block.
End If
End Try
I just put the counter at 200 to test and make sure it works. But I know I need a way to quit on error but not sure how to code it yet. Appreciate any help.

Answer 1

If you don't know how many documents are stored in that remote directory, you have to handle the exception when a page is not found.
It's always possible to receive WebExceptions when a resource is requested from a site, so you should handle this case anyway.
I suggest to use the WebClient class directly instead of Network.DownloadFile(), which may be handy if you want to show a predefined UI of the progress (when it's possible), but using WebClient directly, lets you perform the download asynchrounously if you need it to, using the async/await pattern and the WebClient.DownloadFileTaskAsync() method.
Another suggestion: use a method to download those files, so you can call it from anywhere in your code. You can use a class or a module to store your methods, so you don't clutter your UI and you can also easily reuse these classes or modules in different projects, just including in a project the file that contains them.
Your code could be modified as follow (synchronous version):
You need to pass to the DownloadPdfPages method the remote base address: http://jixxer.com/123, the Path where the files are store (filesPath).
The third and fourth parameters are optional:
- If you don't specify a resourceName, Date.Now.ToString("yyyy-MM-dd") is assumed,
- If you don't specify a startPage, it will default to 1, converted in page1.pdf (the example here asks to start from page 3).
Note: I'm using String Interpolation here: $"page{startPage + pageCount}.pdf".
If your VB.Net version doesn't support it, use String.Format() instead.
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim numberOfPages = DownloadPdfPages("http://jixxer.com/123", "D:\dls\title1", "", 3)
If numberOfPages > 0 Then
MessageBox.Show($"Download completed. Number of pages: {numberOfPages}")
Else
MessageBox.Show("Download failed")
End If
End Sub
Private Function DownloadPdfPages(baseAddress As String, filesPath As String, Optional resourceName As String = "", Optional startPage As Integer = 1) As Integer
If String.IsNullOrEmpty(resourceName) Then resourceName = Date.Now.ToString("yyyy-MM-dd")
Dim resourceAddr = Path.Combine(baseAddress, resourceName, "pdf")
Dim pageCount = 0
Dim client = New WebClient()
Try
Do
Dim documentName = $"page{startPage + pageCount}.pdf"
Dim resourceUri = New Uri(Path.Combine(resourceAddr, documentName), UriKind.Absolute)
Dim fileName = Path.Combine(filesPath, documentName)
client.DownloadFile(resourceUri, fileName)
pageCount += 1
Loop
Catch ex As WebException
If ex.Response IsNot Nothing Then
Dim statusCode = DirectCast(ex.Response, HttpWebResponse).StatusCode
If statusCode = HttpStatusCode.NotFound Then
Return pageCount
End If
ElseIf ex.Status = WebExceptionStatus.ProtocolError AndAlso ex.Message.Contains("404") Then
Return pageCount
Else
' Log and/or ...
Throw
End If
Return 0
Finally
client.Dispose()
End Try
End Function
Asynchronous version, using the WebClient.DownloadFileTaskAsync() method.
Just a few changes ae necessary, note the Async keyword added to both the Button.Click handler and the DownloadPdfPagesAsync() method.
The Await keyword is then used to wait for a method to complete, without blocking the UI:
Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim numberOfPages = Await DownloadPdfPagesAsync("http://jixxer.com/123", "D:\dls\title1", "", 3)
If numberOfPages > 0 Then
MessageBox.Show($"Download completed. Number of pages: {numberOfPages}")
Else
MessageBox.Show("Download failed")
End If
End Sub
Private Async Function DownloadPdfPagesAsync(baseAddress As String, filesPath As String, Optional resourceName As String = "", Optional startPage As Integer = 1) As Task(Of Integer)
If String.IsNullOrEmpty(resourceName) Then resourceName = Date.Now.ToString("yyyy-MM-dd")
Dim resourceAddr = Path.Combine(baseAddress, resourceName, "pdf")
Dim pageCount = 0
Dim client = New WebClient()
Try
Do
Dim documentName = $"page{startPage + pageCount}.pdf"
Dim resourceUri = New Uri(Path.Combine(resourceAddr, documentName), UriKind.Absolute)
Dim fileName = Path.Combine(filesPath, documentName)
Await client.DownloadFileTaskAsync(resourceUri, fileName)
pageCount += 1
Loop
Catch ex As WebException
If ex.Response IsNot Nothing Then
Dim statusCode = DirectCast(ex.Response, HttpWebResponse).StatusCode
If statusCode = HttpStatusCode.NotFound Then
Return pageCount
End If
ElseIf ex.Status = WebExceptionStatus.ProtocolError AndAlso ex.Message.Contains("404") Then
Return pageCount
Else
' Log and/or ...
Throw
End If
Return 0
Finally
client.Dispose()
End Try
End Function