Download file after authenticated through WebBrowser - vb.net

Question

Background
So I am creating a VB .NET program that will basically login to a web page, do a bunch of button clicking, which results in the website generating an excel report that can be downloaded. I have successfully gone through all the steps to produce the file so now I am trying to create a method that will download the file behind the scenes without the "Save as" dialog appearing.
Details
I have managed to trap the download through the Navigating event of the Webbrowser control:
Public Sub a(sender As Object, e As WebBrowserNavigatingEventArgs) Handles WebBrowser1.Navigating
'intercept the excel download. Retrieve the url but cancel dialog
If e.Url.AbsoluteUri.Contains("fmsdownload") Then
Label3.Text = e.Url.AbsoluteUri
e.Cancel = True
'e.Url.AbsoluteUri = the temporarily generated file URL to download from
'INSERT DOWNLOAD METHOD HERE
End If
End Sub
I verified that the e.Url.AbsoluteUri is indeed the correct path. If I copy / paste this URL into Chrome, it downloads.
Question
So ultimately I am simply trying to find a way to download the file after the download link has been generated. Please take a look in the section below for what I have tried as I believe I am close to achieving success.
What I have tried (Please read before posting)
Method 1: My.Computer.Network.DownloadFile(URL,SAVEPATH). This results in the server kicking back The repote server returned an error: (403) Forbidden. This leads me to understand that the authentication isn't being passed which makes sense.
Method 2: I read on a stackoverflow post to try the URLMON to initiate the download (http://www.pinvoke.net/default.aspx/urlmon/URLDownloadToFile%20.html). I thought that this would have some promise but results in the error Unable to find an entry point named URLDownloadToFile in DLL 'URLMON.dll' Here is the code I have used for this method as it may be something simple I am missing:
Private Declare Sub URLDownloadToFile _
Lib "URLMON.dll" (
ByVal lpCaller As Long,
ByVal szUrl As String,
ByVal szFilename As String,
ByVal dwReserved As Long,
ByVal lpBindStatusCallback As Long)
Public Sub a(sender As Object, e As WebBrowserNavigatingEventArgs) Handles WebBrowser1.Navigating
'intercept the excel download. Retrieve the url but cancel dialog
If e.Url.AbsoluteUri.Contains("fmsdownload") Then
Label3.Text = e.Url.AbsoluteUri
e.Cancel = True
Try
Kill(My.Computer.FileSystem.SpecialDirectories.MyDocuments & "\download.xls")
Catch
End Try
URLDownloadToFile(0, e.Url.AbsoluteUri, My.Computer.FileSystem.SpecialDirectories.MyDocuments & "\download.xls", 0, 0)
End If
End Sub
Method 3: After some research it seems like the authentication is stored as cookies so I tried to retrieve the cookies and then provide them back to the WebClient since WebClient supports downloading files. Here is where I capture the cookies:
Dim cookie_collection() As String
Public Sub webbrowser1_documentcompleted(sender As Object, e As
WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted
If WebBrowser1.Document.Cookie Is Nothing Then
Else
Dim cookies As String() = WebBrowser1.Document.Cookie.Split({";"c}, StringSplitOptions.None)
For Each cookie As String In cookies
Dim name As String = cookie.Substring(0, cookie.IndexOf("="c)).TrimStart(" "c)
Dim value As String = cookie.Substring(cookie.IndexOf("="c) + 1)
If cookie_collection Is Nothing Then
ReDim cookie_collection(0)
Else
ReDim Preserve cookie_collection(cookie_collection.Length)
End If
cookie_collection(cookie_collection.Length - 1) = cookie
' MsgBox(cookie)
Next cookie
End If
End Sub
I verified that two cookies are captured during the authentication process:
So I try to reapply the cookies to my WebClient before downloading:
Public Sub a(sender As Object, e As WebBrowserNavigatingEventArgs) Handles WebBrowser1.Navigating
'intercept the excel download. Retrieve the url but cancel dialog
If e.Url.AbsoluteUri.Contains("fmsdownload") Then
Label3.Text = e.Url.AbsoluteUri
e.Cancel = True
Dim client As New System.Net.WebClient
For Each cookie As String In cookie_collection
client.Headers.Add(Net.HttpRequestHeader.Cookie, cookie)
Next
client.DownloadFile(e.Url.AbsoluteUri, My.Computer.FileSystem.SpecialDirectories.MyDocuments & "\download.xls")
End If
End Sub
This unfortunately results in the same error as before The repote server returned an error: (403) Forbidden which makes me realize the authentication isn't being passed still.
I know this is a big post but I feel like method 2 or method 3 should work so it may be possible I am missing something small (I hope).

Answer 1

In your last bit of code, where you use WebClient to retrieve the file, you currently add the cookies to the request as follows:
For Each cookie As String In cookie_collection
client.Headers.Add(Net.HttpRequestHeader.Cookie, cookie)
Next
However, there is only a single parameter in HTTP that includes all cookies at once, therefore you were overriding the previous cookies by pushing a new header for each cookie. You therefore do not need to extract the cookies by using String.Split, but rather pass the parameter with semicolons as it is, since HTTP should work everything out. Hope this solves your issue!
Edit: Here is what I think might work out:
Retrieve the cookies in the DocumentCompleted event like this (third bit of code):
Dim cookie_collection As String
Public Sub webbrowser1_documentcompleted(sender As Object, e As WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted
If WebBrowser1.Document.Cookie IsNot Nothing Then
Dim cookies As String = WebBrowser1.Document.Cookie
If cookie_collection = "" Then
cookie_collection = cookies
Else
cookie_collection &= ";" & cookies
End If
End If
End Sub
Then proceed by simply passing the cookie_collection field to the header:
...
Dim client As New System.Net.WebClient
client.Headers.Add(Net.HttpRequestHeader.Cookie, cookie_collection)
client.DownloadFile(e.Url.AbsoluteUri, My.Computer.FileSystem.SpecialDirectories.MyDocuments & "\download.xls")
...