Wait for internet explorer to load everything? - vb.net

Question

I am scraping a webpage, and waiting for internet explorer to get done loading but for some reason it is not. I'm trying to get a value on the page but the wait part is not waiting therefore the value comes back blank when there should be a value. The IE page has done loading but the value for the elements on the page has not been loaded yet. Is there a way to wait for all elements to get done loading before proceeding to next line of code? Here's my code:
Dim IE As Object
Dim myvalue as string
IE = CreateObject("internetexplorer.application")
IE.navigate("mypage")
While Not IE.ReadyState = WebBrowserReadyState.Complete
Application.DoEvents()
End While
myValue = IE.document.getElementById("theValue").getAttribute("value")
Debug.Print(myValue)

Answer 1

You SHOULD NOT use Application.DoEvents() in order to keep your UI responsive! I really can't stress this enough! More than often using it is a bad hack which only creates more problems than it solves.
For more information please refer to: Keeping your UI Responsive and the Dangers of Application.DoEvents.
The correct way is to use the InternetExplorer.DocumentComplete event, which is raised when the page (or a sub-part of it, such an an iframe) is completely loaded. Here's a brief example of how you can use it:
Right-click your project in the Solution Explorer and press Add Reference...
Go to the COM tab, find the reference called Microsoft Internet Controls and press OK.
Import the SHDocVw namespace to the file where you are going to use this, and create a class-level WithEvents variable of type InternetExplorer so that you can subscribe to the event with the Handles clause.
And voila!
Imports SHDocVw
Public Class Form1
Dim WithEvents IE As New InternetExplorer
Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
IE.Navigate("http://www.google.com/")
End Sub
Private Sub IE_DocumentComplete(pDisp As Object, ByRef URL As Object) Handles IE.DocumentComplete
MessageBox.Show("Successfully navigated to: " & URL.ToString())
End Sub
End Class
Alternatively you can also subscribe to the event in-line using a lambda expression:
Imports SHDocVw
Public Class Form1
Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
Dim IE As New InternetExplorer
IE.Navigate("http://www.google.com/")
AddHandler IE.DocumentComplete, Sub(pDisp As Object, ByRef URL As Object)
MessageBox.Show("Successfully navigated to: " & URL.ToString())
End Sub
End Sub
End Class