Retrieve Data from Website with VBA Excel - vba

Question

I understand there have been answered for similar questions but I am not sure if I could not understand how to approach to the solutions from other people' answers or my the website I need to get the information from is complex. So, please help me.
I would like to get the description field from Delphi for PN#13511996, the value should be "3 Way Gray GT 150 Sealed Female Connector Assembly, Max Current 15 amps" . Could someone help me examine the website and let me know how to get the description?
Sub GetData()
'Added Microsoft HTML Object library to reff
'Added Microsoft XML, v6.0 to reff
Dim xhr As MSXML2.XMLHTTP60
Dim doc As MSHTML.HTMLDocument
Dim desc As String
Set xhr = New MSXML2.XMLHTTP60
With xhr
.Open "GET", "http://ecat.delphi.com/feature?search=13511996", False
.send
If .ReadyState = 4 And .Status = 200 Then
Set doc = New MSHTML.HTMLDocument
doc.body.innerHTML = .responseText
End If
End With
With doc
desc = .getElementsByClassName("ProductDetail.Description").Item(0).innerText
End With
Debug.Print desc
End Sub

Answer 1

This is because you are requesting raw HTML by using GET from XMLHTTP. If you try to Debug.Print doc.body.innerHTML, you will see that the table has not been generated yet, and the text you are looking for is not there at all.
To be able to run the query for item "13511996", you need a real browser. Only then you can generate your table and get DOM document object. Try the following code:
Sub GetData()
Dim aIE As InternetExplorer
Dim desc As IHTMLElement
Set aIE = New InternetExplorer
With aIE
.navigate "http://ecat.delphi.com/feature?search=13511996"
.Visible = True '----> set it to false if you dont want to see the browser
End With
Do While (aIE.Busy Or aIE.ReadyState <> READYSTATE_COMPLETE)
DoEvents
Loop
Set desc = aIE.document.getElementsByClassName("DetailAttributes")(0)
'Debug.Print desc.innerText '---> prints the whole table data
Debug.Print Split(desc.innerText, vbLf)(3) '----> prints the forth data in table
Set aIE = Nothing
Set desc = Nothing
End Sub
And also if you plan to automate this code to run in a loop for multiple queries, you might want to use:
Set desc = Nothing
For i = 1 To 100
On Error Resume Next
Set desc = aIE.document.getElementsByClassName("DetailAttributes")(0)
If Err.Number = 91 Then
GoTo Skip
End If
Exit For
Skip:
Application.Wait (Now() + TimeValue("00:00:001"))
Next i
instead of:
Set desc = aIE.document.getElementsByClassName("DetailAttributes")(0)
This is because sometimes web page seems ready before it fully generates its contents. This causes the code to get out of do loop and proceed to next statement which sets desc object. You won't get an error while setting it because the code will be using previous DOM document object and will be outputting the results of your previous query, which is a bug. Without any errors, your code will run the loop till the end, and you will have a completely twisted output in your hand, which is a waste of time.
To work around this problem, you should set the object to nothing beforehand, and catch the error and wait for the page to load in for loop.
Last but not least, if the guys who build the web page that you are parsing are aware of what they are doing, they will probably protect it from multiple queries from the same source (most likely from multiple sources as well), which might cause their server to collapse if they don't. This protection will be reflected to you as limited number of queries within a limited amount of time. In other words, for example after 100 request within 5 minutes, the web page will not be responding for sometime (for example 2 minutes).
To workaround this problem, you should limit the number of requests and wait for the required time. Suppose that you increment your loop with i variable. Then you need to insert this at the end of your loop:
If i Mod 100 = 0 Then
Application.Wait (Now() + TimeValue("00:02:00"))
End If
I hope the above mentioned solutions solve everyone's past and future problems, which took me a considerable amount of time to figure out.