Can't pull data from a stubborn webpage using vba - vba

Question

Hope you are doing well. The site i tried to scrape category-names from is very simple to look at if you notice it's inspected element but when i create a parser i can't pull the data. I wanted to scrape only the 7 category names from that page. I tried with all possible angles but failed. If anybody helps me point out what I'm doing wrong, I would be very grateful to him. Thanks in advance. FYC, I'm pasting here the code I tried with.
Sub ItemName()
Dim http As New MSXML2.XMLHTTP60, html As New HTMLDocument
Dim topics As Object, topic As Object, posts As Object, post As Object, ele As Object
Dim x As Long
x = 2
http.Open "GET", "http://www.bjs.com/tv--electronics.category.3000000000000144985.2002193", False
http.send
html.body.innerHTML = http.responseText
Set topics = html.getElementsByClassName("categories")
For Each topic In topics
For Each posts In topic.getElementsByTagName("li")
For Each post In posts.getElementsByTagName("a")
Set ele = post.getElementsByTagName("h4")(0)
Cells(x, 1) = ele.innerText
x = x + 1
Next post
Next posts
Next topic
End Sub

Answer 1

Here's one possible solution, I'm using the internet explorer object instead of MSXML. I'm able to retrieve the data from the page, and it's pretty quick.
Here's the full code:
Option Explicit
#If VBA7 Then
Public Declare PtrSafe Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As LongPtr)
#Else
Public Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long)
#End If
Sub ItemName()
On Error GoTo errhand:
Dim ie As Object: Set ie = CreateObject("InternetExplorer.Application")
Dim topics As Object, topic As Object
Dim i As Byte
With ie
.Visible = False
.Navigate "http://www.bjs.com/tv--electronics.category.3000000000000144985.2002193"
Sleep 500 ' Wait for the page to start loading
Do Until .document.readyState = 4 Or .busy = False Or i >= 100
Sleep 100
DoEvents
i = i + 1
Loop
End With
Set topics = ie.document.getElementsByClassName("name ng-binding")
For Each topic In topics
'Print out the element's innertext
Debug.Print topic.innertext
Next
ie.Quit
Set ie = Nothing
Exit Sub
errhand:
Debug.Print Err.Number, Err.Description
ie.Quit
Set ie = Nothing
End Sub

Answer 2

As the content of that site are generated dynamically, so there is no way for xmlhttp request to catch the page source. However, to get around that selenium is good to go, as it works well when it comes to deal with javascriptheavy website. I only used selenium in my below script to get the page source. As soon as it get that, I reverted back to usual vba method to accomplish the process.
Sub Grabbing_item()
Dim driver As New ChromeDriver, html As New HTMLDocument
Dim post As Object
With driver
.get "http://www.bjs.com/tv--electronics.category.3000000000000144985.2002193"
html.body.innerHTML = .ExecuteScript("return document.body.innerHTML;")
.Quit
End With
For Each post In html.getElementsByClassName("name")
x = x + 1: Cells(x, 1) = post.innerText
Next post
End Sub