Scraper clicking on the next page button but fetches nothing - vba

Question

I've written some code in vba in combination with selenium to parse data from different tables spreading across multiple pages. When I run my script I can see that it parses data from the first page and then keep clicking on next page button until there is no more button is available. However, I'm getting the data from first page and seeing the browser clicking on the next page button for nothing cause it doesn't fetch any data from other pages. I don't understand what I'm doing wrong here. Perhaps, the loop I have created has got something to do with it or I don't know. Thanks for taking a look into it. Here is the full code:
Sub Table_data()
Dim driver As New ChromeDriver
Dim tabl As Object, rdata As Object, cdata As Object
driver.Get "https://toolkit.financialexpress.net/santanderam"
driver.Wait 1000
For Each tabl In driver.FindElementsByXPath("//table[#class='fe-datatable']")
For Each rdata In tabl.FindElementsByXPath(".//tr")
For Each cdata In rdata.FindElementsByXPath(".//td")
y = y + 1
Cells(x + 1, y) = cdata.Text
Next cdata
x = x + 1
y = 0
Next rdata
driver.FindElementByLinkText("Next").Click
driver.Wait 1000
Next tabl
End Sub

Answer 1

Consider pressing the Next button outside of your loops. You should use it within another loop, and the loop should terminate when there is no more Next button to press (Run-time Error 7: NoSuchElementError)
Xpath //table[#class='fe-datatable'] returns Page numbers as well. You should be using the inner table which is //table[#class='fe-fund-tableBody'] by class name or if you seek by id //*[#id='docRows']. They will point to the same element.
You might have noticed there are 7 occurrences of the above mentioned element. Your code loops through the empty ones for each page. You can avoid this by looping through the first occurence only, like this: (//table[#class='fe-fund-tableBody'])[1] or (//*[#id='docRows'])[1].
I also would recommend to find a way to implicit/explicit wait instead of wait. If we don't go further to improve anything else, in the end your code should look something like this:
Sub Table_data()
Dim driver As New ChromeDriver
Dim tabl As Object, rdata As Object, cdata As Object
driver.Get "https://toolkit.financialexpress.net/santanderam"
driver.Wait 1000
Do
For Each tabl In driver.FindElementsByXPath("(//*[#id='docRows'])[1]") 'or "(//table[#class='fe-fund-tableBody'])[1]"
For Each rdata In tabl.FindElementsByXPath(".//tr")
For Each cdata In rdata.FindElementsByXPath(".//td")
y = y + 1
Cells(x + 1, y) = cdata.Text
Next cdata
x = x + 1
y = 0
Next rdata
Next tabl
On Error Resume Next
driver.FindElementByLinkText("Next").Click
driver.Wait 1000
Loop Until Err.Number = 7
End Sub

Answer 2

Personally I would change the way you are iterating the pages. It should be like this in pseudo code:
function element getNextButton(){
all_buttons = driver.findElementsByXpath("""//*[#id="Price_1_1"]/tfoot/tr/td/div/div/a""");
next_button = all_buttons[all_buttons.Size()-1];
return next_button;
}
main(){
next_button = getNextButton();
while true{
do something with your current table;
next_button.click();
wait(2); // wait some time till the page loads
next_button = getNextButton();
if next_button.text does not contains 'Next'{
break;
}
}
}
I have just tested it on Python:
from selenium import webdriver
import time
def get_next_button():
buttons = driver.find_elements_by_xpath("""//*[#id="Price_1_1"]/tfoot/tr/td/div/div/a""")
next_element_button = buttons[len(buttons)-1]
return next_element_button
chrome_path = r"chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("https://toolkit.financialexpress.net/santanderam")
time.sleep(5)
next_button =get_next_button()
while(True):
# Do something with the table
next_button.click()
time.sleep(2)
next_button = get_next_button()
if 'Next' not in next_button.text:
break
print 'End'
I am not familiar with vba, but if you do not understand Python I can try to translate it to vba.
EDIT
An "approximation" to VBA solution should be this (please check syntax errors, I have never used VBA):
Function GetNextElement() as Object
Dim all_buttons As Object
Dim next_button As Object
all_buttons= driver.FindElementsByXpath("""//*[#id="Price_1_1"]/tfoot/tr/td/div/div/a""")
next_button = all_buttons[all_buttons.Length-1]
Return next_button
End Function
Sub Table_data()
Dim driver As New ChromeDriver
Dim position as Integer
Dim next_button As Object
driver.Get "https://toolkit.financialexpress.net/santanderam"
driver.Wait 1000
next_button = GetNextElement()
Do While True
// Do something with the table
next_button.Click
driver.Wait 2000
next_button = GetNextElement()
position = InStr(next_button.Text,"Next")
If position = 0 Then
Exit Do
End If
Loop
End Sub