Scrapy/Selenium: How do I call out Selenium in Spider? - selenium

Question

I am new to web-scraping.
I would like to follow all the links in Webpage_A. For each link, I need to select some xpath and download the data in an Excel file.
In the below code, I used
yield response.follow
to follow all the links in Webpage_A
In
def parse_table
I used webdriver and driver.get to access the webpages of the links.
Is there another way to access the webpages instead? Eg, SeleniumRequest?
I noted driver.implicitly_wait(120) doesn't work in letting Chrome remain open.
I have to use sleep. Why doesn't implicitly_wait work?
Finally, why does the Chrome browser auto-close although I didn't use driver.close()?
class testSpider2(scrapy.Spider):
name = 'test_2'
def start_requests(self):
yield SeleniumRequest(
url='CONFIDENTIAL',
wait_time=15,
screenshot=True,
callback=self.parse
)
def parse(self, response):
tables_name = response.xpath("//div[#class='contain wrap:l m-x:a']//li")
for t in tables_name:
name=t.xpath(".//a/span/text()").get()
link = t.xpath(".//a/#href").get()
if link :
yield response.follow(url=link, callback=self.parse_table, meta={'table_name': name, 'link_t': link})
def parse_table(self, response):
chrome_path = "C:/Users/chromedriver.exe"
driver = webdriver.Chrome(executable_path=chrome_path)
name = response.request.meta['table_name']
link_t = response.request.meta['link_t']
driver.get(link_t)
driver.implicitly_wait(120)
button_select=driver.find_element_by_xpath("(//a[text()='Select All'])").click()
button_st_yr=driver.find_element_by_xpath("//select[#name='ctl00$ContentPlaceHolder1$StartYearDropDownList'] /option[1]").click()
button_end_mth=driver.find_element_by_xpath("//select[#name='ctl00$ContentPlaceHolder1$EndMonthDropDownList']/option[text()='Dec']").click()
button_download=driver.find_element_by_xpath("//input[#id='ctl00_ContentPlaceHolder1_DownloadButton']").click()
time.sleep(120)