I am trying to webscrape this website: https://va.betway.com/sports/category/basketball/usa/nba?tab=matches. I am unable to get this elementImage of game stats.
This is my code snippet:
s = Service("./drivers/geckodriver")
options = FirefoxOptions()
options.headless = True
browser = webdriver.Firefox(service=s,options = options)
browser.get(website_hr)
print('Title: %s' % browser.title)
player_prop0 = browser.find_elements(by='id',value = 'root')
player_prop2 = browser.find_elements(by=By.CLASS_NAME,value = 'sc-eZMymg ktcjyc')
I get no value from player_prop2 and player_prop0.
enter image description here
How can I get the data on this page? Thank you
I tried using ID and class to get the game lines for the NBA games
Related
I wrote a code to get an account followers. I log in account and go to profile and click follower box and scroll down until at the end withot problem but when i try to get follower's username I have a problem.
I do not get any error but list turns empty. Can you help me? Thanks
followers_panel = driver.find_element_by_xpath('/html/body/div[2]/div/div/div/div[2]/div/div/div[1]/div/div[2]/div/div/div/div/div[2]/div/div/div[2]')
last_ht, ht = 0, 1
while last_ht != ht:
last_ht = ht
time.sleep(2)
# scroll down and retrun the height of scroll
ht = driver.execute_script("""
arguments[0].scrollTo(0, arguments[0].scrollHeight);
return arguments[0].scrollHeight; """, followers_panel)
list_of_followers = []
# Extract the follower names
followers = followers_panel.find_elements_by_xpath('.//div[#class="PZuss"]')
list_of_followers = []
for follower in followers:
name = follower.find_element_by_xpath('.//a/span/div').text
list_of_followers.append(name)
print(list_of_followers)
does followers get elements? if yes, try out this instead of for loop
list_of_followers = list(map(lambda x: x.text, driver.find_elements_by_xpath('//div[#class="PZuss"]//a/span/div')))
I've been using selenium to scrape data on on this site but i have this error
matches = driver.find_element(By.XPATH,'//*[#id="mainContent"]/div[3]/div[1]/div[2]/section/div[1]/ul/li[1]')
SyntaxError: invalid syntax
So i would like to know if my code below is correct to scrape all the season on this site.
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
website = "https://www.premierleague.com/results?co=1&se=363&cl=-1"
driver.get(website)
sleep(10)
element = driver.find_element(By.XPATH,('//*[#class="_24Il51SkQ29P1pCkJOUO-7"]/button')).click()
sleep(10)
WebDriverWait(driver, 20).until(driver.find_element(By.ID,('//*[#id="advertClose"]')).click()
matches = driver.find_element(By.XPATH,'//*
[#id="mainContent"]/div[3]/div[1]/div[2]/section/div[1]/ul/li[1]')
matches = []
for match in matches:
team = matches.find_element(By.XPATH,('//*
[#id="mainContent"]/div[3]/div[1]/div[2]/section/div[1]/ul/li[1]/div/span')).text
scores = matches.find_element(By.XPATH,('//*[#id="mainContent"]/div[3]/div[1]/div[2]/section/div[1]/ul/li[1]/div/span/span[1]/span[2]')).text
print(match.text)
I am trying to extract userid, rating and review from the following site using selenium and it is showing "Invalid selector error". I think, the Xpath I have tried to define to get the review text is the reason for error. But I am unable to resolve the issue. The site link is as below:
teslamotor review
The code that I have used is following:
#Class for Review webscraping from consumeraffairs.com site
class CarForumCrawler():
def __init__(self, start_link):
self.link_to_explore = start_link
self.comments = pd.DataFrame(columns = ['rating','user_id','comments'])
self.driver = webdriver.Chrome(executable_path=r'C:/Users/mumid/Downloads/chromedriver/chromedriver.exe')
self.driver.get(self.link_to_explore)
self.driver.implicitly_wait(5)
self.extract_data()
self.save_data_to_file()
def extract_data(self):
ids = self.driver.find_elements_by_xpath("//*[contains(#id,'review-')]")
comment_ids = []
for i in ids:
comment_ids.append(i.get_attribute('id'))
for x in comment_ids:
#Extract dates from for each user on a page
user_rating = self.driver.find_elements_by_xpath('//*[#id="' + x +'"]/div[1]/div/img')[0]
rating = user_rating.get_attribute('data-rating')
#Extract user ids from each user on a page
userid_element = self.driver.find_elements_by_xpath('//*[#id="' + x +'"]/div[2]/div[2]/strong')[0]
userid = userid_element.get_attribute('itemprop')
#Extract Message for each user on a page
user_message = self.driver.find_elements_by_xpath('//*[#id="' + x +'"]]/div[3]/p[2]/text()')[0]
comment = user_message.text
#Adding date, userid and comment for each user in a dataframe
self.comments.loc[len(self.comments)] = [rating,userid,comment]
def save_data_to_file(self):
#we save the dataframe content to a CSV file
self.comments.to_csv ('Tesla_rating-6.csv', index = None, header=True)
def close_spider(self):
#end the session
self.driver.quit()
try:
url = 'https://www.consumeraffairs.com/automotive/tesla_motors.html'
mycrawler = CarForumCrawler(url)
mycrawler.close_spider()
except:
raise
The error that I am getting is as following:
Also, The xpath that I tried to trace is from following HTML
You are seeing the classic error of...
as find_elements_by_xpath('//*[#id="' + x +'"]]/div[3]/p[2]/text()')[0] would select the attributes, instead you need to pass an xpath expression that selects elements.
You need to change as:
user_message = self.driver.find_elements_by_xpath('//*[#id="' + x +'"]]/div[3]/p[2]')[0]
References
You can find a couple of relevant detailed discussions in:
invalid selector: The result of the xpath expression "//a[contains(#href, 'mailto')]/#href" is: [object Attr] getting the href attribute with Selenium
I'm extracting NBA stats from my yahoo fantasy account. Below is the code that I made in jupyter notebook using selenium. Each page shows 25 players and a total of 720 players. I did a for loop that will scrape players in increments of 25 instead of one by one.
for k in range (0,725,25):
Players = driver.find_elements_by_xpath('//tbody/tr/td[2]/div/div/div/div/a')
Team_Position = driver.find_elements_by_xpath('//span[#class= "Fz-xxs"]')
Games_Played = driver.find_elements_by_xpath('//tbody/tr/td[7]/div')
Minutes_Played = driver.find_elements_by_xpath('//tbody/tr/td[11]/div')
FGM_A = driver.find_elements_by_xpath('//tbody/tr/td[12]/div')
FTM_A = driver.find_elements_by_xpath('//tbody/tr/td[14]/div')
Three_Points = driver.find_elements_by_xpath('//tbody/tr/td[16]/div')
PTS = driver.find_elements_by_xpath('//tbody/tr/td[17]/div')
REB = driver.find_elements_by_xpath('//tbody/tr/td[18]/div')
AST = driver.find_elements_by_xpath('//tbody/tr/td[19]/div')
ST = driver.find_elements_by_xpath('//tbody/tr/td[20]/div')
BLK = driver.find_elements_by_xpath('//tbody/tr/td[21]/div')
TO = driver.find_elements_by_xpath('//tbody/tr/td[22]/div')
NBA_Stats = []
for i in range(len(Players)):
players_stats = {'Name': Players[i].text,
'Position': Team_Position[i].text,
'GP': Games_Played[i].text,
'MP': Minutes_Played[i].text,
'FGM/A': FGM_A[i].text,
'FTM/A': FTM_A[i].text,
'3PTS': Three_Points[i].text,
'PTS': PTS[i].text,
'REB': REB[i].text,
'AST': AST[i].text,
'ST': ST[i].text,
'BLK': BLK[i].text,
'TO': TO[i].text}
driver.get('https://basketball.fantasysports.yahoo.com/nba/28951/players?status=ALL&pos=P&cut_type=33&stat1=S_AS_2021&myteam=0&sort=AR&sdir=1&count=' + str(k))
The browser will go page by page after it's done. I print out the results. It only scrape 1 player. What did I do wrong?
A picture of my codes and printing the results
It's hard to see what the issue here is without looking at the original page (can you provide a URL?), however looking at this:
next = driver.find_element_by_xpath('//a[#id = "yui_3_18_1_1_1636840807382_2187"]')
"1636840807382" looks like a Javascript timestamp, so I would guess that the reference you've got hardcoded there is dynamically generated, so the element "yui_3_18_1_1_1636840807382_2187" no longer exists.
Update Changes, problem remain****************
cheregeckodriver_autoinstaller.install()
profile = webdriver.FirefoxProfile("/Users/Toshiba/AppData/Roaming/Mozilla/Firefox/Profiles/zq6j04pa.default")
profile.set_preference("dom.webdriver.enabled",False)
profile.set_preference("useAutomationExtension", False)
profile.update_preferences()
desired = DesiredCapabilities.FIREFOX
path = '/Users/Toshiba/Desktop/backup/PythonLearning/Learning/Selenium/geckodriver'
binary = FirefoxBinary("/Users/Toshiba/AppData/Local/Mozilla Firefox/firefox.exe")
ua = UserAgent()
user_agent = ua.random
options = webdriver.FirefoxOptions()
options.add_argument(f'user-agent={user_agent}')
# options.add_argument('-incognito')
wd = webdriver.Firefox(executable_path=path, firefox_profile=profile, desired_capabilities=desired, firefox_binary=binary)
url = "https://docs.google.com/forms/u/0/"
wd.get(url)
# login to google
username_class = "whsOnd zHQkBf".replace(" ",".")
wd.find_element_by_class_name(username_class).send_keys(username)
time.sleep(random.uniform(1, 3))
btn_next_class = "VfPpkd-RLmnJb"
wd.find_elements_by_class_name(btn_next_class)[-1].click()
Problem
Automate testing software detected
Couldn't sign you in
This browser or app may not be secure. Learn more
Try using a different browser. If you’re already using a supported browser, you can refresh your screen and try again to sign in.