Is it possible to web scape the pythagorean win/loss record column using beautiful soup and save data to excel? - pandas

Question

This is my first scrape. I have tried watching some videos online and using google. I haven't had much success with this website, though. Perhaps someone can help me.
This is what I have started with...
from lxml import html
from bs4 import BeautifulSoup
import requests
import pandas as pd
req = requests.get('https://www.baseball-reference.com/leagues/MLB-
standings.shtml')
soup = BeautifulSoup(req.text, "lxml")
W_L = [i.text for i in soup.find_all('td', {'data-stat': 'record_pythag'})]
team = [i.text for i in soup.find_all('td', {'data-stat': 'team_ID'})]
my_dict = dict(zip(team, W_L))
df = pd.DataFrame(my_dict)
writer = pd.ExcelWriter('my1st_webscrape.xlsx')
df.to_excel(writer,'Sheet1')
writer.save()
I would just like the pythagorean win/loss column please. Thanks!

Answer 1

Search for all td tags that include data-stat corresponding to record_pythag and extract .text from each:
W_L = [i.text for i in soup.find_all('td', {'data-stat': 'record_pythag'})]
Note that this assumes all tags found will include text. If one does not it will throw a NoneType error, in which case you could wrap this in a try-except block.
I imagine the dictionary you want to create uses the team name abbreviations as the keys, in which case you can assemble a list of them like so:
team = [i.text for i in soup.find_all('td', {'data-stat': 'team_ID'})]
Then you can create your dictionary:
my_dict = dict(zip(team, W_L))
And pass it to a dataframe:
df = pd.DataFrame(my_dict)
EDIT
It turns out you actually need Selenium in order to pull the information you are interested in. Selenium allows you to automate a web browser and it will load the full page source HTML. You will also need to convert the dictionary to a pandas Series prior to passing it to the DataFrame constructor. See revised code below (note that you will need to download the selenium chrome driver, or whichever web browser driver you are interested in using):
from bs4 import BeautifulSoup
import requests, os
import pandas as pd
from selenium import webdriver
os.chdir('path your chrome driver')
header = {'User-agent' : 'Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5'}
options = webdriver.ChromeOptions(); options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://www.baseball-reference.com/leagues/MLB-standings.shtml')
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()
W_L = [i.text for i in soup.find_all('td', {'data-stat': 'record_pythag'})]
team = [i.text for i in soup.find_all('td', {'data-stat': 'team_ID'})]
my_dict = dict(zip(team, W_L))
df = pd.DataFrame(pd.Series(my_dict))
writer = pd.ExcelWriter('my1st_webscrape.xlsx')
df.to_excel(writer,'Sheet1')
writer.save()