Scrape text from html element by class that's inside a list element - beautifulsoup

Question

I'm trying to scrape the headline from the first h4 element with the class "regularitem".
The output should look like "It took months to hit 3 million reported cases..."
I keep getting list index out of range.
headers = {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Methods': 'GET',
'Access-Control-Allow-Headers': 'Content-Type',
'Access-Control-Max-Age': '3600',
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
}
URL = 'http://rss.cnn.com/rss/cnn_topstories.rss'
req = requests.get(URL, headers)
soup = BeautifulSoup(req.content, 'html.parser')
headline = soup.findAll('h4', attrs = {'class' : 'itemtitle'})[0]
print(headline.get_text)
The page html looks like this:
<li xmlns:dc="http://purl.org/dc/elements/1.1/" class="regularitem">
<h4 class="itemtitle">It took months to hit 3 million reported cases. Now nearly two weeks later, US is on the verge of 4 million.</h4>
<h5 class="itemposttime">
<span>Posted:</span>Wed, 22 Jul 2020 14:52:09 GMT</h5>
<div class="itemcontent" name="decodeable">Tracking US cases | Podcast | Those you've lost<div class="feedflare">
<img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?d=yIl2AUoC8zA" border="0"> <img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?d=7Q72WNTAKBA" border="0"> <img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?i=3M8R-V8mvn8:YP_46RpyuXw:V_sGLiPBpWU" border="0"> <img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?d=qj6IDK7rITs" border="0"> <img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?i=3M8R-V8mvn8:YP_46RpyuXw:gIN9vFwOqvQ" border="0">
</div><img src="http://feeds.feedburner.com/~r/rss/cnn_topstories/~4/3M8R-V8mvn8" height="1" width="1" alt=""></div>
</li>
<li xmlns:dc="http://purl.org/dc/elements/1.1/" class="regularitem">
<h4 class="itemtitle">'We are going as quickly as we possibly can' on vaccine development, Fauci says</h4>
<h5 class="itemposttime"></h5>
<div class="itemcontent" name="decodeable"><div class="feedflare">
<img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?d=yIl2AUoC8zA" border="0"> <img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?d=7Q72WNTAKBA" border="0"> <img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?i=e93U4xIXAew:NWKJndB2i28:V_sGLiPBpWU" border="0"> <img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?d=qj6IDK7rITs" border="0"> <img src="http://feeds.feedburner.com/~ff/rss/cnn_topstories?i=e93U4xIXAew:NWKJndB2i28:gIN9vFwOqvQ" border="0">
</div><img src="http://feeds.feedburner.com/~r/rss/cnn_topstories/~4/e93U4xIXAew" height="1" width="1" alt=""></div>
</li>
I've tried removing the list index and changing to soup.find (see the example below) but when I do that I get: "NoneType' object has no attribute 'get_text"
headers = {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Methods': 'GET',
'Access-Control-Allow-Headers': 'Content-Type',
'Access-Control-Max-Age': '3600',
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
}
URL = 'http://rss.cnn.com/rss/cnn_topstories.rss'
req = requests.get(URL, headers)
soup = BeautifulSoup(req.content, 'html.parser')
headline = soup.find('h4', attrs = {'class' : 'itemtitle'})
print(headline.get_text)

Answer 1

The request you are making is not pulling the html because the html on the page is dynamically loaded. The solution is to use selenium since selenium emulates a real browser and will load the dynamic content.
# these settings may differ based on where you install your chrome webdriver
# alternatively you can use the firefox webdrive - there are plenty of tutorials online that show
# you how to install those
chrome_options = Options()
chrome_options.add_argument('--headless') # or '-start maximized' if you want to see the window open
driver = webdriver.Chrome(options=chrome_options)
headers = {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Methods': 'GET',
'Access-Control-Allow-Headers': 'Content-Type',
'Access-Control-Max-Age': '3600',
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
}
URL = 'http://rss.cnn.com/rss/cnn_topstories.rss'
driver.get(URL)
soup = BeautifulSoup(driver.page_source, 'html.parser')
headlines = soup.findAll('h4', {'class': 'itemtitle'})
for headline in headlines:
headlineText = headline.find('a').text
print(headlineText)
Let me know if this was helpful