The data parased from the website ended up with being an empty data (bs4, python, lxml) - beautifulsoup

Question

Hello Stackoverflow folks,
I have had a very hardtime with parsing information from a website
using Beautifulsoup and lxml.
I tried to bring the address data from "https://www1.nyc.gov/events/events-filter.html#page-1" website.
As far as I googled,
it says I need to
1. find the specific class for the information through "inspection" of the webpage.
2. need to write a code like g_data = soup.find_all("div", {"class": "event-data-detail"})
So I wrote my code as below.
import requests
from bs4 import BeautifulSoup
url = "https://www1.nyc.gov/events/events-filter.html#page-1"
r=requests.get("https://www1.nyc.gov/events/events-filter.html#page-1")
soup = BeautifulSoup(r.content)
links = soup.find_all("a")
g_data = soup.find_all("div", {"class": "event-data-detail"})
print(g_data)
and it shows the error message
Warning (from warnings module): File
"C:/Users/jotna/Desktop/Portfolio/1.py", line 7
soup = BeautifulSoup(r.content) UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for
this system ("lxml"). This usually isn't a problem, but if you run
this code on another system, or in a different virtual environment, it
may use a different parser and behave differently.
The code that caused this warning is on line 7 of the file
C:/Users/jotna/Desktop/Portfolio/1.py. To get rid of this warning,
pass the additional argument 'features="lxml"' to the BeautifulSoup
constructor.
So I fixed the code as below. (Cause a posting in stackoverflow suggested to add lxml code at the end)
import lxml
import requests
from bs4 import BeautifulSoup
url = "https://www1.nyc.gov/events/events-filter.html#page-1"
r=requests.get("https://www1.nyc.gov/events/events-filter.html#page-1")
soup = BeautifulSoup(r.content)
links = soup.find_all("a")
for link in links:
if "http" in link.get("href"):
print ("<a href='%s'>%s</a>" %(link.get("href"), link.text))
g_data = soup.find_all("div", {"span class": "address"})
print(g_data)
However it only shows empty bracket
[]
How can actually bring the address data from the website?
For your information, I uploaded the screen shot of the webpage source as well.
enter image description here

Answer 1

Use their json api instead of bs4, see the code below.
import requests
count = 0
for i in range(185):
count+=1
link = 'https://www1.nyc.gov/calendar/api/json/search.htm?&sort=DATE&pageNumber='+str(count)
req = requests.get(link)
for i in req.json()['items']:
address = (i['address'])
print 'Address:', address
Output
Address: Mulberry Street, Little Italy, New York
Address: Various locations Citywide
Address: SECOND AVENUE between EAST 32 STREET and EAST 33 STREET Manhattan
Address: FIRST AVENUE between EAST 92 STREET and EAST 93 STREET Manhattan
Address: CARROLL STREET between SMITH STREET and COURT STREET Brooklyn
Address: BROADWAY between WEST 114 STREET and WEST 116 STREET Manhattan
Address: CORTELYOU ROAD between RUGBY ROAD and ARGYLE ROAD Brooklyn
Address: QUEENS BOULEVARD between 70 AVENUE and 69 ROAD Queens
Address: 79 STREET between NORTHERN BOULEVARD and 34 AVENUE Queens
Address: PRINCE STREET between MOTT STREET and MULBERRY STREET Manhattan
Address: BUSHWICK AVENUE between NOLL STREET and ARION PLACE Brooklyn
Address: Alley Pond Park Adventure Center
Address: Atlantic Avenue between 4th Avenue and Hicks Street
Address: Alexander von Humboldt statue - Central Park West and 77th Street
Address: SEVENTH AVENUE between WEST 110 STREET and WEST 111 STREET Manhattan
Address: Wave Hill House - West 249th Street and Independence Avenue
Address: Broadway between Liberty Street and Rector Street
Address: Anibal Aviles Playground
Address: Myrtle Avenue between Fresh Pond Road and Wyckoff Avenue