Beautiful soup pagination - only scrapes first page - beautifulsoup

Question

I have one more issue with my BeautifulSoup scraper and was hoping you'd be able to help me. Since adding the code for it to extract information from the details pages, now the scraper only scrapes the first page and no longer scrapes the multiple pages outlined.
I assume it has to do with the loop but I'm not sure how to define this to ensure it doesn't cause issues.
Is there a better way for pagination?
I've posted the full code and bolded the new section that's causing problems.
Full code:
from bs4 import BeautifulSoup
import scrapy
import requests
import csv
import time
import os
pages = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
with open(r'csv', 'a', encoding='utf-8', newline='') as f_output:
csv_print = csv.writer(f_output)
file_is_empty = os.stat(r'C:\csv').st_size == 0
if file_is_empty:
csv_print.writerow(['Title', 'Company', 'Location', 'Salary', 'Summary', 'Link', 'Description', 'URL'])
for page in pages:
source = requests.get('https://www.indeed.com/jobs?q=work+from+home&l=United+States&fromage=1&start={}'.format(page)).text
soup = BeautifulSoup(source, 'lxml')
results = soup.findAll("div", {"class": "result"})
for jobs in soup.find_all(class_='result'):
try:
title = jobs.h2.text.strip()
except Exception as e:
title = None
print('Title:', title)
try:
company = jobs.span.text.strip()
except Exception as e:
company = None
print('Company:', company)
try:
location = jobs.find('span', class_='location').text.strip()
except Exception as e:
location = None
print('Location:', location)
try:
salary = jobs.find('span', class_='no-wrap').text.strip()
except Exception as e:
salary = None
print('Salary:', salary)
try:
summary = soup.find('div', class_='summary').text.strip()
except Exception as e:
summary = None
print('Summary:', summary)
link = jobs.a['href']
if 'http' not in link:
link = ("https://www.indeed.com" + link)
print('Link:', link)
**page = requests.get(link)
soup = BeautifulSoup(page.content, 'html.parser')
job_description = soup.find('div', id='jobDescriptionText').decode_contents(formatter="html")
print('job_description:', job_description)
try:
url = soup.find('div', class_='icl-u-lg-hide').attrs['href']
except Exception as e:
url = None
print('url:', url)**
csv_print.writerow((title, company, location, salary, summary, link, job_description, url))
print('--------')
time.sleep(0.5)
Thank you so much :)
You guys are the best!

Answer 1

Couple things,
BeautifulSoup parse html, it has nothing to do with implementing the pagination process.
The way your code is written, it actually doesnt scrape the first page, it's scraping the last page. You have it iterating through the list of pages, and then have it parse the html (which in this case is the very last request made). you need to parse your results for each page. Meaning for jobs in soup.find_all(class_='result'): needs to be within your loop of for page in pages:
This is just my preference, but I like using pandas. So I used that instead of csv to write to file
Code:
from bs4 import BeautifulSoup
import pandas as pd
import requests
import time
pages = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
titleList = []
companyList = []
locList = []
salList = []
summaryList = []
linkList = []
descList = []
urlList = []
for page in pages:
source = requests.get('https://www.indeed.com/jobs?q=work+from+home&l=United+States&fromage=1&start={}'.format(page)).text
soup = BeautifulSoup(source, 'lxml')
print ('Page: %s' %page)
results = soup.findAll("div", {"class": "result"})
for jobs in soup.find_all(class_='result'):
try:
title = jobs.h2.text.strip()
except Exception as e:
title = None
print('Title:', title)
try:
company = jobs.span.text.strip()
except Exception as e:
company = None
#print('Company:', company)
try:
location = jobs.find('span', class_='location').text.strip()
except Exception as e:
location = None
#print('Location:', location)
try:
salary = jobs.find('span', class_='no-wrap').text.strip()
except Exception as e:
salary = None
#print('Salary:', salary)
try:
summary = soup.find('div', class_='summary').text.strip()
except Exception as e:
summary = None
#print('Summary:', summary)
link = jobs.a['href']
if 'http' not in link:
link = ("https://www.indeed.com" + link)
#print('Link:', link)
page = requests.get(link)
soup = BeautifulSoup(page.content, 'html.parser')
job_description = soup.find('div', id='jobDescriptionText').decode_contents(formatter="html")
#print('job_description:', job_description)
try:
url = soup.find('div', class_='icl-u-lg-hide').attrs['href']
except Exception as e:
url = None
#print('url:', url)
titleList.append(title)
companyList.append(company)
locList.append(location)
salList.append(salary)
summaryList.append(summary)
linkList.append(link)
descList.append(job_description)
urlList.append(url)
print('--------')
time.sleep(0.5)
df = pd.DataFrame({
'Title':titleList,
'Company':companyList,
'Location':locList,
'Salary':salList,
'Summary':summaryList,
'Link':linkList,
'Description':descList,
'URL':urlList})
df.to_csv('file.csv',index=False)
Output:
print (df.head(15).to_string())
Title Company Location Salary Summary Link Description URL
0 Medical Transcription Documentation Specialist... new None None Extract electronic medical record data includi... https://www.indeed.com/rc/clk?jk=b728723376dae... Dane Street is looking for highly motivated ca... None
1 Company Partnerships Data Contractor\nnew new New York, NY 10001 (Flatiron District area) $30 an hour None https://www.indeed.com/rc/clk?jk=188b5c89acc25... <div><div>Do you love gathering, analyzing and... None
2 Customer Experience Associate - work from home... new Baltimore, MD 21230 (Spring Garden Industrial ... $16 - $17 an hour None https://www.indeed.com/rc/clk?jk=bdf4e0b474852... <div><div><div><b>About Hungry Harvest</b></di... None
3 Customer Care Representative I - Houston, TX -... new Houston, TX 77036 (Bellaire area) None None https://www.indeed.com/rc/clk?jk=b144f9b3db19a... <div><p><b>Description</b>\n</p>SHIFT: Day Job... None
4 Talent Acquisition Specialist\nnew new None None None https://www.indeed.com/rc/clk?jk=f9c860e3dea80... <p></p><div><p><b>The Position</b></p><p>\nTal... None
5 Talent Acquisition Specialist\nnew new None None None https://www.indeed.com/rc/clk?jk=5b1609f84a81b... <div><div><div><div>Job Details</div>\n</div><... None
6 Success Consultant\nnew new None None None https://www.indeed.com/rc/clk?jk=e2b62a52550b8... <div><div>WORK FROM HOME</div><div><h6 class="... None
7 Work At Home Customer Service Specialist (Bell... new Belle Glade, FL $13.25 an hour None https://www.indeed.com/rc/clk?jk=0bd3274a26a1f... US55505\n<br/><br/>\nJob Description Details\n... None
8 Data Associate\nnew new None None None https://www.indeed.com/rc/clk?jk=88d65d8102b17... <div></div><div><div><div><div>The DNC Coordin... None
9 States Data Analyst\nnew new None None None https://www.indeed.com/rc/clk?jk=8c84a472e6119... <div></div><div><div><div><div>The DNC Coordin... None
10 Coder Franciscan Health None None Understanding of payer relationships, requirem... https://www.indeed.com/rc/clk?jk=cc0748fa0c360... <div><div>Ambulatory Coding | Franciscan Allia... None
11 Data Associate Democratic National Committee None None None https://www.indeed.com/rc/clk?jk=88d65d8102b17... <div></div><div><div><div><div>The DNC Coordin... None
12 Work At Home Customer Service Specialist (Boyn... HSN Boynton Beach, FL $13.25 an hour None https://www.indeed.com/rc/clk?jk=a59f7532662bc... <p></p><div>US55505\n<p></p><p><b>Job Descript... None
13 Associate Project Manager ConvergeOne None None None https://www.indeed.com/rc/clk?jk=cbc2520fc153b... <div>C1 Company Overview:<div><b>\nConvergeOne... None
14 Customer Solutions Specialist Webstaurant Store, Inc. Albany, GA $19 an hour None https://www.indeed.com/rc/clk?jk=8f5bd656508fd... <div><p>Customer Solutions Specialist</p>\n<p>... None
....