How to get Scrapy to parse CSS - scrapy

Question

I am following this guide to scrape movie titles from my local cinema website. I am using Scrapy Spider and CSS parsing to get this done. Within the HTML for the site, each movie title is constructed like this:
<div class="col-md-12 movie-description">
<h2>Minions: The Rise of Gru<h2>
...
Here is my code that attempts to scrape this info
import scrapy
class CinemaSpider(scrapy.Spider):
name = "cinema"
allowed_domains = ["cannonvalleycinema10.com"]
start_urls = ["https://cannonvalleycinema10.com/"]
def parse(self, response):
movie_names = response.css(".col-md-12.movie-description h2::text").extract()
for movie_name in movie_names:
yield {
'name': movie_name
}
The cinema's website is here. I have tried all sorts of different combinations for what would get the titles I'm looking for to be added to my json file but can't figure it out.
If it helps, I am running this code:
scrapy runspider .\cinema_scrape.py -o movies.json
I am in the proper directory, too.

Answer 1

The page is dynamically loaded so you have try scrapy and json together :
import scrapy
from scrapy import FormRequest
from scrapy.crawler import CrawlerProcess
import json
from scrapy.http import Request
class TestSpider(scrapy.Spider):
name = 'test'
url = 'https://cabbtheatres.intensify-solutions.com/embed/ajaxGetRepertoire'
cookies = {
'PHPSESSID': 'i8l12572hvd3a702d4nfj3vbg0',
}
headers = {
'Accept': 'application/json, text/javascript, */*; q=0.01',
'Accept-Language': 'en-US,en;q=0.9',
'Connection': 'keep-alive',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
# 'Cookie': 'PHPSESSID=i8l12572hvd3a702d4nfj3vbg0',
'Origin': 'https://cabbtheatres.intensify-solutions.com',
'Referer': 'https://cabbtheatres.intensify-solutions.com/embed?location=3663456',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
'X-Requested-With': 'XMLHttpRequest',
'sec-ch-ua': '".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
}
data = {
'location': '3663456',
'date': '2022-07-30',
'lang': 'en',
'soon': '',
}
def start_requests(self):
yield scrapy.FormRequest(
url =self.url,
method='POST',
formdata=self.data,
headers=self.headers,
callback=self.parse_item,
)
def parse_item(self, response):
detail=response.json()
titles=detail['data']
for name in titles:
title=name['title']
print(title)
output:
Minions: The Rise of Gru
Thor Love and Thunder
DC League of Super-Pets
Elvis(2022)
Mrs. Harris Goes to Paris
Where the Crawdads Sing
Top Gun: Maverick
Nope