Use Scrapy to cut down on Piracy - scrapy

Question

I am new to using Scrapy and I know very little of the Python Language. So far, I have installed Scrapy and gone through a few tutorials. After that, I have been trying to find a way to search many sites for the same data. My goal is to use Scrapy to find links to "posts" and links for a few search criteria. As an example, I would like to search site A, B, and C. Each site, I would like to see if they have a "post" about app name X, Y, and Z. If they have any "posts" on X, Y, Z. I would like it to grab the link to that post. If it would be easier... It can scan each post for our Company Name. Instead of X, Y, Z it would search the contents of each "post" for [Example Company name]. The reason that I am doing it this way is so that the JSON that is created just has links to the "posts" so that we can review them and contact the website if need be.
I am on Ubuntu 10.12 and I have been able to scrape the sites that we are wanting but I have not been able to narrow down the JSON to the needed info. So currently we are still having to go through hundreds of links, which is what we wanted to avoid by doing this. The reason that we are getting so many links is because all the tutorials that I have found are for scraping a specific HTML tag. I want it to search the tag to see if it contains any part of our App Titles or Package name.
Like this, it displays the post info but now we have to pick out the links from the json. Saves time but still not really what we are wanting. Part of that, I think is that I am not referencing or calling it correctly. Please give me any help that you can. I have spent hours trying to figure this out.
posts = hxs.select("//div[#class='post']")
items = []
for post in posts:
item = ScrapySampleItem()
item["title"] = post.select("div[#class='bodytext']/h2/a/text()").extract()
item["link"] = post.select("div[#class='bodytext']/h2/a/#href").extract()
item["content"] = post.select("div[#class='bodytext']/p/text()").extract()
items.append(item)
for item in items:
yield item
I am wanting to use this to cut down on Piracy of our Android Apps. If I can have this go out and search the Piracy sites that we want, I can then email the Site or Hosting Company with all of the links that we want removed. Under Copy Right law, they have to comply but they require that we link them to every "post" that they infringe upon which is why App Developers normally do not mess with this kind of thing. They have hundreds of apps so finding the links to your apps takes many hours of work.
Thank you for any help you can offer in advance. You will be helping out many App Developers in the long run!
Grady

Answer 1

Your XPath selectors are absolute. They have to be relative to the previous selector (the .)
posts = hxs.select('//div[#class='post']')
for post in posts:
item = ScrapySampleItem()
item['title'] = post.select('.//div[#class="bodytext"]/h2/a/text()').extract()
item['link'] = post.select('.//div[#class="bodytext"]/h2/a/#href').extract()
item['content'] = post.select('.//div[#class="bodytext"]/p/text()').extract()
yield item