Using Scrapy for Crawling Pages Post Authentication - authentication

Question

Taking cues and ideas from a previous post , I tried to come up with my own code.
However, using my code I've noticed that it doesn't really scrape anything and probably doesn't go beyond the authentication level at all. I say this because I don't see any Error Logs even when I put an incorrect password.
My best guess is that the HTML for the authentication fields is not contained in a "Form" tag and hence the formdata is possibly overlooking it. Might be wrong.
My Code so far:
class LoginSpider(BaseSpider):
name = 'auth1'
start_urls = ['http://www.example.com/administration']
def parse(self, response):
return [FormRequest.from_response(response,
formdata={'employee[email]': 'xyz#abc.com', 'employee[password]': 'XYZ'},
formxpath='//div[#class="form-row"]',
callback=self.after_login)]
def after_login(self, response):
if "authentication failed" in response.body:
self.log("Login failed", level=log.ERROR)
return
# We've successfully authenticated, let's have some fun!
else:
return Request(url="http://www.liveyoursport.com/administration/customers",
callback=self.parse_tastypage)
def parse_tastypage(self, response):
sel = Selector(response)
item = Item()
item ["Test"] = sel.xpath("//h1/text()").extract()
yield item
Here's the HTML section:
<div class="content-row">
<div class="special-header-title span_full">
<h3><span class="blue-text">Sign </span>In</h3>
</div>
</div>
<div class="content-row">
<div class="form-section checkout-address-edit span_80" id="sign-in-form" >
<form accept-charset="UTF-8" action="/employees/sign_in" class="new_employee" id="new_employee" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="✓" /><input name="authenticity_token" type="hidden" value="HQYZa0hNZ2Y+UvtbIk9OxI48Hlsnt+MiYOeV9ql2yWo=" /></div>
<div>
<div class="form-row">
<div class="form-col-1"><label for="employee_email">Email</label></div>
<div class="form-col-2">
<input id="employee_email" name="employee[email]" size="30" type="email" value="" />
</div>
</div>
<div class="form-row">
<div class="form-col-1"><label for="employee_password">Password</label></div>
<div class="form-col-2">
<input id="employee_password" name="employee[password]" size="30" type="password" />
</div>
</div>
</div>
<div class="form-row form-row-controls">
<div class="form-col-1"></div>
<div class="form-col-2">
<input class="sign-in-button f-right" name="commit" type="submit" value="Sign in" />
</div>
</div>
</form> <br>
Forgot your password?<br />
Didn't receive unlock instructions?<br />
</div>

Answer 1

From the docs:
formxpath (string) – if given, the first form that matches the xpath
will be used.
but it seems that you are not matching the form, rather the parent div.
Try it like this:
return [FormRequest.from_response(response,
formdata={'employee[email]': 'xyz#abc.com', 'employee[password]': 'XYZ'},
formxpath='//form[#id="new_employee"]',
callback=self.after_login)]
Also, if you only have one form element on the page, you don't need to define formxpath.