Logging into a website using Mechanize and Nokogiri? - authentication

I am having some difficulty with one of our service providers login forms. The other sites are working fine but for some reason I can't get past their login form.
The website login for is like this:
<form accept-charset="UTF-8" action="/sessions" class="new_user_session" id="new_user_session" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="✓" /><input name="authenticity_token" type="hidden" value="kaLEkPesQfeheronzGTdfnVAzpuUiC+VmjVXBu540n8=" /></div>
<fieldset class="big">
<div class="form-row">
<div class="form-label">
<label for="user_session_email">Email</label>
</div>
<div class="form-field">
<input id="user_session_email" name="user_session[email]" size="30" type="text" />
</div>
</div>
<div class="form-row">
<div class="form-label">
<label for="user_session_password">Password</label>
</div>
<div class="form-field">
<input id="user_session_password" name="user_session[password]" size="30" type="password" />
</div>
<div class="form-comment"><p>Forgot your password?</p></div>
</div>
<div class="form-row optional">
<div class="form-field">
<label for="user_session_remember_me"><input name="user_session[remember_me]" type="hidden" value="0" /><input id="user_session_remember_me" name="user_session[remember_me]" type="checkbox" value="1" /> Remember me for 2 weeks</label>
</div>
</div>
</fieldset>
I have tried to login using the same code as other the other sites but it doesn't work.
# Create a new mechanize object
agent = Mechanize.new
# Load the dial9 website
page = agent.get("http://webapplication.co.uk")
# Select the first form
form = agent.page.forms.first
form.username = 'username
form.password = 'password'
# Submit the form
page = form.submit form.buttons.first
I have also tried a different way of logging in as suggested in other SO questions/answers:
email = 'user#domain.com'
password = 'password
# Create a new mechanize object
agent = Mechanize.new
# Load the postmarkapp website
page = agent.get("https://domain.com")
# Select the first form
form = agent.page.forms.first
form.field_with(:email => "user_session_email").value = email
form.field_with(:password => "user_session_password").value = password
# Submit the form
page = form.submit form.buttons.first
Using this method of authentication I get the following output when running the rake task:
undefined method `email' for [hidden:0x3fef2ab2b994 type: hidden name: utf8 value: ✓]:Mechanize::Form::Hidden
Upon closer inspection the above error seems to be due to the fact that there is a field immediately after the form is started:
<form accept-charset="UTF-8" action="/sessions" class="new_user_session" id="new_user_session" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="✓" /><input name="authenticity_token" type="hidden" value="kaLEkPesQfeheronzGTdfnVAzpuUiC+VmjVXBu540n8=" /></div>
Am I missing something? If so, what? Any pointers are appreciated!

Try changing
form.field_with(:email => "user_session_email").value = email
form.field_with(:password => "user_session_password").value = password
to
form.field_with(:name => "user_session[email]").value = email
form.field_with(:name => "user_session[password]").value = password

Related

SiteFinity- Included Captcha Form Widget does not contain alt tag

SiteFinity noob here.
I've edited the widget via File Manager: Resource Packages>Bootstrap>MVC>Views>Captcha.
the file there is "Write.default.cshtml". I changed the following line-
<img data-sf-role="captcha-image" src='#Url.WidgetContent("assets/dist/img/dummy.jpg")'/>
to
<img data-sf-role="captcha-image" alt="captcha Image src='#Url.WidgetContent("assets/dist/img/dummy.jpg")'/>
However, although saved, this doesn't show up in the widget code when I put it on my form. I used File Manager, as I dont have a connection via .net editor.
Am I in the wrong place? Do I need to somehow restart the application?
Here is complete code:
#model Telerik.Sitefinity.Frontend.Forms.Mvc.Models.Fields.Captcha.CaptchaViewModel
#using Telerik.Sitefinity.Frontend.Mvc.Helpers;
#using Telerik.Sitefinity.Modules.Pages;
#using Telerik.Sitefinity.Services;
#Html.Script(ScriptRef.JQuery, "top", false)
<div data-sf-role="field-captcha-container" style="display:none;" class="#Model.CssClass form-group">
<div>
**<img data-sf-role="captcha-image" src='#Url.WidgetContent("assets/dist/img/dummy.jpg")'/>**
</div>
<a data-sf-role="captcha-refresh-button">#Html.Resource("NewCode")</a>
<div class="form-inline">
<div class="form-group">
<input data-sf-role="violation-messages" type="hidden" value='{"required": "#Model.ValidatorDefinition.RequiredViolationMessage"}' />
<label for='#Html.UniqueId("Textbox")'>#Html.Resource("TypeCodeAbove") </label>
<input id='#Html.UniqueId("Textbox")' type="text" data-sf-role="captcha-input" name="#Model.CaptchaAnswerFormKey" required="required" class="form-control input-sm"/>
</div>
</div>
<input type="hidden" data-sf-role="captcha-ca" name="#Model.CaptchaCorrectAnswerFormKey" />
<input type="hidden" data-sf-role="captcha-iv" name="#Model.CaptchaInitializationVectorFormKey" />
<input type="hidden" data-sf-role="captcha-k" name="#Model.CaptchaKeyFormKey" />
<input type="hidden" data-sf-role="captcha-settings" value="#Model.GenerateUrl"
</div>
#if (SystemManager.IsDesignMode)
{
var scriptUrl = Url.WidgetContent("Mvc/Scripts/Captcha/captcha.js");
var queryAddition = scriptUrl.Contains("?") ? "&" : "?";
var fullScriptUrl = scriptUrl + queryAddition + string.Format("_={0}", DateTime.UtcNow.Ticks.ToString());
<script type="text/javascript" src='#fullScriptUrl'></script>
}
else
{
#Html.Script(Url.WidgetContent("Mvc/Scripts/Captcha/captcha.js"), "bottom", false)
}
You'll need to make sure you are using the Bootstrap package. Go to Design > Page Template and see which resource package your page templates are using. Often times it's not the Bootstrap, but a copy of it.
A restart may help as well.

Generating a valid __RequestVerificationToken from C#

One of the most popular books on ASP.NET Core is "Pro ASP.NET Core 3" by Adam Freeman.
In chapters 7-11, he builds an example application, SportsStore.
As you can see, each product in the listing gets its own 'Add To Cart' button:
If we do 'view source' on this page, we'll see the following HTML for that item in the product list:
<div class="card card-outline-primary m-1 p-1">
<div class="bg-faded p-1">
<h4>
Kayak
<span class="badge badge-pill badge-primary" style="float:right">
<small>$275.00</small>
</span>
</h4>
</div>
<form id="1" method="post" action="/Cart">
<input type="hidden" data-val="true" data-val-required="The ID field is required." id="ID" name="ID" value="1" />
<input type="hidden" name="returnUrl" value="/" />
<span class="card-text p-1">
A boat for one person
<button type="submit" class="btn btn-success btn-sm pull-right" style="float:right">
Add To Cart
</button>
</span>
<input name="__RequestVerificationToken" type="hidden" value="CfDJ8KKqNOS0gwdMvC0-bdjTwWlvCcBJldeidwIX5b2f24gYblS9X1sqCwJWIEsKKOSf8kut0SQsQRLF3R1XBSYZkPGnta9YzRK4tcQl8dq_0uWmjeUhm8yMe90fWDt_x0smmAD1lmb9-BxQF8y_7-IQSz4" /></form>
</div>
Note the input tag towards the bottom:
<input name="__RequestVerificationToken" type="hidden" value="CfDJ8KKqNOS0gwdMvC0-bdjTwWlvCcBJldeidwIX5b2f24gYblS9X1sqCwJWIEsKKOSf8kut0SQsQRLF3R1XBSYZkPGnta9YzRK4tcQl8dq_0uWmjeUhm8yMe90fWDt_x0smmAD1lmb9-BxQF8y_7-IQSz4" />
If we look at the Views\Shared\ProductSummary.cshtml file in the SportsStore project, we'll see the code that is involved with generating these listing items:
#model Product
<div class="card card-outline-primary m-1 p-1">
<div class="bg-faded p-1">
<h4>
#Model.Name
<span class="badge badge-pill badge-primary" style="float:right">
<small>#Model.Price.ToString("c")</small>
</span>
</h4>
</div>
<form id="#Model.ID" asp-page="/Cart" method="post">
<input type="hidden" asp-for="ID" />
<input type="hidden" name="returnUrl" value="#ViewContext.HttpContext.Request.PathAndQuery()" />
<span class="card-text p-1">
#Model.Description
<button type="submit" class="btn btn-success btn-sm pull-right" style="float:right">
Add To Cart
</button>
</span>
</form>
</div>
As you can see, the form element in this case doesn't have an explicit inclusion of the input tag with the __RequestVerificationToken value. This form thus appears to be a tag helper which takes care of generting the input tag with the __RequestVerificationToken token.
As an experiment, let's suppose I have added the following method to Controllers\HomeController:
[HttpGet]
public ContentResult ButtonExample()
{
var token = "...";
return new ContentResult()
{
ContentType = "text/html",
StatusCode = (int)HttpStatusCode.OK,
Content =
String.Format(
#"<!DOCTYPE html>
<html>
<body>
<form id=""1"" method=""post"" action=""/Cart"">
<input type=""hidden"" data-val=""true"" id=""ID"" name=""ID"" value=""1"" />
<button type=""submit"">Add to Cart</button>
</form>
<input name=""__RequestVerificationToken"" type=""hidden"" value=""{0}"" />
</body>
</html>",
token)
};
}
As you can see, this generates a very simple page with a single button which is intended to add the product with ID value 1 (i.e. the Kayak) to the cart.
I of course need to pass an appropriate value for the __RequestVerificationToken.
My question is, is there a way to get this value from C# so that I can include it in the method above?
The idea as shown above would be to set the token value here:
var token = "...";
This is then interpolated into the string that generates the HTML using String.Format.
UPDATE
This page mentions the following:
To generate the anti-XSRF tokens, call the #Html.AntiForgeryToken method from an MVC view or #AntiForgery.GetHtml() from a Razor page.
So I guess the question is, how do we do the equivalent from C# directly instead of from an MVC view or Razor page?
You can add the below code to your form which will generate the __RequestVerificationToken. It is used to prevent CSRF attacks Prevent XSRF/CSRF attacks.
<form action="/" method="post">
#Html.AntiForgeryToken()
</form>

Pug iteration -- express routes

I have a route that returns error messages in express. Using connect-flash. Everything is working fine and I log the messages but I'm unable to display the messages in my pug file. Hopefully someone would direct me to my error/misunderstaning. I follow a tutorial to learn express, so I'll post my code first and then the tutorial code with handlebars.
My route is this:
router.get('/signup', function (req, res) {
var messages = req.flash('error')
console.log(messages)
res.render('signup', { csrfToken: req.csrfToken(), messages: messages, hasErrors : messages.length > 0 });
});
And here is my signup jade file. Notice the if block on top with "each" iteration and then displaying the SINGLE message in P line :
extends layout
block content
//- The error is logged but not displaying in my pug
//- Error iteration to display error messages
if (hasErrors)
each error in hasErrors
p= error
//- end of iteration
div.constainer
div.row
div.col-md-4.offset-4
h1 Signup
form(action="" method="post")
div.form-group
label E-mail:
br
input.form-control( type="text" id="email" name="email")
div.form-group
label Password:
br
input.form-control( type="password" id="password" name="password")
br
input(type="hidden" name = "_csrf" value="#{csrfToken}")
button(type="submit") Signup
The original tutorial has it with handlebars and it works. I'm trying to recreate the same example with pug. Here is the tutorial with handlebars code:
div class="row">
<div class="col-md-4 col-md-offset-4">
<h1>Sign Up</h1>
>>>>>>***{{#if hasErrors}}
<div class="alert alert-danger">
{{# each messages }}
<p>{{this}}</p>
{{/each}}
</div>
>>>>>>{{/if}}***
<form action="/user/signup" method="post">
<div class="form-group">
<label for="email">E-Mail</label>
<input type="text" id="email" name="email" class="form-control">
</div>
<div class="form-group">
<label for="password">Password</label>
<input type="password" id="password" name="password" class="form-control">
</div>
<input type="hidden" name="_csrf" value="{{ csrfToken }}">
<button type="submit" class="btn btn-primary">Sign Up</button>
</form>
</div>
</div>
The code with handlebars code works. I just translated the same code to pug.
If I remove iteration completely and just add #{hasErrors} Jade display true and false so my route is fine, it is my iteration in Pug.
I'm a beginner. Any help would be appreciated.
As confirmed by OP, the each statement should be each error in messages.

Using Scrapy for Crawling Pages Post Authentication

Taking cues and ideas from a previous post , I tried to come up with my own code.
However, using my code I've noticed that it doesn't really scrape anything and probably doesn't go beyond the authentication level at all. I say this because I don't see any Error Logs even when I put an incorrect password.
My best guess is that the HTML for the authentication fields is not contained in a "Form" tag and hence the formdata is possibly overlooking it. Might be wrong.
My Code so far:
class LoginSpider(BaseSpider):
name = 'auth1'
start_urls = ['http://www.example.com/administration']
def parse(self, response):
return [FormRequest.from_response(response,
formdata={'employee[email]': 'xyz#abc.com', 'employee[password]': 'XYZ'},
formxpath='//div[#class="form-row"]',
callback=self.after_login)]
def after_login(self, response):
if "authentication failed" in response.body:
self.log("Login failed", level=log.ERROR)
return
# We've successfully authenticated, let's have some fun!
else:
return Request(url="http://www.liveyoursport.com/administration/customers",
callback=self.parse_tastypage)
def parse_tastypage(self, response):
sel = Selector(response)
item = Item()
item ["Test"] = sel.xpath("//h1/text()").extract()
yield item
Here's the HTML section:
<div class="content-row">
<div class="special-header-title span_full">
<h3><span class="blue-text">Sign </span>In</h3>
</div>
</div>
<div class="content-row">
<div class="form-section checkout-address-edit span_80" id="sign-in-form" >
<form accept-charset="UTF-8" action="/employees/sign_in" class="new_employee" id="new_employee" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="✓" /><input name="authenticity_token" type="hidden" value="HQYZa0hNZ2Y+UvtbIk9OxI48Hlsnt+MiYOeV9ql2yWo=" /></div>
<div>
<div class="form-row">
<div class="form-col-1"><label for="employee_email">Email</label></div>
<div class="form-col-2">
<input id="employee_email" name="employee[email]" size="30" type="email" value="" />
</div>
</div>
<div class="form-row">
<div class="form-col-1"><label for="employee_password">Password</label></div>
<div class="form-col-2">
<input id="employee_password" name="employee[password]" size="30" type="password" />
</div>
</div>
</div>
<div class="form-row form-row-controls">
<div class="form-col-1"></div>
<div class="form-col-2">
<input class="sign-in-button f-right" name="commit" type="submit" value="Sign in" />
</div>
</div>
</form> <br>
Forgot your password?<br />
Didn't receive unlock instructions?<br />
</div>
From the docs:
formxpath (string) – if given, the first form that matches the xpath
will be used.
but it seems that you are not matching the form, rather the parent div.
Try it like this:
return [FormRequest.from_response(response,
formdata={'employee[email]': 'xyz#abc.com', 'employee[password]': 'XYZ'},
formxpath='//form[#id="new_employee"]',
callback=self.after_login)]
Also, if you only have one form element on the page, you don't need to define formxpath.

HTMLInputElemet or webbrowser.document on Windows Phone App Login

Hey I'm try to make a Windows Phone app that logs people in to a webpage, but I'm having trouble getting the app to post the log-in info to the webbrowser.
on the desktop I would have used
HTMLInputElement nameBox = (HTMLInputElement) myDoc.all.item("name", 0);
HTMLInputElement passBox = (HTMLInputElement) myDoc.all.item("pass", 0);
nameBox.value = "textBox1.text";
passBox.value = "textBox2.text;
but the HTMLInputElemet don't exist in Windows Phone.
I also tried using webBrowser1.document but that don't work either on Windows Phone.
It is a log-in using a form and not a query string.
The server side code look like this:
<div class="form-item textfield edit-name">
<div class="label">
<label >E-mail-adresse</label>
</div>
<input type="text" maxlength="60" placeholder="" name="name" id="edit-name" size="60" value="" class="form-text required" /> </div>
<div class="form-item password edit-pass">
<div class="label">
<label >Adgangskode</label>
</div>
<input type="password" name="pass" id="edit-pass" maxlength="128" size="60" class="form-text required" /> </div>
<input type="hidden" name="form_build_id" id="form-b6cb750ff3a79c34d408df98d730a39e" value="form-b6cb750ff3a79c34d408df98d730a39e" />
<input type="hidden" name="form_id" id="edit-user-login" value="user_login" />
<div class="form-actions"><input type="submit" name="op" id="edit-submit" value="Log ind" class="form-submit" />
It is easy in the desktop but I just cant figure out how to do this on the Phone, hope some one can help.
One way to access the fields is using Javascript, thanks to the InvokeScript method:
webBrowser1.InvokeScript("eval", string.Format("document.getElementsByName('name')[0].value='{0}'", "PutTheLoginHere"));