I am using PhantomJS to make calls to a web page, like this:
page.open('http://example.com', function (s) {
console.log(page.content);
phantom.exit();
});
I am using this in the context of Drupal Simpletests, which require me to set a special USERAGENT in order to use the test database instead of the real database. I would like to fetch the web page a specific user agent. For example, in PHP with Curl, I can do this with CURLOPT_USERAGENT before making a cUrl call.
Thanks!
Albert
I found the answer in the useragent.js file in the phantomjs examples directory:
var page = require('webpage').create();
console.log('The default user agent is ' + page.settings.userAgent);
page.settings.userAgent = 'SpecialAgent';
page.open('http://www.httpuseragent.org', function (status) {
if (status !== 'success') {
console.log('Unable to access network');
} else {
var ua = page.evaluate(function () {
return document.getElementById('myagent').innerText;
});
console.log(ua);
}
phantom.exit();
});
Related
I currently have the loopbackJS api hosted on a domain (e.g. http://backend.com), with third party authentication setup via Auth0. I have a front-end hosted as a SPA on another domain (e.g. http://frontend.com)
loopback-component-passport seems to work fine when the front-end is on the same domain as the API, and it sets the userId and access_token cookies accordingly. However, my front-end in production is on a different domain to the API, for example the API auth link would be something like:
"http://backend.com/auth/auth0?returnTo=" + encodeURIComponent("http://frontend.com")
The backend has used the same auth pattern as in the loopback-passport-example, where a providers.json file specifies the connection details for Auth0 (although I have also tried other social providers such as Facebook).
"auth0-login": {
"provider": "auth0",
"module": "passport-auth0",
"clientID": "AUTH0_CLIENT_ID",
"clientSecret": "AUTH0_CLIENT_SECRET",
"callbackURL": "/auth/auth0/callback",
"authPath": "/auth/auth0",
"callbackPath": "/auth/auth0/callback",
"successRedirect": "/",
"failureRedirect": "/login",
"scope": ["email"],
"failureFlash": true
}
The front-end (http://frontend.com) has a link on the page to redirect to the API authentication:
Login
Clicking on this link redirects to Auth0 properly, and I can login. It then redirects to the specified target (http://backend.com or http://frontend.com, whichever is specified). The returnTo query parameter also seems to work as expected.
Is there a way to capture the access_token just before redirecting back to the front-end, and somehow communicate it (e.g. query parameters, unless that would be too insecure).
After some more investigation, I settled on this method to use for passing the access token and userId from loopbackjs backend, to a separate front-end. This was documented on a github pull-request, using a customCallback of passport-configurator.
Other places that have referenced this are this fork, issue #102, issue #14 and pull request #155.
There are 2 options here, either use a fork of loopback-component-passport (e.g. the one referenced above) as your npm dependency, or provide a customCallback as a passport configuration option as documented.
I wanted a little more control on the format of the URL, so ended up with the customCallback method. In loopback-example-passport, inside /server/server.js there is some basic code for passing providers.json to the passport configurator:
var config = {};
try {
config = require('../providers.json');
} catch (err) {
console.trace(err);
process.exit(1); // fatal
}
passportConfigurator.init();
for (var s in config) {
var c = config[s];
c.session = c.session !== false;
passportConfigurator.configureProvider(s, c);
}
This can be essentially replaced with the documented customCallback code, with the passport variable being assigned by passportConfigurator.init():
var providers = {};
try {
providers = require('../providers.json');
} catch (err) {
console.trace(err);
process.exit(1); // fatal
}
const passport = passportConfigurator.init();
Object.keys(providers).forEach(function(strategy) {
var options = providers[strategy];
options.session = options.session !== false;
var successRedirect = function(req) {
if (!!req && req.session && req.session.returnTo) {
var returnTo = req.session.returnTo;
delete req.session.returnTo;
return returnTo;
}
return options.successRedirect || '';
};
options.customCallback = !options.redirectWithToken
? null
: function (req, res, next) {
var url = require('url');
passport.authenticate(
strategy,
{session: false},
function(err, user, info) {
if (err) {
return next(err);
}
if (!user) {
return res.redirect(options.failureRedirect);
}
var redirect = url.parse(successRedirect(req), true);
delete redirect.search;
redirect.query = {
'access_token': info.accessToken.id,
'userId': user.id.toString()
};
redirect = url.format(redirect);
return res.redirect(redirect);
}
)(req, res, next);
};
passportConfigurator.configureProvider(strategy, options);
});
In the above example, I have essentially copied the successRedirect function used in passport-configurator.js, to use the same returnTo query parameter. An option within providers.json can be set e.g. "redirectWithToken": true, which results in redirect only for the auth strategies that need external redirect.
One more final bit of code in case the returnTo redirect is required. If it exists as a query parameter, it should be added at a session level:
app.use(function(req, res, next) {
var returnTo = req.query.returnTo;
if (returnTo) {
req.session = req.session || {};
req.session.returnTo = require('querystring').unescape(returnTo);
}
next();
});
Now, if the backend api is at a URL such as http://api.com, and the front-end is hosted at another domain e.g. http://gui.com, an authentication link can be placed on the front-end:
Login!
This will result in an API auth call, then redirect back to the returnTo link with the access token and userId in the query parameters.
Potentially in the future, one of the issues or other pull requests will be merged that could provide a more ideal method for 3rd party domain redirection, but until then this method work well.
I've read through Durandal login page redirect pattern wow, lots of code to do what I'd think would be pretty simple.
I've also read through https://groups.google.com/forum/#!topic/durandaljs/RdGpwIm1oOU as I'd like the login page to have a simple logo with a login form, but I'd also like routing for a registration and about page as well. The rest of my site will have a menu, header, etc which I don't want to show until the user is logged in. Also, I'm not sure how this approach would update when the user logs in.
Another code example that almost does what I want to do: https://github.com/Useful-Software-Solutions-Ltd/Durandal451/blob/master/Durandal451v2/App/global/session.js
So, what should I do? Is there an official way to do this? There seems to be a mish mash of things out there that people have tried. I would think this would be a really common occurrence but couldn't find anything on the main docs.
I'm not sure this is the simplest way, but this is what I got
you will need to add some extra function after app.start() is triggered.
main.js
var auth = require('authentication'); // Authentication module
app.start().then(function()
{
// This function will wait for the promise
auth.init().then(function(data)
{
// When successfully authenticate, set the root to shell
app.setRoot('views/shell');
}
});
authentication.js
define(function(require)
{
var app = require('durandal/app');
return {
init: function()
{
// Initialize authentication...
return system.defer(function(dfd)
{
// Check if user is authenticate or if there has stored token
var isAuthenticate = someOtherFunctiontoCheck();
if (isAuthenticate)
{
dfd.resolve(true); // return promise
}
else
{
// When not authenticate, set root to login page
app.setRoot('views/login');
}
}
}
};
});
good luck! :)
UPDATE
login.js
define(function(require)
{
var ko = require('knockout');
var auth = require('authentication');
var username = ko.observable();
var password = ko.observable();
return {
username: username,
password: password,
submitForm: function()
{
// Do a login, if success, auth module will take care of it
// and here will take of the error
auth.login(username(), password()).error(function()
{
// notify user about the error (e.g invalid credentials)
});
}
};
});
Authentication.js
define(function(require)
{
var app = require('durandal/app');
return {
init: function()
{
// Initialize authentication...
return system.defer(function(dfd)
{
// Check if user is authenticate or if there has stored token
var isAuthenticate = someOtherFunctiontoCheck();
if (isAuthenticate)
{
dfd.resolve(true); // return promise
}
else
{
// When not authenticate, set root to login page
app.setRoot('views/login');
}
}
},
login: function(username, password)
{
// do authenticate for login credentials (e.g for retrieve auth token)
return $.ajax({
url : 'api/login',
type : 'POST',
data : {
username: username,
password: password
}
}).then(function(token){
// on success, stored token and set root to shell
functionToStoreToken(token);
// Set root to shell
app.setRoot('views/shell');
});
}
};
});
I have two urls X and Y. When casperjs is on X page, it have to call Y page and after getting the response it should continue.
casper.start("X URL",function() {
var casper2 = require('casper').create();
casper2.start();
casper2.open("Y URL", function() {
RESPONSE
});
casper2.run();
}).then... >> It have to wait for response.
How can i do?
You have to use casper.then after the click. Here is some code:
var x = require('casper').selectXPath;
var url = 'http://stackoverflow.com/';
var casper = require('casper').create({
verbose: false,
logLevel: 'debug'
});
casper.test.comment('Starting Testing');
casper.start(url, function() {
//console.log(this.getCurrentUrl());
this.test.assert(
this.getCurrentUrl() === url, 'url is the one expected'
);
this.test.assertHttpStatus(200, + 'site is up');
casper.waitForSelector(x("//a[normalize-space(text())='Questions']"),
function success() {
this.test.assertExists(x("//a[normalize-space(text())='Questions']"));
this.click(x("//a[normalize-space(text())='Questions']"));
},
function fail() {
this.test.assertExists(x("//a[normalize-space(text())='Questions']"));
}
);
casper.then(function() {
//console.log(this.getCurrentUrl());
this.test.assertUrlMatches("http://stackoverflow.com/questions",
"clicked through to questions page");
});
casper.thenOpen('http://reddit.com', function() {
this.test.assertUrlMatches("http://www.reddit.com/",
"On Reddit");
});
});
casper.run(function() {
this.echo('finished');
this.exit();
});
Basically it goes to stackoverflow.com, waits until the Questions button is loaded, clicks on it and checks whether the redirected url is valid.
You can uncomment //console.log(this.getCurrentUrl()); if you want to see the specific url.
Now lets say you want to go to to an entirely new page, we can do use thenOpen api.
I highly recommend you read this blog: http://blog.newrelic.com/2013/06/04/simpler-ui-testing-with-casperjs-2/
I am filling out and submitting a form using PhantomJS and then outputting the resulting page. The thing is, I have no idea if this thing is being submitted at all.
I print the resulting page, but it's the same as the original page. I don't know if this is because it redirects back or I didn't submit it or I need to wait longer or or or. In a real browser it sends a GET and receives a cookie, which it uses to send more GETS before eventually receiving the final result - flight data.
I copied this example How to submit a form using PhantomJS, using a diferent url and page.evaluate functions.
var page = new WebPage(), testindex = 0, loadInProgress = false;
page.onConsoleMessage = function(msg) {
console.log(msg);
};
page.onLoadStarted = function() {
loadInProgress = true;
console.log("load started");
};
page.onLoadFinished = function() {
loadInProgress = false;
console.log("load finished");
};
var steps = [
function() {
//Load Login Page
page.open("http://www.klm.com/travel/dk_da/index.htm");
},
function() {
//Enter Credentials
page.evaluate(function() {
$("#ebt-origin-place").val("CPH");
$("#ebt-destination-place").val("CDG");
$("#ebt-departure-date").val("1/5/2013");
$("#ebt-return-date").val("10/5/2013");
});
},
function() {
//Login
page.evaluate(function() {
$('#ebt-flightsearch-submit').click() ;
# also tried:
# $('#ebt-flight-searchform').submit();
});
},
function() {
// Output content of page to stdout after form has been submitted
page.evaluate(function() {
console.log(document.querySelectorAll('html')[0].outerHTML);
});
}
];
interval = setInterval(function() {
if (!loadInProgress && typeof steps[testindex] == "function") {
console.log("step " + (testindex + 1));
steps[testindex]();
testindex++;
}
if (typeof steps[testindex] != "function") {
console.log("test complete!");
phantom.exit();
}
}, 50);
The site of interest is rather complicated to scrape. I logged the HTTP traffic from the US KLM site and got this:
GET /travel/us_en/apps/ebt/ebt_home.htm?name=on&ebt-origin-place=New+York+-+John+F.+Kennedy+International+%28JFK%29%2CNew+York&ebt-destination-place=Paris+-+Charles+De+Gaulle+Airport+%28CDG%29%2C+France&c%5B0%5D.os=JFK&c%5B0%5D.ost=airport&c%5B0%5D.ds=CDG&c%5B0%5D.dst=airport&c%5B1%5D.os=CDG&c%5B1%5D.ost=airport&c%5B1%5D.ds=JFK&inboundDestinationLocationType=airport&redirect=no&chdQty=0&infQty=0&c%5B0%5D.dd=2013-07-31&c%5B1%5D.dd=2013-08-14&c%5B1%5D.format=dd%2Fmm%2Fyyyy&flex=true&ebt-cabin-class=ECONOMY&adtQty=1&goToPage=&cffcc=ECONOMY&sc=false HTTP/1.1
Your injected values for the form elements are not what their server is looking for.
Inside page.evaluate(), you are sandboxed, but the sample code includes a hook to get sandboxed console activity onto the external console. For other debugging, you can also include object inspectors, etc., but they have to be injected into the page or part of the code passed into evaluate().
I am developing a chrome extension and I have an iframe which I use in my extension. This is what I do with my iframe
"When I drag and drop a image to my iframe I handle the drop event in one of the content scripts and pass that function call my extension code. There I create a xmlhttprequest object and then send the URL of the image to a php file in my server."
This is what is happening. I get a readyState of "4" but there is no POST request going out of my browser. I checked with the "NETWORK" tab in the browser but there is no POST request going out of the browser (I have listed my site in the permissions section of the manifest file).
This is my code --.>
JScript.js(One of the content scripts )
drop: function(event, ui) {
var imgurl=$(ui.draggable).attr('src');
imgurl="IMGURL="+imgurl;
_post("www.somedomain.come/testing.php",imgurl,function(result){ alert("success")});
}
This is my proxy in the same content script-->
_post = function(url, data, callback)
{
console.log("sending post");
chrome.extension.sendRequest({
msgType:'post',
data: data,
url:url
}, function(response){
alert(response);
});
}
This my OnRequest function handler in background.html -->
chrome.extension.onRequest.addListener(function(request, sender, sendResponse){
if (request.msgType === 'post') {
alert("Now in OnRequest function");
// console.log("Now in Onrequest Function");
alert("Url: "+request.url + "\n Data : "+ request.data);
ajaxcallingfunction(request);
alert("completed the ajax call");
sendResponse("success");
}
});
var ajaxcallingfunction = function(request){
var xhr = new XMLHttpRequest();
xhr.open("POST",request.url, false);
xhr.onreadystatechange = function(){
alert(xhr.readyState);
if (xhr.readyState == 4) {
alert(xhr.readyState);
}
}
xhr.send(request.data);
alert("after xhr call");
};
You have http:// in front of your url, right?
xhr.readyState doesn't tell much, it just means that it is done. Check out what's inside xhr.status, it would contain error code. If everything is ok it should be 200:
xhr.onreadystatechange = function(){
if (xhr.readyState == 4) {
alert(xhr.status);
}
}