Capybara, Poltergeist, PhantomJS, evaluate_script, callback - phantomjs

I am using the following code, which uses the imagesLoaded package with a callback to tell me when an element with a particular csspath has finished loading all of its images:
imagesLoadedScript = "imagesLoaded( '#{csspath}', { background: true }, function(message) { console.log('PHANTOM CLIENT REPORTING: #{csspath} Images Loaded'); return message; })"
imagesLoadedScript = imagesLoadedScript.strip.gsub(/\s+/,' ')
#session.evaluate_script(imagesLoadedScript)
The timing of the console.log statement, on inspection of PhantomJS logs with debug on, indicates that Capybara/Poltergiest is not waiting for the images to load, as expected, before it moves on to the next statement. I also cannot return a true (or false) value from inside the callback as I would like.
Capybara responds with
{"command_id":"678f1e2e-4820-4631-8cd6-413ce6f4b66f","response":"(cyclic structure)"}
Anyone have any ideas on how to return a value from inside a callback in a function executed via evaluate_script?
Many thanks.

TLDR; You can't
evaluate_script doesn't support asynchronous functions - you must return the result you want from the function passed in. One way to do what you want would be to execute the imagesLoaded script and have the callback set a global variable, and then loop on an evaluate_script fetching the result of the global until it's what you want - A very basic implementation would be something like
imagesLoadedScript = "window.allImagesLoaded = false; imagesLoaded( '#{csspath}', { background: true }, function() { window.my_images_loaded = true })"
#session.execute_script(imagesLoadedScript)
while !#session.evaluate_script('window.allImagesLoaded')
sleep 0.05
end
Obviously this could be made more flexible with a timeout ability, etc.
A second option would to write a custom capybara selector type for images with a loaded filter, although with the need for background image checking it would become pretty complicated and probably too slow to be useful.

Just in case someone finds this later.
I did roughly what Thomas Walpole suggested in his answer, in a more roundabout fashion, but taking advantage of Poltergeist's inherent waiting capabilities;
#to check that the target has loaded its images, run images loaded
#after a small timeout to allow the page to get the images
#append a marker div to the dom if the images have successfully loaded
imagesLoadedScript = "var item = document.querySelector('#{csspath}');
window.scroll(0, item.offsetTop);
function imagesDone(path, fn) {
imagesLoaded( path, function(instance) {
console.log('PHANTOM CLIENT REPORTING: ' + path + ' Images Loaded');
fn(true);
})
}
setTimeout(function(){
imagesDone('#{csspath}', function(done) {
var markerDiv = document.createElement('div');
markerDiv.id = 'ImagesLoadedMarker';
document.getElementsByTagName('html')[0].appendChild(markerDiv);
});
}, 1000)"
#then we strip the new lines and spaces that we added to make it readable
imagesLoadedScript = imagesLoadedScript.strip.gsub(/\s+/,' ')
#now we just execute the script as we do not need a return value
#session.execute_script(imagesLoadedScript)
#then we check for the marker, using capybara's inbuilt waiting time
if #session.has_xpath? "//*[#id ='ImagesLoadedMarker']"
Rails.logger.debug "!!!!! PhantomClient: Images Loaded Reporting: #{csspath} Images Loaded: Check Time #{Time.now} !!!!!"
#session.save_screenshot(file_path, :selector => csspath)
else
Rails.logger.debug "!!!!! PhantomClient: Images Loaded Reporting: #{csspath} Images NOT Loaded: Check Time #{Time.now} !!!!!"
#session.save_screenshot(file_path, :selector => csspath)
end

Related

Vue component method behaving weirdly - loader animation not working while data is processing

I'm trying to figure out why this happens - loader for long processing of data does not show.. only after the processing is done. Huge (few thousands items) object of key-value items and want them to make filterable - that works - but takes few seconds. I'm using VueJS 2.
I wanted to show "please wait" message while it runs, using the isworking value. I have a span with v-if="isworking", defined with value false as initial value.
On first line I set the this.isworking prop, but instead of seeing the "please wait", the function hangs for few seconds to do a search, and THEN sets the isworking prop to true - I tried that by commenting the last isworking=false - can't figure out why it waits to change it to true for the huge processing to end.
That window.deaccent method is fn to replace all accented characters in string with basic ansii chars, nothing special.
In template, I have a simple:
<form #submit="searchmath">
<span v-if="working">please wait</span>
<input v-model=...>
<div v-for="(item,index) in searchmatchitems"> ... </div>
</form>
Method in component:
searchmatch: function($event){
this.isworking = true;
this.$forceUpdate(); // tried also this, does not help
$event.stopPropagation();$event.preventDefault();
try{
var searchid = window.deaccent(this.search_string.toLowerCase());
var searchobj = this.cdata;
let result = Object.keys(this.cdata).filter(function(el,i,c){
var elk = window.deaccent(el).toLowerCase();
var elv = window.deaccent(searchobj[el]);
return elk.indexOf(searchid) > -1 || elv.indexOf(searchid) > -1;
}, searchid);
this.searchmatchitems = result;
this.isworking = false;
} catch(e){ console.log(e); this.isworking = false; return [];}
}
I also tried moving the event.preventDefault() to bottom, just to be sure it does not affect anything, but no luck.
That cdata is a simple key-value object with many props like this, counting about 4000 items
data: {
cdata: {
"lorem": "aa",
"ipsum": "bc",
"dolor": "de",
....
},
isworking: false,
....
}
2 issues here.
First, you use working in your template, and isworking in your data and code.
Second, your method does not call any async code, so it sets isworking to true, does work, and then sets isworking to false. You template will never have a chance to refresh in this case, and the UI will freeze until the method returns.
For instance, if you made an async call to a network endpoint, and then set isworking to false in the callback, you would get the results you are expecting.
If you have long running code and wish to prevent the UI from freezing, you will need to use a web worker thread

How do you poll for a condition in Intern / Leadfoot (not browser / client side)?

I'm trying to verify that an account was created successfully, but after clicking the submit button, I need to wait until the next page has loaded and verify that the user ended up at the correct URL.
I'm using pollUntil to check the URL client side, but that results in Detected a page unload event; script execution does not work across page loads. in Safari at least. I can add a sleep, but I was wondering if there is a better way.
Questions:
How can you poll on something like this.remote.getCurrentUrl()? Basically I want to do something like this.remote.waitForCurrentUrlToEqual(...), but I'm also curious how to poll on anything from Selenium commands vs using pollUntil which executes code in the remote browser.
I'm checking to see if the user ended up at a protected URL after logging in here. Is there a better way to check this besides polling?
Best practices: do I need to make an assertion with Chai or is it even possible when I'm polling and waiting for stuff as my test? For example, in this case, I'm just trying to poll to make sure we ended up at the right URL within 30 seconds and I don't have an explicit assertion. I'm just assuming the test will fail, but it won't say why. If the best practice is to make an assertion here, how would I do it here or any time I'm using wait?
Here's an example of my code:
'create new account': function() {
return this.remote
// Hidden: populate all account details
.findByClassName('nextButton')
.click()
.end()
.then(pollUntil('return location.pathname === "/protected-page" ? true : null', [], 30000));
}
The pollUntil helper works by running an asynchronous script in the browser to check a condition, so it's not going to work across page loads (because the script disappears when a page loads). One way to poll the current remote URL would be to write a poller that would run as part of your functional test, something like (untested):
function pollUrl(remote, targetUrl, timeout) {
return function () {
var dfd = new Deferred();
var endTime = Number(new Date()) + timeout;
(function poll() {
remote.getCurrentUrl().then(function (url) {
if (url === targetUrl) {
dfd.resolve();
}
else if (Number(new Date()) < endTime) {
setTimeout(poll, 500);
}
else {
var error = new Error('timed out; final url is ' + url);
dfd.reject(error);
}
});
})();
return dfd.promise;
}
}
You could call it as:
.then(pollUrl(this.remote, '/protected-page', 30000))
When you're using something like pollUntil, there's no need (or place) to make an assertion. However, with your own polling function you could have it reject its promise with an informative error.

How to use callbacks in phantomjs

I am using phantomjs for testing one web application. but i am facing problem with page load means sometimes phantom script executed but dom element is not loaded. How to use callbacks for sorting this kind of issues
resourceReceived(request),resourceRequested(resource),resourceError(resource)
If you want to execute code after the page has finished loading, use this:
page.onLoadFinished = function()
{
// function body
var pageTitle = page.evaluate(function() {
console.log('Page Name: ' + document.title);
return document.title;
});
};

Grab the resource contents in CasperJS or PhantomJS

I see that CasperJS has a "download" function and an "on resource received" callback but I do not see the contents of a resource in the callback, and I don't want to download the resource to the filesystem.
I want to grab the contents of the resource so that I can do something with it in my script. Is this possible with CasperJS or PhantomJS?
This problem has been in my way for the last couple of days. The proxy solution wasn't very clean in my environment so I found out where phantomjs's QTNetworking core put the resources when it caches them.
Long story short, here is my gist. You need the cache.js and mimetype.js files:
https://gist.github.com/bshamric/4717583
//for this to work, you have to call phantomjs with the cache enabled:
//usage: phantomjs --disk-cache=true test.js
var page = require('webpage').create();
var fs = require('fs');
var cache = require('./cache');
var mimetype = require('./mimetype');
//this is the path that QTNetwork classes uses for caching files for it's http client
//the path should be the one that has 16 folders labeled 0,1,2,3,...,F
cache.cachePath = '/Users/brandon/Library/Caches/Ofi Labs/PhantomJS/data7/';
var url = 'http://google.com';
page.viewportSize = { width: 1300, height: 768 };
//when the resource is received, go ahead and include a reference to it in the cache object
page.onResourceReceived = function(response) {
//I only cache images, but you can change this
if(response.contentType.indexOf('image') >= 0)
{
cache.includeResource(response);
}
};
//when the page is done loading, go through each cachedResource and do something with it,
//I'm just saving them to a file
page.onLoadFinished = function(status) {
for(index in cache.cachedResources) {
var file = cache.cachedResources[index].cacheFileNoPath;
var ext = mimetype.ext[cache.cachedResources[index].mimetype];
var finalFile = file.replace("."+cache.cacheExtension,"."+ext);
fs.write('saved/'+finalFile,cache.cachedResources[index].getContents(),'b');
}
};
page.open(url, function () {
page.render('saved/google.pdf');
phantom.exit();
});
Then when you call phantomjs, just make sure the cache is enabled:
phantomjs --disk-cache=true test.js
Some notes:
I wrote this for the purpose of getting the images on a page without using the proxy or taking a low res snapshot. QT uses compression on certain text file resources and you will have to deal with the decompression if you use this for text files. Also, I ran a quick test to pull in html resources and it didn't parse the http headers out of the result. But, this is useful to me, hopefully someone else will find it so, modify it if you have problems with a specific content type.
I've found that until the phantomjs matures a bit, according to the issue 158 http://code.google.com/p/phantomjs/issues/detail?id=158 this is a bit of a headache for them.
So you want to do it anyways? I've opted to go a bit higher to accomplish this and have grabbed PyMiProxy over at https://github.com/allfro/pymiproxy, downloaded, installed, set it up, took their example code and made this in proxy.py
from miproxy.proxy import RequestInterceptorPlugin, ResponseInterceptorPlugin, AsyncMitmProxy
from mimetools import Message
from StringIO import StringIO
class DebugInterceptor(RequestInterceptorPlugin, ResponseInterceptorPlugin):
def do_request(self, data):
data = data.replace('Accept-Encoding: gzip\r\n', 'Accept-Encoding:\r\n', 1);
return data
def do_response(self, data):
#print '<< %s' % repr(data[:100])
request_line, headers_alone = data.split('\r\n', 1)
headers = Message(StringIO(headers_alone))
print "Content type: %s" %(headers['content-type'])
if headers['content-type'] == 'text/x-comma-separated-values':
f = open('data.csv', 'w')
f.write(data)
print ''
return data
if __name__ == '__main__':
proxy = AsyncMitmProxy()
proxy.register_interceptor(DebugInterceptor)
try:
proxy.serve_forever()
except KeyboardInterrupt:
proxy.server_close()
Then I fire it up
python proxy.py
Next I execute phantomjs with the proxy specified...
phantomjs --ignore-ssl-errors=yes --cookies-file=cookies.txt --proxy=127.0.0.1:8080 --web-security=no myfile.js
You may want to turn your security on or such, it was needless for me currently as I'm scraping just one source. You should now see a bunch of text flowing through your proxy console and if it lands on something with the mime type of "text/x-comma-separated-values" it'll save it as data.csv. This will also save all the headers and everything, but if you've come this far I'm sure you can figure out how to pop those off.
One other detail, I've found that I've had to disable gzip encoding, I could use zlib and decompress data in gzip from my own apache webserver, but if it comes out of IIS or such the decompression will get errors and I'm not sure about that part of it.
So my power company won't offer me an API? Fine! We do it the hard way!
Did not realize I could grab the source from the document object like this:
casper.start(url, function() {
var js = this.evaluate(function() {
return document;
});
this.echo(js.all[0].outerHTML);
});
More info here.
You can use Casper.debugHTML() to print out contents of a HTML resource:
var casper = require('casper').create();
casper.start('http://google.com/', function() {
this.debugHTML();
});
casper.run();
You can also store the HTML contents in a var using casper.getPageContent(): http://casperjs.org/api.html#casper.getPageContent (available in lastest master)

How to precompile processingjs sketch to speed load times?

The loading times of my processingjs webpage are getting pretty hairy. How can I precache the compilation of processing to javascript?
It would be acceptable for my application to compile on first entering the webpage (maybe keeping the result in the local store?) and then reuse the compilation on subsequent loads.
There's two ways to drive down load time as experienced by the user. The first is using precompiled sketches, which is relatively easy: github repo, or even just download the master branch using github's download button (https://github.com/processing-js/processing-js), and then look for the "./tools/processing-helper.html" file. This is a helper page that lets you run or compile sketches to the JavaScript source that Processing.js uses. You will still need to run this alongside Processing, since it ties into the API provided, but you can use the "API only" version for that. Take the code it generates, prepend "var mySketch = ", and then do this on your page:
<script src="processing.api.js"></script>
<script>
function whenImGoodAndReady() {
var mySketch = (function.....) // generated by Processing.js helper
var myCanvas = document.getElementById('mycanvas');
new Processing(myCanvas, mySketch);
}
</script>
Just make sure to call the load function when, as the name implies, you're ready to do so =)
The other is to do late-loading, if you have any sketches that are initially off-screen.
There's a "lazy loading" extension in the full download for Processing.js - you can include that on your page, and it will make sketches load only once they're in view. That way you don't bog down the entire page load.
Alternatively, you can write a background loader that does the same thing as the lazy loading extension: turn off Processing.init, and instead gather all the script/canvas elements that represent Processing sketches, then loading them on a timeout using something like
var sketchList = [];
function findSketches() {
/* find all script/canvas elements */
for(every of these elements) {
sketchList.append({
canvas: <a canvas element>,
sourceCode: <the sketch code>
});
}
// kickstart slowloading
slowLoad();
}
function slowLoad() {
if(sketchList.length>0) {
var sketchData = sketchList.splice(0,1);
try {
new Processing(sketchData.canvas, sketchData.sourceCode);
setTimeout(slowLoad, 15000); // load next sketch in 15 seconds
} catch (e) { console.log(e); }
}
}
This will keep slow-loading your sketches until it's run out.