PhantomJS process keeps running in background after calling program.kill() - selenium

I'm using phantomjs and webdriverio to fetch and render a webpage that's loaded by javascript, then save it to be parsed later by cheerio.
Here's the code for that:
import phantomjs from 'phantomjs-prebuilt'
const webdriverio = require('webdriverio')
const wdOpts = {
desiredCapabilities: {
browserName: 'phantomjs'
}
}
async parse (parseUrl) {
return phantomjs.run('--webdriver=4444').then(program => {
return webdriverio.remote(wdOpts)
.init()
.url(parseUrl)
.waitForExist('.main-ios', 100000)
.pause(5000)
.getHTML('html', true)
.then((html) => {
program.kill()
return html
})
})
}
Even though I call program.kill() I notice that the phantomjs in the list of processes, and it does use up quite a bit of RAM and CPU.
I'm wondering why the process doesn't terminate.

.close() just closes the window. There is a known bug, if it is the last window it stays open.
.quit() should do it, but there are issues associated with that as well.
PhantomJS bug report: https://github.com/detro/ghostdriver/issues/162
someone has a decent workaround posted at the bottom of that thread:
https://github.com/SeleniumHQ/selenium/issues/767#issuecomment-140367536
this fix shoots a SIGTERM to end it: (In python, but might be usefull)
# assume browser = webdriver.PhantomJS()
browser.service.process.send_signal(signal.SIGTERM)
browser.quit()
I like to just open a Docker container with my automation, and run it in there. Docker closes it up for me, however that is prolly out of scope for what you want to do.. i would recommend the above SIGTERM+quit method.

PhantomJS is a 2 component product. There is the Javascript which runs on the client (Whether web or other Script) side as part of your code. Then there is the part that runs as a server-side application (The command line call)
It has been my experience with PhantomJS that when an error is encountered, the PHantomJS server side "hangs" but is unresponsive. If you can update your call to this script to provide output logging, you may b able to see what the error is that PhantomJS application is encountering.
phantomjs /path/to/script/ > /path/to/log/file 2>&1
Hope this Helps! If you'd like me to clarify anything, or elaborate I'm happy to update my answer, just let me know in a comment, Thanks!

Related

WebdriverIO: Execution context is not available in detached frame

I'm trying to automate a flow that I've been doing at least once a day for the past 3 years. It's an unnecessarily convoluted process that is taking time out of my day. It's an internal proprietary system, so I can't give you any details about it, but suffice to say that I cannot change it. I have to try to automate it the way it is, and I'm stuck with the following error:
Execution context is not available in detached frame "https://..." (are you trying to evaluate?)
I'm using WebdriverIO with the default configuration, which uses Puppeteer internally. When I googled the error, I got a suggestion to add the following flags:
'--disable-web-security', '--disable-features=IsolateOrigins,site-per-process'
I did that, but it didn't help:
remote({
logLevel: 'trace',
capabilities: {
browserName: 'chrome',
'goog:chromeOptions': {
args: ['--disable-web-security', '--disable-features=IsolateOrigins,site-per-process']
}
}
})
I want to try it out in the REPL to do some more debugging, but I haven't figured out how to add those args when launching the REPL:
npx wdio repl chrome
Everything I've tried so far has failed. If someone knows how to do it, please let me know.
I don't do web automation normally and I chose WebdriverIO, since I have used it in the past for mobile automation. I'm wondering if this is a Puppeteer specific issue and if it would work in Selenium, but I'm not quite sure how to set that up. Puppeteer was just working out of the box.
Any guidance on how to get past this error would be greatly appreciated.
Turns out it was pretty simple:
browser.switchToFrame($('#iFrame'))
// Doing stuff inside the frame that causes the next page to show.
// Now that iFrame is gone and I have to switch out of it, to continue:
browser.switchToParentFrame()
// After the above call the execution context is not in a detached frame anymore.

Restart fixture when it fails

I wrote a test that passes 95% of the time, failing the other 5%. I still don't know why it fails (my guess is that components are not rendered correctly).
I implemented a page reload call to reload the page and try again, but it's not very reliable.
What's the best way to restart the fixture in case it fails?
Here's a sample test that intentionally fails to emulate my selector that works most of the time, but fails sometimes.
import { Selector } from 'testcafe';
const URL = 'https://www.youtube.com/watch?v=RWQtB6Xv01Q';
fixture `Portal Experience playback`
.page `${URL}`;
test('Testing YouTube', async t => {
await t.click(Selector('.wrong-selector')); // to emulate my unstable test
});
Results in
✖ Testing YouTube
1) The specified selector does not match any element in the DOM tree.
Is it possible to put the test in a for loop and have it break out of the loop in case the test passes?
The quarantine mode serves this purpose. In this mode, TestCafe will restart a failed test again until it passes or fails 3 times and will consider it failed only if it fails 3 times. Get more details about the quarantine mode in this article: Quarantine Mode.

How to prevent selenium from opening new firefox windows?

I am running Selenium Firefox Webdriver on Python for webscraping and when I am going around diferent pages, some of those have some mechanism that open new windows, something in the way of this:
$(function(){
window.open(url, windowName[, windowFeatures]);
});
And it is some kind of malicious webpage that keeps opening random pages on new windows and after some minutes my PC runs out of memory and crashes.
So want I want is to load some feature on the webdriver so it doesn't allow pages to open new windows.
I have tryed not to load JS but this feature is no longer working I guess.
Also, if you know some option or preference to ignore script tags I would like to know it.
Thanks in advice.
Try loading a JS file just in your tests which overwrites the window.open function. Something like:
(function(){
window.open = function() { return false; }
})();
Notice this is an immediatly invoked function exectution.

Nightwatch pipe all console messages to a file

I'm using nightwatch to run some QA tests on code I have little control over. The app consists of multiple windows being opened through the use of popups. Because it is difficult to know exactly when to try and grab the console within my tests, I'd like a way of piping all console messages from any window that is running within the context of selenium, to an output file so I can sift through it.
Right now I have this in my test
.getLog('browser', function (result) {
console.log(typeof result)
fs.writeFile("browser.log", JSON.stringify(result), function(err) {
if(err) {
return console.log(err);
}
console.log("Log Saved");
});
});
This may work for some cases, but sometimes the code I am hoping to catch happens on an unload event. In a situation like that, catching the error at the correct time proves challenging. I was hoping that nightwatch might expose some sort of event listener api where I could observe the browser.log file for example, and update the contents of the file automatically. Does such a feature exist? Is there a better way to do this?

testing existing pages with mocha-phantomjs

I'm not quite getting how to use PhantomJS and Mocha together, specifically through mocha-phantomjs.
I've read some tutorials (the one at tutsplus is quite helpful) and am not seeing how I can test external pages using Phantom and Mocha. I'm sure I just need a nudge.
In the tutorial the author creates a tests.js file with some DOM setup/manipulation, as well as some mocha assertions. Then, he creates a separate HTML file that loads the tests.js file and uses mocha-phantomjs to fire up phantom and run the tests.
That's where I'm a little confused, how the mochaPhantomJS.run() method actually does things behind the scenes, whether it knows to search the js file for a describe block and run all tests within, that sort of thing. I don't need chapter and verse, but a high-level summary would be ideal.
Also, if I want to test an outside page, how can I best do that? In the tutorial all the DOM investigation and testing is done on the test page. If I want to test a different page, do I change or setup my assertions differently? Should I call the PhantomJS API from those tests and point to an external site?
Mocha will run tests that have been specified in javascript that has been included in the html page that is launched. If you look at the example html page on mocha-phantomjs it expects the test definitions using describe/it calls to be in the test/mycode.js file. If you put something like
These tests are only testing what is in the main file and associated javascript, there isn't anything special that mocha-phantomjs provides to test external html files. If you want to test your own html files I think you can head in a couple of directions, I came up with these:
first option: Create a way to load the parts of you app that you want to test into the main testing html file. How to do this depends a lot on your application setup. It is probably well-suited for a modular system. Just include the javascript from your application and test it. Not so good for full-page-html tests.
second option: Open new windows with the pages to test from the main testing html file (from within phantom that is). You can open a page using window.open() just like a normal browser. I created a proof of concept like this:
describe('hello web', function () {
it('world', function (done) {
var windowReference = window.open("fileundertest.html");
// windowReference.onload didn't work for me, so I resorted to this solution
var intervalId = window.setInterval(function () {
if (windowReference.document && windowReference.document.readyState === 'complete') {
window.clearInterval(intervalId);
expect(windowReference.document.title).to.equal("File Under Test");
done();
} else {
console.log('not ready yet');
}
}, 10);
});
}
)
This solution is relatively simple, but has the same drawbacks as any page-loading solution: you never know when the page is fully initialized and have to resort to some sort of timeout/wait system to wait for the page to get into the correct state. If you want to test a lot of separate html files these delays start to add up. Additionally waiting for 'onload' on the page that I opened wouldn't work so I created my own function based on setInterval and a (non-perfect) test on the document that was being loaded. I found out there are differences in behavior between loading an html page from the filesystem and loading the same page via a web-server.
third option: Create a mocha test that you run nodejs-style from the command line, and launch phantomjs with a specific page as part of your mocha tests. This is what I'd say you need if your system really depends on html pages that are quite different from each other.
I quickly tested the third option, here is my test based on the example I found on the phantom page (which is an alternative solution to phantomjs that is used by mocha-phantomjs -- I've used neither for more than brief experiments so I cannot recommend which one to use)
'use strict';
var chai = require('chai'),
phantom = require('phantom');
var expect = chai.expect;
describe('hello node', function () {
it('world', function (done) {
phantom.create(function (ph) {
ph.createPage(function (page) {
page.open("http://www.google.com", function (status) {
console.log("opened google? ", status);
page.evaluate(function () { return document.title; }, function (result) {
console.log('Page title is ' + result);
ph.exit();
expect(result).to.equal("Google");
done();
});
});
});
});
});
}
)
While it is possible to test this way I'd say that maybe the overhead of the communication between the code in the phantom-world and the testing code in the nodejs world isn't worth it. You can of course move a lot of general functionality to a couple of helper functions, but you are still stuck with having to call page.evaluate() to perform specific tests on the page. The same issues with timeouts/waits as above apply.
As an aside: do already know CasperJS? Maybe it can be helpful for your setup should you choose to build something on 'plain' phantomjs.