I am running Selenium Firefox Webdriver on Python for webscraping and when I am going around diferent pages, some of those have some mechanism that open new windows, something in the way of this:
$(function(){
window.open(url, windowName[, windowFeatures]);
});
And it is some kind of malicious webpage that keeps opening random pages on new windows and after some minutes my PC runs out of memory and crashes.
So want I want is to load some feature on the webdriver so it doesn't allow pages to open new windows.
I have tryed not to load JS but this feature is no longer working I guess.
Also, if you know some option or preference to ignore script tags I would like to know it.
Thanks in advice.
Try loading a JS file just in your tests which overwrites the window.open function. Something like:
(function(){
window.open = function() { return false; }
})();
Notice this is an immediatly invoked function exectution.
Related
I need to have selenium automation download a file using firefox.
The automation successfully clicks the download, but a MIME opens to ask what to do. I need selenium to ignore this and just download the file.
Everything I have read days I should be able to use a function like this
const firefoxOption = new firefox.Options().set_preference(
'browser.helperApps.neverAsk.saveToDisk',
'application/zip,text/csv,text/txt',
);
But when I run this, it fails before it ever gets to clicking the download because this piece throws an error.
TypeError: (intermediate value).set_preference is not a function
what am I doing wrong??
ALL answers listed on SO show this exact code snippet and it is not working.
const firefoxOption = new firefox.Options().setPreference(
'browser.helperApps.neverAsk.saveToDisk',
'application/zip,text/csv,text/txt',
);
That method is for python
https://www.selenium.dev/selenium/docs/api/javascript/module/selenium-webdriver/firefox_exports_Options.html
I'm using phantomjs and webdriverio to fetch and render a webpage that's loaded by javascript, then save it to be parsed later by cheerio.
Here's the code for that:
import phantomjs from 'phantomjs-prebuilt'
const webdriverio = require('webdriverio')
const wdOpts = {
desiredCapabilities: {
browserName: 'phantomjs'
}
}
async parse (parseUrl) {
return phantomjs.run('--webdriver=4444').then(program => {
return webdriverio.remote(wdOpts)
.init()
.url(parseUrl)
.waitForExist('.main-ios', 100000)
.pause(5000)
.getHTML('html', true)
.then((html) => {
program.kill()
return html
})
})
}
Even though I call program.kill() I notice that the phantomjs in the list of processes, and it does use up quite a bit of RAM and CPU.
I'm wondering why the process doesn't terminate.
.close() just closes the window. There is a known bug, if it is the last window it stays open.
.quit() should do it, but there are issues associated with that as well.
PhantomJS bug report: https://github.com/detro/ghostdriver/issues/162
someone has a decent workaround posted at the bottom of that thread:
https://github.com/SeleniumHQ/selenium/issues/767#issuecomment-140367536
this fix shoots a SIGTERM to end it: (In python, but might be usefull)
# assume browser = webdriver.PhantomJS()
browser.service.process.send_signal(signal.SIGTERM)
browser.quit()
I like to just open a Docker container with my automation, and run it in there. Docker closes it up for me, however that is prolly out of scope for what you want to do.. i would recommend the above SIGTERM+quit method.
PhantomJS is a 2 component product. There is the Javascript which runs on the client (Whether web or other Script) side as part of your code. Then there is the part that runs as a server-side application (The command line call)
It has been my experience with PhantomJS that when an error is encountered, the PHantomJS server side "hangs" but is unresponsive. If you can update your call to this script to provide output logging, you may b able to see what the error is that PhantomJS application is encountering.
phantomjs /path/to/script/ > /path/to/log/file 2>&1
Hope this Helps! If you'd like me to clarify anything, or elaborate I'm happy to update my answer, just let me know in a comment, Thanks!
I've written a website scraper for use in a project.
I'm controlling Firefox through Sahi using Mink to visit each site and interact with any elements where necessary. I've managed to get this working perfectly on all sites I've tried except for one...
I'm trying to get the markup from https://www.o2.co.uk/shop/phones/
I'm using the exact same code for this page, as I have for all others:
// Configure driver
$this->driver = new \Behat\Mink\Driver\SahiDriver('firefox',
new \Behat\SahiClient\Client(
new \Behat\SahiClient\Connection(null, CRAWL_SERVER, 9999)
)
);
// Init session:
$this->session = new \Behat\Mink\Session($this->driver);
// Start session:
$this->session->start();
// Open the url
$this->session->visit($config['url']);
// Get the markup from the page
$markup = $this->session->getPage()->getContent();
When I use this code to attempt to get the markup from https://www.o2.co.uk/shop/phones/ Mink just seems to hang, waiting for something to happen.
It would seem that maybe something on this page is preventing either Sahi or Mink from returning the markup. I've also tried running other functions instead of getContent(), such as $this->session->wait(2000); and attempting to search through getPage using the find command.
If anyone has any idea as to why this is happening I would be very interested in finding out why and how I can make this work.
tl;dr
Why is Mink/Sahi timing out on this site?
I'd like to reopen the question posed here and here about testing file uploading within Nightwatch.js which uses selenium.
Both links have the recommended solution of setting the value of the file input element as the url. In my use case, I've been unable to get this to work. Even setting the value tag manually, outside of nightwatch, of the input where type="file", does not change the url. I've tried this on Chrome, Firefox, and IE10, within the dev tools.
An alternative solution I've looked at was trying to emulate the entire file upload process keystrokes. This would follow the path of clicking the file upload button, typing the path, and typing enter. This would be done through the .click and .key methods. However, you lose focus of the actual file upload window, which delays the keystrokes until that window is closed. Other developers have seemed to be able to fix this solution directly in selenium using the .findElement and .sendKeys methods in java, but I could not figure out how to do this within javascript and nightwatch itself.
Any ideas?
// My test
module.exports = {
"Standard File Upload" : function (browser) {
browser
.url("http://localhost:3000")
.waitForElementVisible('body', 1000)
.waitForElementVisible('input[type="file"]', 1000)
.setValue('input[type="file"]','http://localhost:3000/testfile.txt')
.click('#submit')
.pause(1000)
.assert.containsText('h3', 'File Uploaded Successfully')
.end();
}
};
// http://localhost:3000/testfile.txt was tested manually in the file upload window and worked successfully
<!-- My input tag -->
<input id="fileUpload" type="file" name="textfile"/>
There were two seperate issues with my setValue() method implementation.
Using the --verbose tag in the nightwatch command led me to an issue
where it was not actually finding the input tag during the
setValue(), however it was during the waitForElementVisible().
Changing input[type="file"] to input#fileUpload solved this
issue.
Secondly, the following ways of describing the path were not working...
'textfile.txt'
'http://localhost:3000/testfile.txt' (Will work if typed manually into file upload window)
What did work was using require('path').resolve(__dirname + '/testfile.txt')
Take a look here to see the discussion that led to the fix. Thanks goes out to richard-flosi.
The working code:
module.exports = {
"Standard File Upload" : function (browser) {
browser
.url("http://localhost:3000")
.waitForElementVisible('body', 1000)
.waitForElementVisible('input#fileUpload', 1000)
.pause(1000)
.setValue('input#fileUpload', require('path').resolve(__dirname + '/testfile.txt')) // Works
// .setValue('input#fileUpload', "testfile.txt") // Will not work
// .setValue('input#fileUpload', "http://localhost:3000/testfile.txt") // Will not work
// .setValue('input[type="file"]', require('path').resolve(__dirname + '/testfile.txt')) // Will not work
.click('#submit')
.pause(1000)
.assert.containsText('h3', 'File Uploaded Successfully')
.end();
}
};
I'm not sure why you're having these issues, maybe check to see if you are using the latest version of selenium server and nightwatch. This code works for me 100% in Chrome, Safari, Firefox, IE7/8/9/10/11 (not tested in IE6 but assume it as well).
driver.setValue('input#fileUpload', __dirname + '\\testfile.txt')
In my case, I had an additional problem because the file I was trying to upload was too high up in my directory structure.
As soon as I moved the file to the same level (or in a subdirectory of) the actual test files, things worked.
From a script living in my page-objects folder:
// No dice:
var fullPath = require('path').resolve(__dirname + '/../../somefile.pdf');
// Works:
var fullPath = require('path').resolve(__dirname + '/../somefile.pdf');
this.setValue('input#fileUpload', fullPath);
I am working on writing a story for a bdd framework which uses jbehave/selenium/webdriver and am having a problem where the test encounters an error while running the story but appears to be fine when running manually. I'm having a problem where javascript for the functionality I'm testing behaves slightly different when I'm running tests manually on firefox vs through selenium web driver on the same system/version of firefox and this difference is causing a js error.
I've debugged and basically the root of the problem appears to be that var request_XML_container = $('div_appendpoint_id'); returns something different when I'm running the test manually vs when I run through the bdd framework.
var request_XML_container = $('div_appendpoint_id');
request_XML_container.innerHTML = encoded_xml_from_request;
var pos = method_to_get_position('id_of_place_div_should_be_appended_to');
// JS exception is thrown saying that style is not defined **ONLY**
// when running through web driver. Running test manually on
// same system and same browser works fine.
request_XML_container.style.left = (pos[0] - 300) + 'px';
request_XML_container.style.top = (pos[1] + 25) + 'px';
request_XML_container.style.display = "block";
Why this would work fine when running manually that var request_XML_container = $('div_appendpoint_id'); would return an item with style defined, but when running through webdriver that the style attribute of the element would not be defined?
UPDATE: I had originally thought that this was updating an iframe, but I read the markup wrong and the iframe I saw is a sibling - not a parent - of the element where the response is being appended to. I'm trying to simply append the response to a div. To be honest, this only makes things more confusing as grabbing a div by id should be pretty straight forward and I'm now sure why webdriver would be producing a different return element in this situation.
UPDATE 2: Steps to reproduce and information about the system I'm on:
Use webdriver to navigate to this url: http://fiddle.jshell.net/C3VB5/11/show/
Have webdriver click the button. It should not work
Run your test again, but pause put a breakpoint at your code to click the driver
Click the button on the browser that webdriver opened. It should not work
Refresh the browser page on the browser that webdriver opened. Now, it should work.
System details:
OS : OS X 10.8.5 (12F37)
IDE : Eclipse Kepler: Build id: 20130614-0229
Browser (used manually and by webdriver) : Firefox 23.0.1
Selenium version: 2.35.0
UPDATE 3: I have provided this maven project on github to aid in reproducing: https://github.com/dkwestbr/WebdriverBug/tree/master/Webdriver
Synopsis/tl:dr; Basically, in certain situations it appears as though webdriver is overwriting the '$()' javascript method with a method that does not return an HTMLElement with innerHTML or style defined (among other things). This post details the issue and how to reproduce.
I have opened this ticket to track the issue: https://code.google.com/p/selenium/issues/detail?id=6287&thanks=6287&ts=1379519170
I have confirmed that this is a bug with the Thucydides framework (understandable since they still aren't at a 1.0 release).
Issue can be tracked here: https://java.net/jira/browse/THUCYDIDES-203