I am using phantomjs to build a web crawler. My current crawler is a Python script using selenium, which requires a running Firefox browser. While Selenium is good for debugging scripts (as I can use firebug to inspect the webpage), it cannot be deployed on linux servers without GUI. So I am trying to translate my Python script to phantomjs.
To debug phantomjs scripts, I save the page source html and render a png screenshot to disk, and open it in Firefox to inspect the source page. I am wondering if there is a better way to do this, e.g. a plugin for phantomjs, etc.
This little hack is using a simple technique: It grabs the screen as PhantomJS or CasperJS sees it with captureBase64('png') and then it is POSTing the image into the receiving server which then sends it via socket.io to the browser which displays it is as inline image.
The source code is available on github:
https://github.com/maciejjankowski/flaming-octo-puss
I'm not sure if PhantomJS has it, but CasperJS does (and the latter adds some extra functions to PhantomJS)
and use remote debugging as others suggest
As for most of the JS scripts, it's not so trivial to debug phantomjs script, because there is no IDE/compiler behind this.
First, I higly suggest you to read the Troubleshooting section.
As said by torazaburo, your best option is to use Remote Debugging : scripts can be run in a Web Inspector interface-like console : step-in, step-out, break points, local variables... many classic debugger features are available. If you're familliar with Chrome developpers tools, it's pretty the same.
Though not a solution to your Phantomjs troubles,I think Selenium with Python bindings can be used very efficiently as a headless scraper in a Linux environment.
You can use PyVirtualDisplay, a Python wrapper for Xvfb and Xephyr to fake a display.PyVirtualDisplay needs Xvfb as a dependency. On Ubuntu, first install Xvfb:
sudo apt-get install xvfb
then install PyVirtualDisplay from Pypi:
pip install pyvirtualdisplay
Sample Selenium script in Python in a headless mode with PyVirtualDisplay:
#!/usr/bin/env python
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
# now Firefox will run in a virtual display.
# you will not see the browser.
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()
display.stop()
Related
I had redhat Linux server command line but there is no GUI for that and I need to run selenium code on this server with firefox. As I am new to selenium so I am not sure whether it will work or not.
For achieving this I had install Firefox on my Redhat linux machine successfully but not able to trigger that as it is showing no display found.
Is there any other way to achieve this for headless browsing, where virtually a firefox will be opened and selenium code can be executed without GUI. Is it possible to do??
You can use HtmlUnitDriver of selenium to use headless browsing but it will not open firefox and may be not loading all content to it's cache as same as firefox
Refer Below:-
https://code.google.com/p/selenium/wiki/HtmlUnitDriver
Another thing you can use is Phantomjs with selenium Webdriver. This is most using pattern by industries for headless browsing
Refer below:-
http://www.guru99.com/selenium-with-htmlunit-driver-phantomjs.html
Yes you can trigger selenium file using command line
http://www.tutorialspoint.com/testng/testng_executing_tests.htm
Bash file is running fine in windows for testng but it is not working in linux/mac
Now if still your firefox is not opening then it is basically issue of some setting with your OS with firefox
Refer that too :-
https://serverfault.com/questions/279036/open-terminal-via-ssh-run-firefox-display-not-found
May be you have problem with some permission in red hat
Hope it will help you :)
I have built a Polymer based application. I'd like to write some end-to-end tests (not unit tests, but user behavior integration tests) for it. How do I do this currently (May 2015)?
I spent the past few days looking into this problem. Despite the vast number of pages devoted to one related topic or another on the web, nothing documents a solution to this problem. I was able to piece together something that works. So here it is. Hope this is useful for those looking to write end-to-end tests for Polymer apps. I am sure a better solution will come about as Polymer, web components, and shadow DOM technologies mature.
Some details: Linux, want automate testing in a script (ideally headless), app consists of many Polymer elements and is served by and loads data from a Django server.
Failed attempt with PhantomJS
First, I tried using casperjs and phantomjs. phantomjs is headless and casperjs has very good navigation support, so I thought those would be a good combination to end up with. Unfortunately, phantomjs does not support HTML5 imports and the webcomponents.js polyfill does not seem to work on phantomjs.
Using Selenium
I ended up with a Selenium/ChromeDriver based solution, using the selenium python client. I did not test this with Firefox driver so I don't know if that works. Here is what you need to do:
To make things easier, create a directory to put stuff in:
mkdir selenium
Install google chrome. I did this via the Google website and downloaded the linux version. Then, download selenium server 2.45 and chromedriver 2.15, into the selenium directory. E.g.
$ ls selenium/
chromedriver selenium-server-standalone-2.45.0.jar
Then, install Python Selenium API
$ pip install selenium
Run a simple test:
$ cd selenium
$ cat > test.py
from selenium import webdriver
driver = webdriver.Chrome('./chromedriver')
driver.get("http://localhost:8000/myapp/")
driver.implicitly_wait(3)
print driver.title
content = driver.find_element_by_css_selector('myelement::shadow h3')
print content.text
driver.close()
$ xvfb-run python test.py
(xvfb is necessary to make the running of test.py headless)
test.py prints out the content of the h3 element in a custom Polymer element called myelement. E.g. if DOM looks like
<myelement>
<h3>Hello World</h3>
</myelement>
Then test.py prints "Hello World".
The h3 element appears in the shadow DOM of myelement. Chrome dev tool lists the CSS selector for this h3 as "myelement #shadow-root h3". Using Selenium, you can access this h3 using "myelement::shadow h3" as the CSS selector.
Tests and Test Data
I organized my tests as Python unittest test cases, and wrote a test driver script. The driver script forks, creates a Django dev server in the child process, and runs "python -m unittest" in the parent process. Each test case uses Python selenium API to connect to the Django dev server. In the setUp and tearDown methods of each test case, I inject test data into the database using my Django model classes.
I run the driver script under xvfb -- "xvfb-run python driver.py" -- to make it headless.
Dealing with Ajax and two-way bindings
My Polymer app uses ajax to load data and two-way bindings to render HTML templates. Polymer also renders templates and updates the DOM asynchronously. In my test cases, I depend on Selenium's conditional wait to detect when data loading completes and DOM re-renders. Implicit waiting (which isn't a good idea anyways) did not work for me for some reason; the implicit wait just returns immediately. I also updated my HTML templates to be more testable -- adding DOM IDs and unique text or CSS selectors to differentiate different stages of the app.
Caveats
A button with only as its inner HTML becomes un-clickable using Selenium. If you have such a button, use ActionChains to click it:
chain = ActionChains()
chain.move_to_element(element)
chain.click()
chain.perform()
I know I'll get crap for this but it's worth it if someone can help.
I can't find a guide, tutorial, or instructions anywhere for installing web driver on a windows machine. I've got a site running on a homestead vagrant box and need a way of running acceptance tests locally. I had tests setup using PhpBrowser but those don't simulate JavaScript. If anyone knows of a guide to do this or a better way to run acceptance tests it would help immensely. I've got Ajax calls so PhpBrowser and resources like it won't work.
Thanks!
Just download the selenium webserver the jar file and run it in a command shell with java -jar selenium-server-standalone-2.43.1.jar. The selenium server will now listen on the default port 4444. If you run your test it should work as expected. Keep in mind that selenium webserver opens firefox and uses it for testing. It's pretty useful for watching the test cases.
If you want a headless test (no visible browser) you need to download phantomjs. Unpack it and run the phantomjs.exe with --webdriver=4444 as an argument (so phantomjs.exe --webdriver=4444).
Download and Run the selenium-server-standalone-2.43.1.jar as stated in the comment. I had to add firefox_binary: C:\Program Files (x86)\Mozilla Firefox\firefox.exe under capabilities: in my acceptance.suite.yml file. I also added it to the path variable but I'm not sure if that made a difference. Had to add the firefox_binary to make it work.
--- TL;DR
At this point I suggest everyone to tied their Continuous Integration server/service to https://ghostinspector.com/
OLD QUESTION
after three days googling and testing I gave up, and I need help.
My objective is allow my co-workers to record one or more tests with Selenium IDE. Export them, upload them into a server, and get this server running these tests using the webdriver with htmlunit. As we build or fix the app, we will upload the tests to make out test library.
Record a test with Selenium IDE is okay. But getting it running is the problem. The machine we intend to let the tests is an linux amazon server. No front-end stuff, no kde, gtk, so no firefox, chrome, etc... This is why I've specified the htmlunit driver.
So far I wasn't able to get this task running even into my machine - Ubuntu 12.04 x86_64.
I downloaded the selenium-server tarball, and tried running:
java -jar selenium-server.jar -htmlSuite "*webdriver" "our.site.org" "/path/to/testsuite1.html" "/path/to/report1.html"
No success. Even changing the "*webdriver" (using other pops-up a browser screen).
I've tried running the server and the standalone server and connecting via browser.
I've tried PHP bindings by facebook.
I've tried PHPUnit and Testing Selenium classes - along with their respectives exported scripts from Selenium Formatters.
I really do not know where I'm slipping. Can anyone give me a safe direction, tutorial, etc, to follow with?
--- EDIT
Okay, my question may be resumed to:
What si the command line that would allow me to run selenese scripts with selenium-server, using the HtmlUnit driver?
Are you using Continuous Integration?
If so, you should consider getting a plugin to have your CI software run the Selenium tests. Works like a charm for me with Jenkins.
Considering your particular setup, you could both have the amazon linux server run the tests with HTMLUnitDriver, and declare other machines (with a GUI and proper browser) as "nodes" to run your test on other browsers.
Link to a tutorial
Have you read this blog post by David Burns (Automated Tester):
http://www.theautomatedtester.co.uk/tutorials/selenium/selenium_rc_setup.htm
He describes the way to run selenese tests using HTMLSuite.
We are going to use the HTMLSuite commands of the Selenium Remote
Control. This allows you run your Selenese Test Suites as is. The
command should look like java -jar selenium-servre.jar -htmlsuite
. Browser
could be : -*firefox
-*chrome
-*iexplore
-*iehta
-*safari
-*custom /path/to/browser
The path to the test suite and the results file should be a full path.
Here is an example command; java -jar selenium-server.jar -htmlsuite
*iexplore http://www.theautomatedtester.co.uk c:\testsuite\testsuite.html c:\testsuite\results.html
I would point out that htmlunit does not seem to be a supported option so I would expect to use -*custom and provide a path to htmlunit.
This is legacy functionality so there is a chance it doesn't work as expected any more. HTMLSuite expects the tests to be in Selenese (HTML table) format, you mention trying with the PHP binding, I would not expect this to work. If you do want to use some PHP bindings I would suggest using Adam Saunter's fork of the facebook bindings, they are the most up to date and best supported.
https://github.com/Element-34/saunter.php
With Selenium WebDriver you can point to start a HtmlUnit in a already started node
In Java you'll do something like this:
IWebDriver driver = new RemoteWebDriver(new Uri("http://localhost:4444/wd/hub"), DesiredCapabilities.HtmlUnit());
To start the node just make sure to set browserName to 'htmlunit'.
I am able to run python scripts throgh selenium rc.
I dont know to run these scripts in selenium grid.I am not able to get the correct answer from anywhere.is it possible to run python scripts from selenium eclipse.
Please anyone provide me the solution as soon as possible.
Thx.
first of all you have to install selenium with this code:
pip install -U selenium
Alternately, you can download the source distribution from PyPI (e.g. selenium-2.53.1.tar.gz), unarchive it, and run:
python setup.py install
both of the methods described above install selenium as a system-wide package That will require administrative/root access to there machine. You may consider using a virtualenv to create isolated Python environments instead.
You're asking a lot of things that don't really make sense. E.g., there is no such thing as "selenium eclipse." Anyway, I'll try to answer the best I can.
Your Python scripts should have the client connect to the Selenium Grid Hub instead of a specific RC Server. The hub is then responsible for dispatching the request to any registered nodes in the grid. There is nothing Python-specific involved with grid.