I am starting up a test suite for an internal JavaScript UI library for the place I work.
I have written about 10 Selenium 1 test cases or so in Python, for exploratory purposes, and so far it's gone well. I wrote the tests in Python and am using RC to run the tests in IE7,8,FF and Safari on the Mac so far so good.
I know Selenium2 is in alpha but read somewhere that it's production ready, whatever that means.
1) Since I am just starting out writing this suite, should I be using the webdriver API and Sel2, or is that not ready for prime time?
2) Can I use Sel2 with RC yet? It's unclear to me from Selenium's website if that is possible.
3) Anyone have experience with the Python driver for selenium 2? Or should I just write tests in Java?
Disclaimer: I am a Selenium Committer
I would say write your tests in Selenium 2 where possible as the bits that are complete are ready for use and work well.
There are bits that have not been fully implemented yet like how to handle alerts and a few others but the current API is stable, what it does in the background is changing but that shouldn't affect your tests unless we introduce a bug.
You can use Selenium 2 with RC.
http://code.google.com/p/selenium/downloads/detail?name=selenium-server-standalone-2.0a5.jar
I would use the Java API if possible. Since the other APIs are ports and are not as up to date.
Related
I am using Karma for unit tests and it works great.
The tests I would like to run would look similar to this:
1. var $input = $("#foo");
2. var $button = $("#bar");
3. $input.val("HELLO WARLD");
4. $button.trigger("click");
5. expect(window.appData.foo).toBe("HELLO WARLD");
Selenium looks like it might be the right choice but I don't know if there is a better option.
Yes. You can use Selenium for end-to-end automated tests on existing web applications.
Selenium is designed to automate web browsers. It's certainly a good choice for what you want. I'm sure there are other options, but as Selenium is used by several big companies, including Google for their automated web testing technologies, you will be safe with Selenium assuming you implement it optimally.
The only thing to keep in mind, is that GUI testing is NOT supposed to replace unit tests.
GUI testing should be your least used testing methodology. You can, however, use Selenium to test your Acceptance tests which is what many companies do.
Why can't we just record all of our test cases in Selenium IDE, export it to Java/WebDriver and run it in WebDriver with Eclipse?
I need a clear explanation as I am very much confused in using WebDriver!
And can anyone please explain why IDE recorded scripts fail in WebDriver?
why cant we just record all of our test cases in IDE, export it to java/webdriver and run it in webdriver
Great question, and here is the answer:
Selenium IDE is a Record and Playback tool, which is very easy to use, but it's very unreliable. Record and playback is typically a frowned upon in web applications. Since web applications are frequently changed, the IDE is not an ideal solution for a production environment, because of the maintenance nightmare that may arise.
Let me give you a practical example. You record your test, and you find an element with a dynamic ID. Sure we can import it into eclipse, but what happens when that test starts failing down the road? why not simply make your test agile and independent to catch these in the first place.
It also boils down to your principles of test automation. Test automation in MY opinion (and several other professionals), believe that test automation should be approached from a programming perspective. Programmers should write the tests, and maintain the tests. Ideally, your quality assurance personnel should be trained to write and maintain their own tests.
So again, back to your question, the IDE is designed to be a quick solution to automation, NOT a solution to a full regression suite.
And can anyone please explain why IDE recorded scripts fail in Webdriver?
I haven't used the IDE in a while, but the reason they fail, is because the scripts that are exported, are simply the steps, not an entire java file. This also is because Selenium IDE exportations are supposed to be agnostic when it comes to how to run your test. Say I'm a user of jUnit.. what if Selenium IDE exported it to TestNG all the time? That wouldn't be fair.. honestly i'd rather create my own tests than changing that one line every single time i create my test file.
You may read the full text of a research conducted, called Why do Record/Replay Tests
of Web Applications Break?
Why can't we just record all of our test cases in Selenium IDE, export it to Java/WebDriver and run it in WebDriver with Eclipse.
You can actually do this with Selenium IDE quite easily. Record your test case / test suite in Selenium IDE, export to "Java / JUnit 4 / Webdriver" to a .java file. This will generate a JUnit test that you can import and run from Eclipse (with the correct version of JUnit of course).
It's not 100% reliable, and you may need to make some manual changes/corrections, but in general it works pretty well. Start with a single small testcase and work from there.
Why not just record in the IDE and play the recording with WebDriver???
The IDE Does Not Know What to Wait For
Suppose Bob is performing manual checks and recording his interaction with Selenium IDE. He performs an operation that takes a while to update the GUI, and when the operation has finished, he clicks a button. Bob knows that the GUI has finished updating because a spinner that was show when the operation started is removed when the operation is finished. How is the IDE going to capture the fact that the operation must have finished before clicking on the button? (Note here this is not a case where the button is disabled until the operation is finished.) This is not a hypothetical: when you test dynamic tables (like those managed by DataTables), the user can change anything at any time.
This can be one of the reasons a sequence of commands created with the IDE would fail with WebDriver. It just does not know what to wait for.
It is possible to add waits manually in the IDE but if you are doing this, then you are no longer "just record[ing] all of our test cases". And what you are doing becomes more like writing code for WebDriver.
Users are Inefficient
Not long ago there was a Selenium question in which the user wanted to get Selenium to click on the last record in a table that had multiple pages of records. So the question was, how can I page through the table, all the way down to the last page and then click the last record? Someone (maybe me, maybe someone else) pointed out that if the table is sortable, it could be sorted in reverse order and then the Selenium code could click on the first record. That's 2 operations rather than p+1 operations: clicking p times, where p is the number of pages, plus 1 time for the click on the record.
When we write code for WebDriver, we have the opportunity to write the tests to avoid taking the scenic route to get the results we want. See the next point for the technical details as to why it matters.
Selenium Operations Can Be Costly
If the software that runs your Selenium commands is local, and your browser is local, you may not feel that Selenium operations can be costly but if you run your commands on one machine and the browser is remote, then you will notice a significant slowdown. For instance, if you spawn your browser in a Sauce Labs VM or in a BrowserStack VM to run a test suite, network delays are going to add significant time to how long it takes the suite to complete. For a complete application test suite, this can mean many minutes more.
The IDE produces a sequence of commands that each require a round-trip between the Selenium script and the browser. Each round-trip adds up. Suppose I want to check that two elements contain the same text. I'm not going to use Selenese because I don't usually use the IDE but using WebDriver code in Python, the script could be:
a = driver.find_element_by_id("a")
b = driver.find_element_by_id("b")
assert_equal(a.text, b.text)
This code requires 4 round-trips: one round-trip per find_element... and one per access to the text field. The same test could be written:
a_text, b_text = driver.execute_script("""
var a = document.getElementById("a");
var b = document.getElementById("b");
return [a.textcontent, b.textContent];
""");
assert_equal(a_text, b_text);
This needs only one round-trip. When you use the IDE to record actions, the sequence of commands is like the earlier snippet: lots of round-trips. When you write your code for WebDriver, you have the opportunity to optimize as you code.
This does not mean that the Selenium IDE has no use but I would never ever think of just recording tests with it and then playing those recordings with WebDriver.
If you are a newbie, you can actually use 80% of your IDE script as your webdriver JAVA script, u just have to slightly improvise your exported IDE script and it will work fine.
But when you start testing complex functionalities you have to learn some of the basic java methods to work it out!
I beg to differ. I see no reason in having to maintain extra code when a simple record and playback can achieve same results in a fraction of the time.
I do agree with the impl of the recorded script has to be agile. It can't just be as simple as the build in commands and one has to make it agile and even inplement their own macros (aka commands/aliases) specific to the app under test.
Still, if implemented right and with the help of few Selenium IDE plugins like flow control it becomes a breeze to maintain such complex test suits and have them played back via the HTML Runner. I've used Selenium IDE HTML scripts to fully cover complex sites and reused the same script for all responsive modes of the app.
Maintanance is simple as the IDE will help you quickly adjust a failed cmd and AJAX waits can easily be accounted for correctly by waiting for DOM content to change on the page. I'm still not convinced of this AJAX excuse for dismissing Selenium IDE.
For a Maven integration of flow controls command/aliases/macros used by Selenium IDE and Maven's SureFire test plugin see:
https://github.com/paulbors/sideflow
Feel free to use it in your product and improve as needed.
webdriver allows you to execute your test cases captured with Selenium IDE in a Selenium server or via RC to multiple servers. This includes options to integrate with Selenium-enabled providers (see link at the bottom).
For example, if you've captured your test case in IDE, you could easily integrate with your own selenium server, or a provider like Saucelabs to handle the running of your test cases in Firefox on Ubuntu, Safari on OSX and IE10 on Windows 8. As with any test case, you can run it from your IDE (Eclipse / IDEA / etc ) your builder (maven / gradle /), and / or via your CI system.
Check out https://saucelabs.com/selenium for some examples. We've also used our selenium test cases for load testing with Soasta - WebDriver allows simple IDE tests to be used in many different contexts.
Because when the complexity of your code will increase, it will be difficult for you to manage it using IDE. Maintenance of your scripts will be difficult using IDE.
Also Exporting test cases from IDE to webdriver is not 100% reliable.
Other people have said a lot of helpful answers. One thing that is missing is that if you write code instead of using the IDE you have the power to make changes to the database. It depends on the website you are testing how useful this is, but here are some useful examples I've experienced. In one case the website had multiple sections of a class. One of our tests was merging these sections into one. The IDE couldn't do this. But with webdriver we can add all of our test data into the database, perform the merge, and then remove the data from the database.
Another advantage that webdriver has is Selenium Grid. With Grid you can run your tests in parallel and on multiple browsers. This makes your testing run faster and you can more thoroughly test the code.
Selenium 2 has been in beta phase for last few months. I would like to know learning’s if any of our us have analyzed/migrated from selenium 1 to selenium 2
How much was the effort involved in terms of Changes# to accomodates 2 features. Methods/API changes#
How much was perf improvements in terms of run time of tests in Selenium 2
Any best practices/learning shared would be useful
Going through the transition myself. If you had Selenium 1 experience, Selenium 2 feels quite different actually. Here is my Selenium 2 pros/cons vs. Selenium 1 I see so far (I use Python so some of them are Python specific):
Pros:
Much faster.
No need to run a separate server.
Gone are wait_for_page_to_load(), wait_for_element_present(), etc. All element interactions, clicks, etc. are blocking now, which is good. The only problem is with asynchronously loaded content (Ajax) though, see Con bellow.
Cons:
Loading/waiting for asynchronous content which used to be "free" with wait_for_page_to_load() requires coding now. These are the solutions I found so far:
use PageFactory/AjaxElementLocatorFactory like explained here, unfortunately I couldn't find the equivalent for Python.
use webdriver.implicitly_wait(N), this seems to be doing the trick with Python but using that seems to cause my script to miss changing elements which it used to detect before.
don't do sleep(T), loop until element appears, etc, that defeats the purpose of the whole thing (and makes wait_for_page_to_load look beautiful)...
The whole thing still feels a bit raw. Different drivers and bindings seem to miss different functionality. Not to say you can't use it but be ready to find 'alternate solutions' for certain problems.
The documentation is a bit dubious (related to the prev. point I guess). Especially for Python. Be ready to read code and experiment a lot (which luckily is easy with Python). Most of the 'tutorials' you'll find on the web (again, esp. Python, Java seems to be much better covered) are just to get you started with the plainest of web applications.
No PHP bindings, not a big one I prefer Python but our original suite was PHP so I noticed.
SeleniumIDE seems to be useless with Selenium 2.
Other differences:
The page elements you are accessing have to be 'visible' on the page at the moment when you ask selenium to find them. For example if you have a menu (containing a list of links) which opens when you hover your mouse over, you have to make sure it is open/visible before you click on a link inside (which wasn't the case in Selenium 1). This has it's uses since you'd be testing what an user would see on the page but requires the extra code. I found two solutions:
run a Javascript which would open your menu, in my case driver.execute_script("document.getElementById('dashboard_menu_navigation').show()") then click the menu item driver.find_element_by_link_text('Orders').click()
use the Mouse / Keyboard classes to simulate actual interaction, this seems to be broken in the Python bindings though (see Cons above):
Example (which throws 'WebElement' object has no attribute 'mouse_move_to' today):
element=driver.find_element_by_id('mn_dashboard')
mouse=Mouse()
mouse.move_to(element)
The Cons list seems longer but that is mostly if you are coming from Selenium 1. I do prefer the lightness and speed of Selenium 2 and despite the early code (using 2.0b4 at the time of writing) the whole thing is quite usable.
Hope to save someone some time...
Moving from Selenium 1 to Selenium 2 is as simple moving from
Selenium selenium = new DefaultSelenium("localhost", 4444, "*firefox", "http://www.example.com");
selenium.open("/");
to
Webdriver driver = new FirefoxDriver();
Selenium selenium = new WebDriverBackedSelenium(driver, "http://www.example.com");
selenium.open("/");
Since Selenium 2 is more tightly bound to the browser you will see huge difference. I have seen tests running at least 2x faster but in some cases I have seen it running 4x faster.
All the same best practises that you learnt during Selenium will be the translated across
Slides that might help too are posted here:
http://www.slideshare.net/rogerjhu1/lessons-learned-migrating-tests-to-selenium-v2
You can use WebDriverBackedSelenium class to convert WebDriver interface into Selenium interface:
Selenium selenium = new WebDriverBackedSelenium(webDriver, baseUrl);
or you can use the method getUnderlyingWebDriver() to convert Selenium interface into WebDriver interface:
WebDriver webDriver = ((WebDriverBackedSelenium)selenium).getUnderlyingWebDriver();
I posted slides of one of my recent presentation on how to get started with Selenium 2 here:
http://www.slideshare.net/sebarmeli/getting-started-with-selenium-2
One of the best practices in Selenium 2 is the PageObject pattern.
If you are using Java and Maven, you may want to check out this Archetype plugin (that also gives you a first taste of the PageObject pattern):
https://github.com/sebarmeli/Selenium2-Java-QuickStart-Archetype
I am currently looking at choosing a automated web testing framework for use with a current web project. I am pretty sure that I am going to use selenium. I have been reading a little about selenium 2.0 or webdriver. Does webdriver have a recorder plugin like selenium 1.0 or is it just an API for web tests?
I would appreciate it if anyone could point me in the direction of some good examples of using it.
Thanks
The maintainers of Selenium, sort of answer this question here:
http://www.viddler.com/explore/saucelabs/videos/27/
Look for timestamp 00:56:00 and 00:43:00
Basically, there will probably be IDE support for Selenium 2 eventually. But it isn't there yet and it's not clear what it will look like.
You should probably be avoiding the IDE anyway. The tests that you record using the IDE tend to be fragile, unmaintainable and hard to read. Once you've learned the basics of Selenium, it's probably better to write your tests in a real programming language.
Have you seen - http://seleniumhq.org/docs/03_webdriver.html and
http://seleniumhq.org/docs/04_webdriver_advanced.html
Using Selenium IDE we can format recorded code in different languages - Selenium IDE > Option > Format
But did not see any formatter for webdriver, though bumped on to this -
https://addons.mozilla.org/en-US/firefox/addon/webdriver-backed-formatters/
Only a few minor adjustments are required to get the existing 1.0 C# scripts running under the 2.0 beta of Selenium.
Check out this blog post
Once the RTW release of Selenium is ready, I'm hoping that a better formatter is ready. Until then, I'm able to use my scripts.
You can use Selenium IDE 1.9.x (I am using 1.9.1) or 1.8.x for recording purpose. And then you can export the recording tests into your desired programming language. For example, if you want to convert your recording code (recorded by Selenium IDE) into C# , follow the steps below:
Click File menu at Selenium IDE
Mouse over "Export Test Case As" and then click "C#/NUnit/WebDriver"
Save the file at your desired location
In the same way, you can convert your code to Java, PHP, Ruby etc.
Note: Some code might not be converted exactly in some cases. In that case, it needs to modify code manually
My project is compatible only with Internet Explorer. I want the test scripts to get generated automatically as it is done in Selenium IDE.
Can i use Selenium RC to test my application? I could not use Selenium IDE as it can be used only with Mozilla Firefox.
seleniumrc works with IE. You can specify the browser and the path to it within the config file.
It can be easily integrated into night builds via ant.
stick to writing the testcase in java.
Selenium RC and Selenium Grid are both really good at running tests against IE. You can see all the browsers that are supported by Selenium here and Selenium is Designed to write for one browser and work in the rest. THere are a few little quirks that wont work in every browser but 99% of the time it will.
Selenium RC works with IE, but is very buggy with IE 6 (to the point of being unusable). Generating the scripts is not trivial and there are many methods of doing it. We have created a Firefox extension that examines objects via introspection to make click recording easy. There are many options out there but your best bet is to write your tests with Firefox/Firebug (or Chrome). They will make object location much simpler and if you are careful the locator strings should still work in IE.
There could be two answer to you question:
Besides Selenium, though it has ample of advantages, I am reading about another tool which uses same API which Selenium use. The only changes in API I have seen so far is it reduces the complexity of functions thus making it more easier and simpler for user who is learning.
The tool is called 'Helium' and it has 50% (and more) less complex functions and code as Selenium has.
The only problem with this tool is it is paid tool for learning purpose and for implementing not-so-big scale project you can use it. But yeah after some time its gonna cost you.
I have implemented some code on Helium. Please let me know , if you face any issue initially or you are thinking to implement it.
Other being, you can use Selenium Builder(http://khyatisehgal.wordpress.com/2014/05/26/selenium-builder-exporting-and-execution/) which is an advanced form of Selenium IDE. It imports your command in different languages and does work more effectively and efficiently as Selenium IDE does(http://khyatisehgal.wordpress.com/2014/05/25/selenium-builder/)
Please let me know , if you have any doubt in any of the tool.
I know Watin is compatible with IE and Firefox. If you want to generate the test code you can use the Watin Test Recorder
This of course is implying that you are using .Net
... Or you could just use the .net bindings that comes along with the latest couple of versions, then you can just run 'em through nUnit.
For ex. Selenium IDE users Katalon Recorder might be a good match. Supports different browsers.