I love Selenium IDE; mainly because it enables me to automate many regression tests without me having to write any code at all!
Now, I want to do load testing on my website i.e execute the same quantum of testing activity concurrently with many browser windows putting load on the website to check if it holds integrity. I want to stick to the GUI (Selenium IDE) and still want to accomplish this.
I searched and found Selenium Grid can help me accomplish this, however, running Grid involves exporting test cases to Java TestNG WebDriver or something similar and then setting up test suites. This will definitely involve some complexity in terms of coding.
The question is can I avoid this and stick to Selenium IDE (the GUI) to perform this task?
Many thanks in advance!
The only way (that i can think of) is running the same script on multiple computers at the same time. I would run the script as fast as you can without having it crash in normal circumstances. Grab a few volunteers and get everyone to click RUN at the same time.
Alternatively you could run multiple VM clients if you don't have access to multiple computers. Your internet connection may be the only barrier to entry with the VM method
Why can't we just record all of our test cases in Selenium IDE, export it to Java/WebDriver and run it in WebDriver with Eclipse?
I need a clear explanation as I am very much confused in using WebDriver!
And can anyone please explain why IDE recorded scripts fail in WebDriver?
why cant we just record all of our test cases in IDE, export it to java/webdriver and run it in webdriver
Great question, and here is the answer:
Selenium IDE is a Record and Playback tool, which is very easy to use, but it's very unreliable. Record and playback is typically a frowned upon in web applications. Since web applications are frequently changed, the IDE is not an ideal solution for a production environment, because of the maintenance nightmare that may arise.
Let me give you a practical example. You record your test, and you find an element with a dynamic ID. Sure we can import it into eclipse, but what happens when that test starts failing down the road? why not simply make your test agile and independent to catch these in the first place.
It also boils down to your principles of test automation. Test automation in MY opinion (and several other professionals), believe that test automation should be approached from a programming perspective. Programmers should write the tests, and maintain the tests. Ideally, your quality assurance personnel should be trained to write and maintain their own tests.
So again, back to your question, the IDE is designed to be a quick solution to automation, NOT a solution to a full regression suite.
And can anyone please explain why IDE recorded scripts fail in Webdriver?
I haven't used the IDE in a while, but the reason they fail, is because the scripts that are exported, are simply the steps, not an entire java file. This also is because Selenium IDE exportations are supposed to be agnostic when it comes to how to run your test. Say I'm a user of jUnit.. what if Selenium IDE exported it to TestNG all the time? That wouldn't be fair.. honestly i'd rather create my own tests than changing that one line every single time i create my test file.
You may read the full text of a research conducted, called Why do Record/Replay Tests
of Web Applications Break?
Why can't we just record all of our test cases in Selenium IDE, export it to Java/WebDriver and run it in WebDriver with Eclipse.
You can actually do this with Selenium IDE quite easily. Record your test case / test suite in Selenium IDE, export to "Java / JUnit 4 / Webdriver" to a .java file. This will generate a JUnit test that you can import and run from Eclipse (with the correct version of JUnit of course).
It's not 100% reliable, and you may need to make some manual changes/corrections, but in general it works pretty well. Start with a single small testcase and work from there.
Why not just record in the IDE and play the recording with WebDriver???
The IDE Does Not Know What to Wait For
Suppose Bob is performing manual checks and recording his interaction with Selenium IDE. He performs an operation that takes a while to update the GUI, and when the operation has finished, he clicks a button. Bob knows that the GUI has finished updating because a spinner that was show when the operation started is removed when the operation is finished. How is the IDE going to capture the fact that the operation must have finished before clicking on the button? (Note here this is not a case where the button is disabled until the operation is finished.) This is not a hypothetical: when you test dynamic tables (like those managed by DataTables), the user can change anything at any time.
This can be one of the reasons a sequence of commands created with the IDE would fail with WebDriver. It just does not know what to wait for.
It is possible to add waits manually in the IDE but if you are doing this, then you are no longer "just record[ing] all of our test cases". And what you are doing becomes more like writing code for WebDriver.
Users are Inefficient
Not long ago there was a Selenium question in which the user wanted to get Selenium to click on the last record in a table that had multiple pages of records. So the question was, how can I page through the table, all the way down to the last page and then click the last record? Someone (maybe me, maybe someone else) pointed out that if the table is sortable, it could be sorted in reverse order and then the Selenium code could click on the first record. That's 2 operations rather than p+1 operations: clicking p times, where p is the number of pages, plus 1 time for the click on the record.
When we write code for WebDriver, we have the opportunity to write the tests to avoid taking the scenic route to get the results we want. See the next point for the technical details as to why it matters.
Selenium Operations Can Be Costly
If the software that runs your Selenium commands is local, and your browser is local, you may not feel that Selenium operations can be costly but if you run your commands on one machine and the browser is remote, then you will notice a significant slowdown. For instance, if you spawn your browser in a Sauce Labs VM or in a BrowserStack VM to run a test suite, network delays are going to add significant time to how long it takes the suite to complete. For a complete application test suite, this can mean many minutes more.
The IDE produces a sequence of commands that each require a round-trip between the Selenium script and the browser. Each round-trip adds up. Suppose I want to check that two elements contain the same text. I'm not going to use Selenese because I don't usually use the IDE but using WebDriver code in Python, the script could be:
a = driver.find_element_by_id("a")
b = driver.find_element_by_id("b")
assert_equal(a.text, b.text)
This code requires 4 round-trips: one round-trip per find_element... and one per access to the text field. The same test could be written:
a_text, b_text = driver.execute_script("""
var a = document.getElementById("a");
var b = document.getElementById("b");
return [a.textcontent, b.textContent];
""");
assert_equal(a_text, b_text);
This needs only one round-trip. When you use the IDE to record actions, the sequence of commands is like the earlier snippet: lots of round-trips. When you write your code for WebDriver, you have the opportunity to optimize as you code.
This does not mean that the Selenium IDE has no use but I would never ever think of just recording tests with it and then playing those recordings with WebDriver.
If you are a newbie, you can actually use 80% of your IDE script as your webdriver JAVA script, u just have to slightly improvise your exported IDE script and it will work fine.
But when you start testing complex functionalities you have to learn some of the basic java methods to work it out!
I beg to differ. I see no reason in having to maintain extra code when a simple record and playback can achieve same results in a fraction of the time.
I do agree with the impl of the recorded script has to be agile. It can't just be as simple as the build in commands and one has to make it agile and even inplement their own macros (aka commands/aliases) specific to the app under test.
Still, if implemented right and with the help of few Selenium IDE plugins like flow control it becomes a breeze to maintain such complex test suits and have them played back via the HTML Runner. I've used Selenium IDE HTML scripts to fully cover complex sites and reused the same script for all responsive modes of the app.
Maintanance is simple as the IDE will help you quickly adjust a failed cmd and AJAX waits can easily be accounted for correctly by waiting for DOM content to change on the page. I'm still not convinced of this AJAX excuse for dismissing Selenium IDE.
For a Maven integration of flow controls command/aliases/macros used by Selenium IDE and Maven's SureFire test plugin see:
https://github.com/paulbors/sideflow
Feel free to use it in your product and improve as needed.
webdriver allows you to execute your test cases captured with Selenium IDE in a Selenium server or via RC to multiple servers. This includes options to integrate with Selenium-enabled providers (see link at the bottom).
For example, if you've captured your test case in IDE, you could easily integrate with your own selenium server, or a provider like Saucelabs to handle the running of your test cases in Firefox on Ubuntu, Safari on OSX and IE10 on Windows 8. As with any test case, you can run it from your IDE (Eclipse / IDEA / etc ) your builder (maven / gradle /), and / or via your CI system.
Check out https://saucelabs.com/selenium for some examples. We've also used our selenium test cases for load testing with Soasta - WebDriver allows simple IDE tests to be used in many different contexts.
Because when the complexity of your code will increase, it will be difficult for you to manage it using IDE. Maintenance of your scripts will be difficult using IDE.
Also Exporting test cases from IDE to webdriver is not 100% reliable.
Other people have said a lot of helpful answers. One thing that is missing is that if you write code instead of using the IDE you have the power to make changes to the database. It depends on the website you are testing how useful this is, but here are some useful examples I've experienced. In one case the website had multiple sections of a class. One of our tests was merging these sections into one. The IDE couldn't do this. But with webdriver we can add all of our test data into the database, perform the merge, and then remove the data from the database.
Another advantage that webdriver has is Selenium Grid. With Grid you can run your tests in parallel and on multiple browsers. This makes your testing run faster and you can more thoroughly test the code.
I am starting up a test suite for an internal JavaScript UI library for the place I work.
I have written about 10 Selenium 1 test cases or so in Python, for exploratory purposes, and so far it's gone well. I wrote the tests in Python and am using RC to run the tests in IE7,8,FF and Safari on the Mac so far so good.
I know Selenium2 is in alpha but read somewhere that it's production ready, whatever that means.
1) Since I am just starting out writing this suite, should I be using the webdriver API and Sel2, or is that not ready for prime time?
2) Can I use Sel2 with RC yet? It's unclear to me from Selenium's website if that is possible.
3) Anyone have experience with the Python driver for selenium 2? Or should I just write tests in Java?
Disclaimer: I am a Selenium Committer
I would say write your tests in Selenium 2 where possible as the bits that are complete are ready for use and work well.
There are bits that have not been fully implemented yet like how to handle alerts and a few others but the current API is stable, what it does in the background is changing but that shouldn't affect your tests unless we introduce a bug.
You can use Selenium 2 with RC.
http://code.google.com/p/selenium/downloads/detail?name=selenium-server-standalone-2.0a5.jar
I would use the Java API if possible. Since the other APIs are ports and are not as up to date.
Is it better to write/record selenium tests in html format and run them directly in the server with "-htmlSuite" or to write the tests in java/C#/... and run them in the server using selenium-rc?
What is the recommended solution?
I would always recommend people writing their Selenium Tests in a programming language because it allows the tests to be a lot more expressive.
You can create common methods that all tests use and if that changes you can then update 1 method to get n tests passing because they all fail on that item. One example of this is the Page Object model which is a development method for tests that suggest you create a DSL for each page you interact with and then your tests read a lot more fluently to both technical and non-technical people.
If you write your tests in a programming language you can also take advantage of Selenium Grid which runs your tests in parallel to make them faster
I'd highly recommend you look at moving to Selenium in C#, Ruby,PHP or Python. I found many timing issues resolving once I got away from the HTML Selenese.
I might keep in mind how you plan on executing your recorded tests. For instance, it might be trivial to incorporate Selenium's generated JUnit tests into a pre-existing JUnit-based testing framework.
I have been running test cases in Selenium IDE using Firefox. This works quite well however when I create a suite by "adding test cases" I can never get the suite to execute. I can't figure out if the problem is with Selenium or me. Any advise for this novice.
I've never used the Test Suites feature before, but when I tried to reproduce your issue I found that saving and then opening a test suite was enough to break it. I was never able to get it to run again. Apparently this is a known issue:
http://jira.openqa.org/browse/SIDE-191
The depressing part is that the bug linked to above was reported in March 2008. I guess Test Suites are not at the top of the priority list. On the other hand, I wouldn't recommend abandoning Selenium completely. If you're comfortable with any of the supported programming languages, simply export your tests and run them with RC. This is a great solution in the long-term because it allows you to add control flow and more sophisticated logic to your tests.
You might consider upgrading to the latest version of Selenium...they're on Version 1.0 as of Jun 30 2009. Lots of bug fixes, including the 'minor' annoyance of not being able to add new test cases to a test suite via the IDE.
That might help...
Does only 1 test execute? Or what is the problem? You should have the Test Case list on the left when the suite is open. Then press the Play Entire Test Suite button