I'm currently using Selenium RC and JUnit to test some basic login and registration scenarios. The problem is that my tests don't always give the same results. Sometimes running them will be fine and the tests pass. Other times, they'll get stuck at certain points during the login/registration process and time out. I've been trying to debug this for a long time, but with no permanent success.
Is Selenium being flaky and has anyone else had similar issues?
I have used selenium for 3 years. Sometimes I have some strange situations but usually it is my wine or software problem. It's good practice to auto stop script or use screenshot function. To see that source of problem.
Yes, when I use Ajax validation the results are inconsistent. I am testing using the Yii framework. I am generating random valid passwords, but from time to time Selenium goes too fast (!) for the Ajax to focus.
If I slow down the speed to 100 it tends to work about four out of five times. Any slower than that and the tests are agonizingly slow.
$this->setSpeed(100);
Related
Why can't we just record all of our test cases in Selenium IDE, export it to Java/WebDriver and run it in WebDriver with Eclipse?
I need a clear explanation as I am very much confused in using WebDriver!
And can anyone please explain why IDE recorded scripts fail in WebDriver?
why cant we just record all of our test cases in IDE, export it to java/webdriver and run it in webdriver
Great question, and here is the answer:
Selenium IDE is a Record and Playback tool, which is very easy to use, but it's very unreliable. Record and playback is typically a frowned upon in web applications. Since web applications are frequently changed, the IDE is not an ideal solution for a production environment, because of the maintenance nightmare that may arise.
Let me give you a practical example. You record your test, and you find an element with a dynamic ID. Sure we can import it into eclipse, but what happens when that test starts failing down the road? why not simply make your test agile and independent to catch these in the first place.
It also boils down to your principles of test automation. Test automation in MY opinion (and several other professionals), believe that test automation should be approached from a programming perspective. Programmers should write the tests, and maintain the tests. Ideally, your quality assurance personnel should be trained to write and maintain their own tests.
So again, back to your question, the IDE is designed to be a quick solution to automation, NOT a solution to a full regression suite.
And can anyone please explain why IDE recorded scripts fail in Webdriver?
I haven't used the IDE in a while, but the reason they fail, is because the scripts that are exported, are simply the steps, not an entire java file. This also is because Selenium IDE exportations are supposed to be agnostic when it comes to how to run your test. Say I'm a user of jUnit.. what if Selenium IDE exported it to TestNG all the time? That wouldn't be fair.. honestly i'd rather create my own tests than changing that one line every single time i create my test file.
You may read the full text of a research conducted, called Why do Record/Replay Tests
of Web Applications Break?
Why can't we just record all of our test cases in Selenium IDE, export it to Java/WebDriver and run it in WebDriver with Eclipse.
You can actually do this with Selenium IDE quite easily. Record your test case / test suite in Selenium IDE, export to "Java / JUnit 4 / Webdriver" to a .java file. This will generate a JUnit test that you can import and run from Eclipse (with the correct version of JUnit of course).
It's not 100% reliable, and you may need to make some manual changes/corrections, but in general it works pretty well. Start with a single small testcase and work from there.
Why not just record in the IDE and play the recording with WebDriver???
The IDE Does Not Know What to Wait For
Suppose Bob is performing manual checks and recording his interaction with Selenium IDE. He performs an operation that takes a while to update the GUI, and when the operation has finished, he clicks a button. Bob knows that the GUI has finished updating because a spinner that was show when the operation started is removed when the operation is finished. How is the IDE going to capture the fact that the operation must have finished before clicking on the button? (Note here this is not a case where the button is disabled until the operation is finished.) This is not a hypothetical: when you test dynamic tables (like those managed by DataTables), the user can change anything at any time.
This can be one of the reasons a sequence of commands created with the IDE would fail with WebDriver. It just does not know what to wait for.
It is possible to add waits manually in the IDE but if you are doing this, then you are no longer "just record[ing] all of our test cases". And what you are doing becomes more like writing code for WebDriver.
Users are Inefficient
Not long ago there was a Selenium question in which the user wanted to get Selenium to click on the last record in a table that had multiple pages of records. So the question was, how can I page through the table, all the way down to the last page and then click the last record? Someone (maybe me, maybe someone else) pointed out that if the table is sortable, it could be sorted in reverse order and then the Selenium code could click on the first record. That's 2 operations rather than p+1 operations: clicking p times, where p is the number of pages, plus 1 time for the click on the record.
When we write code for WebDriver, we have the opportunity to write the tests to avoid taking the scenic route to get the results we want. See the next point for the technical details as to why it matters.
Selenium Operations Can Be Costly
If the software that runs your Selenium commands is local, and your browser is local, you may not feel that Selenium operations can be costly but if you run your commands on one machine and the browser is remote, then you will notice a significant slowdown. For instance, if you spawn your browser in a Sauce Labs VM or in a BrowserStack VM to run a test suite, network delays are going to add significant time to how long it takes the suite to complete. For a complete application test suite, this can mean many minutes more.
The IDE produces a sequence of commands that each require a round-trip between the Selenium script and the browser. Each round-trip adds up. Suppose I want to check that two elements contain the same text. I'm not going to use Selenese because I don't usually use the IDE but using WebDriver code in Python, the script could be:
a = driver.find_element_by_id("a")
b = driver.find_element_by_id("b")
assert_equal(a.text, b.text)
This code requires 4 round-trips: one round-trip per find_element... and one per access to the text field. The same test could be written:
a_text, b_text = driver.execute_script("""
var a = document.getElementById("a");
var b = document.getElementById("b");
return [a.textcontent, b.textContent];
""");
assert_equal(a_text, b_text);
This needs only one round-trip. When you use the IDE to record actions, the sequence of commands is like the earlier snippet: lots of round-trips. When you write your code for WebDriver, you have the opportunity to optimize as you code.
This does not mean that the Selenium IDE has no use but I would never ever think of just recording tests with it and then playing those recordings with WebDriver.
If you are a newbie, you can actually use 80% of your IDE script as your webdriver JAVA script, u just have to slightly improvise your exported IDE script and it will work fine.
But when you start testing complex functionalities you have to learn some of the basic java methods to work it out!
I beg to differ. I see no reason in having to maintain extra code when a simple record and playback can achieve same results in a fraction of the time.
I do agree with the impl of the recorded script has to be agile. It can't just be as simple as the build in commands and one has to make it agile and even inplement their own macros (aka commands/aliases) specific to the app under test.
Still, if implemented right and with the help of few Selenium IDE plugins like flow control it becomes a breeze to maintain such complex test suits and have them played back via the HTML Runner. I've used Selenium IDE HTML scripts to fully cover complex sites and reused the same script for all responsive modes of the app.
Maintanance is simple as the IDE will help you quickly adjust a failed cmd and AJAX waits can easily be accounted for correctly by waiting for DOM content to change on the page. I'm still not convinced of this AJAX excuse for dismissing Selenium IDE.
For a Maven integration of flow controls command/aliases/macros used by Selenium IDE and Maven's SureFire test plugin see:
https://github.com/paulbors/sideflow
Feel free to use it in your product and improve as needed.
webdriver allows you to execute your test cases captured with Selenium IDE in a Selenium server or via RC to multiple servers. This includes options to integrate with Selenium-enabled providers (see link at the bottom).
For example, if you've captured your test case in IDE, you could easily integrate with your own selenium server, or a provider like Saucelabs to handle the running of your test cases in Firefox on Ubuntu, Safari on OSX and IE10 on Windows 8. As with any test case, you can run it from your IDE (Eclipse / IDEA / etc ) your builder (maven / gradle /), and / or via your CI system.
Check out https://saucelabs.com/selenium for some examples. We've also used our selenium test cases for load testing with Soasta - WebDriver allows simple IDE tests to be used in many different contexts.
Because when the complexity of your code will increase, it will be difficult for you to manage it using IDE. Maintenance of your scripts will be difficult using IDE.
Also Exporting test cases from IDE to webdriver is not 100% reliable.
Other people have said a lot of helpful answers. One thing that is missing is that if you write code instead of using the IDE you have the power to make changes to the database. It depends on the website you are testing how useful this is, but here are some useful examples I've experienced. In one case the website had multiple sections of a class. One of our tests was merging these sections into one. The IDE couldn't do this. But with webdriver we can add all of our test data into the database, perform the merge, and then remove the data from the database.
Another advantage that webdriver has is Selenium Grid. With Grid you can run your tests in parallel and on multiple browsers. This makes your testing run faster and you can more thoroughly test the code.
I am a novice in testing.
I am working on Linux.
I was reading about testing in headless mode and came across two things. One was X virtual frame buffer which does graphical operations in memory. So, no output is displayed. The implementation details I found in this link http://www.seleniumtests.com/2012/04/headless-tests-with-firefox-webdriver.html.
The other one that I came across was HtmlUnitDriver. This also does not open any browser while running the test. I wrote a basic sample code using HtmlUnitDriver and the assertions seem to work fine.
I understand that HtmlUnitDriver doesn't work too well with javascript. But apart from this, are there any major differences to choose one over the other?
I am going to be testing a web application that does have some abount of javascript in it.
I am a novice in this field. So, any answers, suggestions, etc. will be appreciated.
Thank you in advance
From my experience with both approaches:
HtmlUnit will in most practical cases be faster than a real browser with xvfb -- simply because it doesn't spend time rendering the pages. (A data point: 17 secs. HtmlDriver vs. 62 secs. FirefoxDriver for a specific test suite I'm using now).
It is easier to run several tests concurrently -- and it consumes a lot less resources -- using HtmlUnit. This can be very important if you have a large number of tests and you need them to finish fast (e.g. you want to follow the 10-minute-build rule).
As you said, HtmlUnit has its own quirks with JavaScript and the DOM. Not better or worse than any other browser (Firefox, Safari, IE, Chrome, ... -- they all have their own quirks), but one on which it is very questionable to spend time fixing bugs. I also find such bugs very difficult to diagnose, but that may be only my ignorance.
One advantage of real browsers + xvfb is that you can always use the exact same tests without xvfb and see what's going on -- possibly even use a console to run some JavaScript to diagnose issues. I sometimes feel quite blind when working with HtmlUnit, and because of above-mentioned quirks you can't always use the same exact tests code in both environments.
So, in summary, unless total test duration is important and you're ready to spend some time fighting HtmlUnit, it's just easier to go with a regular browser + xvfb.
I also like using xvnc, which has the added benefit of allowing you to connect to the screen of a running test and see what's going on (not sure whether you can do that with xvfb).
My duty is to pick up and continue developing PHP website for a small sized business client. Project has no testing code. I want to quickly establish at least very basic regression testing for the backend of the site.
I need to test the full contents of the web page char to char. Must see the diff of failed tests.
I need to be able to set up cookies and GET/POST data.
Once every few days I am updating the local database from production database. I would like to then have an overview of failed tests and very quickly update my test suits so that everything is passing again.
Is using WatiN or Selenium a good idea? My local environment is Linux.
About Selenium (and only Selenium as I don't know WatiN) - it can only do what you can do in your broswer. It can click, type in fields, submit forms, take screenshots (that's a very good one), set up cookies (so yes to this one). You can always set up GET data through the URL. But I am not aware of any technique in Selenium that would allow you to set up POST data in any other way than navigating in a browser. Also, because the tests are in your browser, they are not particularly fast. E.g, on our product, a single thorough test with ~250 steps takes about 10 minutes on my computer to complete. Of course, you can always divide that between many computers using Selenium Grid. It's just more work.
To conclude it - I'd say yes, Selenium is good for your needs as there are so many ways to write a good test in it that everyone finds his style. It is good for quick checks, functionality affirmations, but also for full-scale tests etc. But if you want to do some really advanced stuff, then that's a job for a long time. Selenium offers so much functionality in so many different ways that it is definitely a full-time job to understand them and know how to use them.
Try Selenium-IDE for 20 minutes. It is just an addon for Firefox that can record your actions and then replay them. If you like what you see, go for it. If no, hire someone who will.
I'm not sure if I am too late here, but in regards to WatiN it is only IE based, so if you plan to use any other browser you are better off with Selenium WebDriver (though WatiN has some Firefox support). From what I have found (I have used both WatiN and Selenium) Selenium can achieve more low level interactions (also see Selenium Grid), but really I think it is dependant on what you are looking to achieve and personal preference. If you have time to write your own wrapper to interact with WatiN/Selenium you will find the tests themselves are rather quick to run. Also, the beauty of Automation is that once these tests are written you can run them and walk away while they complete.
I've moved from SeleniumRC to WebDriver for nearly two years. But I have to say that I haven't felt there's any obvious advantages for webdriver over rc.
Now I have 200+ test cases with C# driver against a website. But when I run them thoroughly for regression testing, I usually get 150 passed and 50+ failed/errored. After running failed test cases for the second time, many of them passed, only few of them were proven to be issues with the testing code.
As I can see it, sometimes WebDriver is really performing slowly, while I never met with such a situation when I was using SeleniumRC before.
As a result, I started to doubt the necessity to move from rc to webdriver, because it took much more time for me to verify errors and failures than before.
So my question is, are there any advantages for webdriver over rc to make it worth it to move from rc to webdriver? If so, can you please kindly tell me? Also, tell me about the disadvantages.
Selenium RC injects Javascript into the page to drive the interactions. Webdriver interacts directly with the browser. Injecting additional Javascript has disadvantages, with the Selenium HQ site stating it rather well.
While Selenium was a tremendous tool, it wasn’t without its drawbacks.
Because of its Javascript based automation engine and the security
limitations browsers apply to Javascript, different things became
impossible to do. To make things “worst”, webapps became more and more
powerful over time, using all sorts of special features new browsers
provide and making this restrictions more and more painful.
http://seleniumhq.org/docs/01_introducing_selenium.html#selenium-history
Another way to think about it is that testing a page that has had more JS added, you're not really testing the original page; you're testing a modified page.
What #OCary said.
However, the new and more powerful WebDriver has some limitations, too. Because it is a work-in-progress, it's behaviour between different versions changes slightly from time to time. Also, not nearly all intended features have been implemented yet, it will take some more time to have it stable, bug-free and fully developed. For example: the SafariDriver has just landed, the window controls are missed, you can't download files in a convenient way etc.
But a healthy development on WebDriver is better than today's non-existant development on Selenium 1 (It just won't get any better.), right?
From my experience Selenium RC is the most stable and robust web testing framework I have used. I recently started evaluating WebDriver (aka Selenium 2) for the reason that everybody says it is the future of Selenium. So far I am not impressed. Simple things (like clicking a button) do not work consistently across different browsers and require different workarounds.
I realize that there are limitations in what you can do with JavaScript, but I would not want to sacrifice the stability of my tests to get over those limitations.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
On my last project, I created some test cases through Selenium, then automated them so they would run on every build launched from hudson. It worked fantastic, and was consistent for about a month.
Then the tests started failing. It was, most times, timing issues which caused the failures. After about two weeks of effort put in over the course of the next two months, it was decided to drop the Selenium tests. They should have been passing, but the responses and timing of the web application were varying to the extent to which tests would fail when they should have passed.
Did you have a similar experience? Is Selenium still a good tool to use for Web Application testing?
Selenium is great tool for web testing, although it's important to make sure your tests are reliable. Timing issues are common, so I would suggest the following:
Make sure you set a sensible timeout value. I find between 1-2 minutes works well.
Don't have pauses in your tests - they are the main cause for timing issues. Instead use the waitFor* commands. The waitForCondition is very useful
Identify external calls that can cause timeouts and block that traffic from the machine running tests. You can do this on a firewall level or simply redirect the domain to localhost in your hosts file.
Update:
You should also consider using Selenium Grid. It wont directly help with your timeouts, but it can provide a quicker feedback loop for your failures. If you're using TestNG to run your tests you can get it to automatically rerun failures - this gives the tests failing due to timeouts a second chance.
At my previous job we investigated using it as a test tool but found it too fussy to bother integrating into our process. Pretty much the same experience as you.
This was two or three years ago in version 0.8 or so though, I would have expected it to get better since then.
I've had a similar experience. We created a project that would bootstrap a selenium proxy and run an automated suite of tests, but unfortunately it clashed with our build server in a huge way. There were too many browser inconsistencies and third party dependencies for us to reliably add it to our build. It was also too slow for us, and added too much time to our builds.
Most of the errors we would run into would be timeouts.
We ended up keeping the project and use it for integration tests on major releases. The bootstrapping code that we used has proved invaluable in other areas as well.
Probably best to be run after a nightly build when there's the time for it. It, or Watin, coulod be integrated with your build scripts.
Very much depends on your team, but if you've a small testing team this can be priceless for picking up some very obvious runtime issues.
I'd keep the scope modest and really use them for some sanity testing that at least each page can load.
I did have a similar experience with Selenium. We had a legacy system which we built a sort of testing framework around so that we could test the changes we were making. This worked great at the start but eventually some of the earlier tests began to fail (or take too long to run) so we started to turn off more and more of the tests.
To fix some of the issues we stopped selenium from opening and closing a browser for each test i.e. the tests were broken up into blocks and for each block of tests the browser would only be opened once. This reduced the time taken to run the tests from several hours to 30 minutes.
Despite the issues I think Selenium is a great tool for testing web-based applications. Many of the problems we experienced centered on the fact that the system we were testing was a legacy system. If you like test-driven development then Selenium fits in very well with that development practice.
EDIT:
Another good thing about Selenium is the ability to track what developer introduced the error as well as where the error is (source file). This makes life so much easier when it comes to fixing the error.
We initially tried to use selenium on our build machine but tests were very brittle and we found we spent a lot of time trying to keep old tests running when changes occurred to unrelated functionality accessed through the same set of pages. We were automating the tests through nunit.
I would use selenium more as a customer acceptance and integration testing tool. I'd agree with using it for a nightly build on functionality that is stable.
At a first glance, Selenium looks great. Unfortunately, as sometimes happens with open source projects, they rush to implement new features instead of making it more stable.