Along with a basic doubt I had here(which unfortunately did not get much attention) I have another basic doubt in context with preparing a Java Framework for Selenium-WebDriver.
I am not satisfied with the answers here and need some quality advice on best practices from real world case scenarios. (Counting on SO)
The main question is - Which one is better suited for Selenium? TestNG or JUnit? And Why?
I got some basic differences here,here(for Automation but unclear answers -all ask to pick one) and many more. Looking for some more details which will help me decide better.
P.S: I have experience with Selenium+JUnit. Will I regret if I go the same way?
simplicity with JUnit vs. adjustability with TestNG - that's all
Firstly, both work quite well.
If you want to scale down your execution time and have enough time to find out the best possible configuration I would recommend TestNG. In case of parallelization that's your tool
If you prefer the plainness of a top down execution JUnit is what you're looking for
--EDIT--
Regarding JUnit 3
I'm in the process of switching from JUnit to TestNG with a project right now. It started as an experiment until switching over completely became a 'done deal' in my mind. Some things that sold me on it are the ability to easily tag tests (groups), easily create dependencies (don't run test y unless test x succeeds first), more robust out of the box reporting (robust may not be the right word, but I find them much easier to read and manage) and just more whizz-bang options in general, easily available and extendable. I'm early in the process and may yet come to regret it and switch back, but unless there's some major unforeseen disaster lurking in the road ahead I don't think so. If you do make the switch, look up the Eclipse plugin. Apply it to an existing test and be sure and take out any references to JUnit. You'll have to do some polishing and tweaking here and there, but it's no big deal once you get going.
Also bear in mind that the stock asserts have the same names as JUnit assertions, but the order of expected and actual values is opposite between the two. Dig in a bit and you'll see what I mean. Again, no big deal, just need to be aware of it.
Best wishes with whichever you choose. You can make either do what you want, I'm just finding TestNG a bit more flexible and...pleasantly surprising, which is nice to experience now and then in Java land! ;)
Related
We have an application in production and there are slim chances that it would get enhanced further. We are curious whether shall we consider automating same. if yes, how it would be beneficial. Thanks!
You need to answer yourself if automation will benefit you. If you don't see benefits from automation - there is none, other than gathering knowledge and experience while developing automation.
Other non-straight benefits:
- robust smoke tests (if update finally comes)
- automated tests are sort of behavior documentation (take a look at BDD)
Making test-automation have positive ROI is probably the hardest step in whole automation. Without experience on what to automate and how to do it, it's negative in most cases. So gathering such knowledge and experience is benefit, which of course comes with a cost. If you have a gut feeling that this case won't benefit you - it probably won't, and you should look for a different place to gather experience.
Good luck
Automation is always a hot topic in our office. I agree with G. Swiec that ROI is going to be your biggest concern. I personally think there is value in having a robust automation suite for an app. Even if you don't plan on enhancing your app further, the tests could help catch bugs you missed or even if dependencies go wonky. If you have decent tools and the hours to do it, I think it could add value. However, these benefits are quickly outweighed if you don't have the man hours to research and implement an automated testing strategy.
You may want to consider other popular avenues like contract testing. Good contract testing can help relieve the need to write big automated e2e tests altogether!
My software company never did BDD or even TDD before. Testing before meant to simply try out the new software some days before deployment.
Our recent project is about 70% done. We also use it as a playground for new technologies, tools and ways of developement. My boss wanted that I switch to it to "test testing".
I tried out Selenium 2 and RSpec. Both are promising, but how to catch up months of developement? Further problems are:
new language
never wrote a line of code by myself
huge parts are written by freelancer
lots of fancy hacking
no documentation at all besides some source comments and flow charts
All I was able to was to do was to cover a whole process with Selenium. This appeared to be quite painfully (but still possible), since the software was obivously never meant to be testet this way. We have lots of dynamically generated id ´s, fancy jQuery and much more. Dont even know how to get started with RSpec.
So, is it still possible to apply BDD to this project? Or should I run far away and never come back?
Before you start - have you asked your boss what he values from the testing? I'd clarify this with your boss first. The main benefits of system-level BDD are in the conversations with business stakeholders before the code is written. You won't be able to get this if all you're doing is wrapping existing code. At a unit level, the main benefits are in questioning the responsibilities of classes and their behavior, which, again, you won't be able to get. It might be useful for helping you understand the value of the application and each level of code, though.
If your boss just wants to try out BDD or TDD it may be simpler to start a new project, or even grab an existing project from someone else to wrap some tests around. If he genuinely wants to experiment with BDD over legacy code, then it may be worth persisting with what you have - #Esko's book suggestion rocks. Putting higher-level system tests around the existing functionality can help you to avoid breaking things when you refactor the lower-level code (and you will need to do so, if you want to put tests in place). I additionally recommend Martin Fowler's "Refactoring".
RSpec is best suited to applying BDD at a unit level, as a variant of TDD. If you're looking to wrap automated tests around your whole environment, take a look at Cucumber. It makes reusing steps a lot simpler. You can then call Selenium directly from your steps.
I put together a page of links on BDD here, which I hope will help newcomers to understand it better. Best of luck.
Reading the book Working Effectively with Legacy Code might be helpful. There is also a short version in PDF form.
I'm looking for an implementation strategy to ease A-B testing / Beta testing. I don't see any code/plugin available for any framework. If not for a direct solution, let us at least brain-storm the requirements/expectations from the component:
There are already a few threads around my query..
Is there a PHP CMS with builtin A/B Testing Support?
Anyone got any good strategies for A/B testing with the Play Framework?
Beta Testing
As no one's answered this question, I'll attempt to do so.
Basically, I'm not sure if there's a directly useful connection between your PHP framework and your A/B testing needs. I think this is mainly because what you're testing can be almost anything: the colour of a conversion-sensitive button, a page layout, an entire registration funnel, etc. These don't inherently have anything to do with your PHP framework and there are lots of options for how you could do your testing.
Another issue is that you might not really know the parameters of what you're testing until you start testing. Your testing might lead you down a way that you didn't really even consider, so how could you have accounted for it in how you built the site? If you need a REALLY wide window for what you'll be testing, you're probably better off not building it at all and using some type of vapor/smoke-testing to get the basic concepts right first. Not everything can be subjected to testing and you'll still need subjectively-generated hypotheses as your test cases (and your testing will be only as good as your hypotheses).
If you have something very specific that you need to test repeatedly over time and want to build this flexibility into the system, then I'd look for the most obvious solution in the framework to make it happen. For example, if you're using Symfony and if you think that you'll need to test 50 different sidebar variations for a page over the course of 6 months, it probably makes sense to build it as a slot/component so you can build some logic around simplifying your testing and swap those sidebars with ease. I'm not sure why it would need to be anything more complicated than that.
Overall, I'd also add that the role of A/B testing should to guide your product to sell/convert/monetize/engage better. Unless you're building some type of a testing platform, I wouldn't over-think it. I tend to see that most sites fail to test sufficiently not because the system isn't flexible enough for various test cases but because top management won't give enough product/dev time for it, or because people aren't making enough use of their analytics packages to draw even the most basic of conclusions.
Hope that helps.
http://phpabtest.com/ looks like a pretty easy to use framework that comes free!
First i have no experience on parasoft .test or jtest experience. I have read the datasheet that the product could automatically generate unit test.
but I am woundering how useful the auto generated unit test are. Does it really do not need any other effort by developer?
any experience sharing are welcome.
thanks a lot!
We used JTest for our product recently. We didn't use the standard product, we used the Eclipse Plugin. The standard product is built on the OSGI framework (read: it's like Eclipse), but you have to import and create your projects. We were already using Eclipse, so it made sense for us to simply use the plugin, which has all of the same capabilities.
While there are many things that JTest can do for you, there are also many irritating things about it. For example, Jtest's static analysis tool is what is really worthwhile, IMHO. It can look for lots of errors and has a pretty good reporting system. But, while unit test generation is okay, but I think I spent as much or more time fixing and enhancing the generated tests than I would have just making them myself. Administering Jtest is also somewhat complicated and involved.
The built-in mechanisms to make unit tests, stub objects, parameterized unit tests, etc. are not well documented. At least, my little brain couldn't make good use of them in the two years we used the product. However, a lot of their super awesome features (like GUI tracing, command-line interface, the Bug Detective, reporting system etc.) all require extra, very expensive licenses.
Really, Jtest just gives you an easy way to manage the execution of static and unit testing. But it's really expensive. I can't believe they charge thousands of dollars per license of that stuff. You'll also find that they will want to train you, which you almost need because the documentation is pretty bad. Which is odd, because the user's guide is like 900 pages long.
But here's a big hint: you can do it for free. If I had to do it over, I would have pushed hard for using these products (which, oddly enough, look and feel very similar to Jtest)
http://code.google.com/javadevtools/codepro/doc/index.html
I wouldn't get Jtest thinking that this will be a small something to add to your developer's routine. Jtest can become a huge time and process sink.
Jtest is very very useful.Yes it generates it own test cases which requires lot more efforts for fixing them.I use it in different form.I delete all the generated unnecessary test cases.I made one another file which create database connection and set various other parameters sets.Also after configuration the code will work without mocking if all of the code is ready and if it is not ready than you can stubs the required methods.
Static code analyzer is good(for checking null pointer exception)
Checking code conventions is very good.
Write your custom code guidlines as use cases and execute it on your code.
Code coverage.
Debug while testing.
The auto generated unit tests still needs a developer to decide what results are correct or not, so you have to sit down and do the job. A lot of the boiler plate code is of course auto generated, so a small time saver there. I haven't used it much, but did evaluate jtest for an earlier employer. Seemed like a great product, if I remember correctly. :)
Alas there will never be a silver bullet that addresses all unit testing requirements, but JTest & .Test (& C++Test for that matter) about as close as you will get. Uggwar is correct that the developer will still need to verify outcomes for the basic auto generated tests, however there is a whole lot more to it.
These tools can be used to create basic regression tests, these are there to tell you when something has changes, not whether what it is testing is right or wrong. You can also trace a running application and then generate JUnit/NUnit/CPPUnit tests that recreate what was going on in the application. These tend to be far more useful tests, which are used as regression tests for items of functionality.
Other functionality includes the ability to generate stubs, use spreadsheets as datasources and provide an object repository. There is a while lot more too ....
Give them a try.
http://www.parasoft.com
I've started a new role in my life. I was a front end web developer, but I've now been moved to testing web software, or more so, automating the testing of the software. I believe I am to pursue a BDD (Behavior Driven Development) methodology. I am fairly lost as to what to use, and how to piece it together.
The code that is being used/written is in Java to write a web interface for the application to test. I have documentation of the tests to run, but I've been curious how to go about automating it.
I've been directed to Cucumber as one of the "languages" to help with the automation. I have done some research and come across a web site for a synopsis of BDD Tools/Frame works,
8 Best Behavior Driven Development (BDD) Tools and Testing Frameworks. This helped a little but then I got a little confused of how to implement it. It seems that Selenium is a common denominator in a lot of the BDD frameworks for testing a GUI, but it still doesn't seem to help describe what to do.
I then came across the term Functional Testing tool, and I think that confused me even more. Do they all test a GUI?
I think the one that looked like it was all one package was SmartBear TestComplete, and then there is, what seems to be, another similar application by SmartBear called, SmartBear TestLeft, but I think I saw that they still used Cucumber for BDDing it. There a few others that looked like they might work as well, but I guess the other question is what's the cheapest route?
I guess the biggest problem I have is how to make these tests more dynamic, as the UI/browser dimensions can easily change from system to system, and how do I go about writing automation that can handle this, and tie into a BDD methodology?
Does anyone have any suggestions here? Does anybody out there do this?
Thanks in advance.
BDD Architecture
BDD automation typically consists of a few layers:
The natural language steps
The wiring that ties the steps to their definition
The step definitions, which usually access page objects
Page objects, which provide all the capabilities of a page or widget
Automation over the actual code being exercised, often through the GUI.
The wiring between natural language steps and the step definitions is normally done by the BDD tool (Cucumber).
The automation is normally done using the automation tool (Selenium). Sometimes people do skip the GUI, perhaps targeting an API or the MVC layer instead. It depends how complex the functionality in your web page is. If in doubt, give Selenium a try. I've written automation frameworks for desktop apps; the principle's the same regardless.
Keeping it maintainable
To make the steps easy to maintain and change, keep the steps at a fairly high level. This is frequently referred to as "declarative" as opposed to "imperative". For instance, this is too detailed:
When Fred provides his receipt
And his receipt is scanned
And the cashier clicks "Refund to original card"
And the card is inserted...
Think about what the user is trying to achieve:
When Fred gets a refund to his original card
Generally a scenario will have a few Givens or Thens, but typically only one When (unless you have something like users interacting or time passing, where both events are needed to illustrate the behaviour).
Your page objects in this scenario might well be a "RefundPageObject" or perhaps, if that's too large, a "RefundToCardPageObject". This pattern allows multiple scenario steps to access the same capabilities without duplication, which means that if the way the capabilities are exercised changes, you only need to change them in one place.
Different page objects could also be used for different systems.
Getting started
If you're attacking this for the first time, start by getting an empty scenario that just runs and passes without doing anything (make the steps empty). When you've done this, you'll have successfully wired up Cucumber.
Write the production code that would make the scenario run. (This is the other way round from the way you'd normally do it; normally you'd write the scenario code first. I've found this is a good way to get started though.)
When you can run your scenario manually, add the automation directly to the steps (you've only got one scenario at this point). Use your favourite assertion package (JUnit) to get the outcome you're after. You'll probably need to change your code so that you can automate over it easily, by e.g.: giving relevant test ids to elements in your webpage.
Once you've got one scenario running, try to write any subsequent scenarios first; this helps you think about your design and the testability of what you're about to do. When you start adding more scenarios, start extracting that automation out into page objects too.
Once you've got a few scenarios, have a think about how you might want to address different systems. Avoid using lots of "if" statements if you can; those are hard to maintain. Injecting different implementations of page objects is probably better (the frameworks may well support this by now; I haven't used them in a while).
Keep refactoring as you add more scenarios. If the steps are too big, split them up. If the page objects are too big, divide them into widgets. I like to organize my scenarios by user / stakeholder capabilities (normally related to the "when" but sometimes to the "then") then by different contexts.
So to summarize:
Write an empty scenario
Write the code to make that pass manually
Wire up the scenario using your automation tool; it should now run!
Write another scenario, this time writing the automation before the production code
Refactor the automation, moving it out of the steps into page objects
Keep refactoring as you add more scenarios.
Now you've got a fully wired BDD framework, and you're in a good place to keep going while making it maintainable.
A final hint
Think of this as living documentation, rather than tests. BDD scenarios hardly ever pick up bugs in good teams; anything they catch is usually a code design issue, so address it at that level. It helps people work out what the code does and doesn't do yet, and why it's valuable.
The most important part of BDD is having the conversations about how the code works. If you're automating tests for code that already exists, see if you can find someone to talk to about the complicated bits, at least, and verify your understanding with them. This will also help you to use the right language in the scenarios.
See my post on using BDD with legacy systems for more. There are lots of hints for beginners on that blog too.
Since you feel lost as to where to start, I will hint you about some blogs I have written that talks a bit about your problem.
Some categories that may help you:
http://www.thinkcode.se/blog/category/Cucumber
http://www.thinkcode.se/blog/category/Selenium
This, rather long and old post, might give you hints as well:
http://www.thinkcode.se/blog/2012/11/01/cucumberjvm-not-just-for-testing-guis
Notice that versions are dated, but hopefully it can give some ideas as what too look for.
I am not an expert on the test automation but I am currently working on this part. So let me share some idea and hope it will help you at the current stage.
We have used selenium+cucumber+intellij for testing web application. We have used testcomplete+cucumber+intellij for testing java desktop application.
As to the test of web application, we have provided a test mode in our web application, which allows us to get some useful details of the product and the environment; and also allows us to easily trigger events through clicking the button and inputting text into the test panel under test mode.
I hope these are helpful for you.