I'm looking for some tool which will provide me the code coverage of my functional tests (Not the unit testing code coverage ). To elaborate more, assume QA team executes their tests suites using selenium. At the end of the tests, I would like to know the amount of code (target code , not the test code base) got invoked / tested .
I found a similar post for .Net , but in my case the webserver is Apache and application server is jBoss
Coverage analysis for Functional Tests
Also, we have never done this type of analysis before, is this worth the effort, anyone who tried it ?
I used to do code coverage testing on assembly code and Java code. It is definitely worth it. You will find as you get the coverage close to 100% that it gets more and more difficult to construct tests for the remaining code.
You may even find code that you can prove can never be executed. You will find code on the fringes that has never been tested and you will be forced to run multi user tests to force race conditions to occur, assuming that the code had taken these into account.
On the assembly code, I had a 3000 line assembly program that took several months to test, but ran for 9 years without any bugs. Coverage testing proved its worth in that case as this code was deep inside a language interpreter.
As far as Java goes I used Clover: http://www.atlassian.com/software/clover/overview
This post: Open source code coverage libraries for JDK7? recommends Jacoco, but I've never tried it.
Thanks for the pointers #Peter Wooster. I did a lot of digging into the clover documentation but unfortunately there is not even a good indication that functional / integration is supported by clover, leave alone a good documentation.
Lucky I got to a link within the clover documentation itself which talks about this and looks promising (Thanks to Google search). I was using the ant so didn't even search in this Maven2 area. This talks about ant as well though :)
https://confluence.atlassian.com/display/CLOVER/Using+Clover+in+various+environment+configurations
I will be trying this , will update more on this soon !
Related
How automated test works and how can I try one? How tools for automated testing works and what they do?
If possible please post with examples to clarify the ideas.
Any help on this topic is very welcome! Thanks.
Automated testing means writing a script for the tasks that we test manually.
Tools include softwares where we write a few lines of code in a sequence as we wish to perform a partiular test. Then running that script to perform the tests and generate results.
Automated testing saves the hours that we manually spend in repeating a series of test cases for a particular test.
Probably the best place to start is the xUnit libraries (JUnit, PHPUnit, jsUnit, etc.). If you're interested in testing of Web interfaces, there's something called Selenium. They provide lots of code examples to look at. These tools allow you to set up some input values, run some code, and then it allows you to verify the final output of that code matches your expectations (aka assertions).
In more sophisticated development teams, these automated tests are run every time new code gets submitted to the project to make sure no new bugs are introduced. As Priyanka mentioned, they save lots of time and eliminate the possibility of human error because the tests are run automatically where they would otherwise be done manually.
I'm sorry for not being more specific. This is a very broad topic of discussion.
I have been writing tests for my Ruby code for a while, but as a frontend developer I am obviously interested in bring this into the code I write for my frontend code. There is quite a few different options which I have been playing around with:
CasperJS
Capybara & Rspec
Jasmine
Cucumber or just Rspec
What are people using for testing? And further than that what do people test? Just JavaScript? Links? Forms? Hardcoded content?
Any thoughts would be greatly appreciated.
I had the same questions a few months ago and, after talking to many developers and doing a lot of research, this is what I found out. You should unit test your JavaScript, write a small set of UI integration tests and avoid record and playback testing tools. Let me explain that in more detail.
First, consider the test pyramid. This is a interesting analogy created by Mike Cohn that will help you decide which kind of testing you should be doing. At the bottom of the pyramid are the unit tests, which are solid and provide fast feedback. These should be the foundation of your test strategy and thus occupy the largest part of the pyramid. At the top, you have the UI tests. Those are the tests that interact with your UI directly, like Selenium does for example. Although these tests might help you find bugs, they are more expensive and provide very slow feedback. Also, depending on the tool you use, they become very brittle and you will end up spending more time maintaining these tests than writing actual production code. The service layer, in the middle, includes integration tests that do not require an UI. In Rails, for instance, you would test your REST interface directly instead of interacting with the DOM elements.
Now, back to your question. I found out that I could greatly reduce the number of bugs in my project, which is a web application written in Spring Roo (Java) with tons of JavaScript, simply by writing enough unit tests for JS. In my application, there is a lot of logic written in JS and that is the kind of thing that I am testing here. I am not concerned about how the page will actually look or if the animations plays as they should. I test if the modules I write in JS will execute the expected logic, if element classes are correctly assigned and if error conditions are well handled. For these tests, I've been using Jasmine. This is a great tool. It is very easy to learn and has nice mocking capabilities, which are called spies. Jasmine-jQuery adds more great functionality if you are using jQuery. In particular, it allows you to specify fixtures, which are snippets of the HTML code, so you don't have to manually mock the DOM. I have integrated this tool with maven and these tests are part of my CI strategy.
You have to be careful with UI tests, specially if you rely on record/playback tools like Selenium. Since the UI changes often, these tests keep breaking and you will spend a lot of time finding out if the tests really failed or if they are just outdated. Also, they don't add as much value as unit tests. Since they need an integrated environment to run, you will mostly like run them only after you finished developing, when the cost of fixing things is higher.
For smoke/regression tests, however, UI tests are very useful. If you need to automate these, then you should watch out for some dangers. Write your tests, don't record them. Recorded tests usually rely on automatically generated xpaths that break for every little change you do on your code. I believe Cucumber is a good framework for writing these tests and you can use it along with WebDriver to automate the browser interaction. Code thinking about tests. In UI tests, you will have to make elements easier to find so you don't have to rely on complex xpaths. Adding class and id elements where you usually wouldn't will be frequent. Don't write tests for every small corner case. These tests are expensive to write and take too long to run. You should focus on the cases that explore most of your functionality. If you write too many tests at this level you will probably test the same functionality that you have previously tested on your unit tests (supposing you have written them).
In my current project I am using Spock and Geb to write the UI tests. I find these tools amazing. They are written in Groovy, which suits better my Java project.
There are lots of options and tools for that. But their choice depends on whether you have a web UI or it's a desktop app?
Supposing from the tools you've mentioned it's Web UI. I would suggest Selenium (aka WebDriver): http://seleniumhq.org/docs/
There is a variety of languages it supports (Ruby is in the list). It can be run against a variety of browsers, ad it's quite easy to use with lots of tutorials and tips available.
Oh, and it's free, of course :)
I though as this post gets a lot of likes, I would post my answer to my question as I do write lots of tests now and how you test front end has moved on a lot now.
So in terms of FE testing I spent lot of time using karma with Jasmine, although karma will work nicely with other test suites like mocha & qunit. While these are great and karma allows you to interface directly with browsers to run your tests. The downside is as your test suite gets large it can become quite slow.
So recently I have moved to Jest which is much faster and if your writing react app, using enzyme with snap shot testing give you really good coverage. Talking of coverage Jest has Istanbul coverage built in and set up and mocking is really easy simple to use. The downside it doesn't test in browser and it using something called jsdom which is fast, but does have a few nuisances. Personally I don't find this a big deal particularly when I compile my code through webpack/babel which means the cross browser bugs are fairly few and far between, so generally isn't an issue if you manually test anyway (and imo you should).
In terms of working within the rails stack, this much easy now that the webpacker gem is now available and using npm and node is generally much more excepted. I would recommend using nvm to manage your node versions
While this isn't strictly testing, I would also recommend using linting as this also picks up a lot of issues in your code. For JS I use eslint with prettier and scss/css I use stylelint
In terms on what to test, I think as Carlos talks about the test pyramid is still relevant, after all the theory doesn't change, just the tools. I would also add to be practical about tests, I would always test, but to what level and coverage will depend on the project. It is important to manage your time and spending hours/days testing a short lifecycle project. Larger/longer term projects the benefits of a larger test suite is obviously greater.
Anyway I hope that helps people that look at the question.
I only recently started working on my latest Haskell project, and would really like to test it. I was wondering where testing currently stands, with regards to the cutting edge frameworks, test running procedures and test code organization. It seems that previously tests were just a separate binary that returned a different exit code if tests passed or failed - is this still the currently adopted setup, or are there other ways to integrate with cabal now?
Quickcheck may not be cutting edge anymore (at least for Haskell practitioners).
But in combination with HUnit it's quite easy to get almost 100% coverage (I use HPC for converage analysis).
I'm currently using OCUnit that ships with Xcode 3.2.4 for doing unit testing of my application. My workflow is often to set some break points in a failing unittestin order to quickly inspect the state. I'm using Apple's OCUnit setup:
http://developer.apple.com/library/ios/#documentation/Xcode/Conceptual/iphone_development/135-Unit_Testing_Applications/unit_testing_applications.html
but the setup from above gives me some headaches. Apple distinguish between Application tests and Logic tests. As I see it:
You cannot debug logic tests. It's as if they're invisibly run when you build your project.
You can debug application tests, but you have to run these on the device and not the simulator (what is the reason for this?)
This means that everything moves kind of slowly with my current workflow. Any hints on getting app tests to run on the simulator? Or any pin pointers to another test-framework?
Would eg. google-toolbox-for-mac work better in general or for my specific needs?
Also, general comments on using breakpoints in the unit tests are welcome! :-)
I have used the Google Toolbox testing rig in the past and it worked fine, it ran both on the Simulator and the device and I could debug my tests if I wanted to. Recently I got fed up with bundling so much code with each of my projects and tried the Apple way.
I also find the logic/app tests split weird, especially as I can’t find any way to debug the logic tests now. (And if you’re using some parts of AVFoundation that won’t build for Simulator, you are apparently out of luck with logic tests completely, which seems strange.) One of the pros is that I can run the logic tests quickly during build.
I guess this does not help you that much – the point is that you can debug the tests under GTM. And you might also want to check out this related question.
I know this isn't really a good answer, nor is it completely helpful to your cause. But I've been using NSLog to run my unit tests (if nothing outputs to the console, then success). When I comment out my tests method then the tests wouldn't run. I found this much more predictable and reliable than OCUnit. I'd much rather use a real true unit tester, but it was too frustrating to deal with the often strange errors that could occur from OCUnit and also the other shortfalls/lack of features you describe above.
In my Python development, doctest has really helped both to
make writing unit tests less annoying, and
integrate usage examples with documentation.
I was wondering, is there anything like this available in the Matlab world? It doesn't have to literally use code comments as a test, but if it had those two desirable qualities, that would be great!
This exists now! There are three versions out there:
An Octave-compatible version that still gets some development
A version of Matlab-xUnit with an integrated doctest runner
The original version from 2010, which is self-contained and emits results in TAP format
Not yet, but there is something to generate documentation called M2HTML.
It is very useful, and you can take a look at the examples at that page to see how wonderful results it's producing (even dependency graphs :) ).
For unit testing in MATLAB, even if there are many solutions, mlUnit was the most efficient to me.
There is no direct equivalent to doctest in MATLAB.
There is, however, a nice unit testing framework on the Matlab File Exchange.