In my Python development, doctest has really helped both to
make writing unit tests less annoying, and
integrate usage examples with documentation.
I was wondering, is there anything like this available in the Matlab world? It doesn't have to literally use code comments as a test, but if it had those two desirable qualities, that would be great!
This exists now! There are three versions out there:
An Octave-compatible version that still gets some development
A version of Matlab-xUnit with an integrated doctest runner
The original version from 2010, which is self-contained and emits results in TAP format
Not yet, but there is something to generate documentation called M2HTML.
It is very useful, and you can take a look at the examples at that page to see how wonderful results it's producing (even dependency graphs :) ).
For unit testing in MATLAB, even if there are many solutions, mlUnit was the most efficient to me.
There is no direct equivalent to doctest in MATLAB.
There is, however, a nice unit testing framework on the Matlab File Exchange.
Related
I'm looking for some tool which will provide me the code coverage of my functional tests (Not the unit testing code coverage ). To elaborate more, assume QA team executes their tests suites using selenium. At the end of the tests, I would like to know the amount of code (target code , not the test code base) got invoked / tested .
I found a similar post for .Net , but in my case the webserver is Apache and application server is jBoss
Coverage analysis for Functional Tests
Also, we have never done this type of analysis before, is this worth the effort, anyone who tried it ?
I used to do code coverage testing on assembly code and Java code. It is definitely worth it. You will find as you get the coverage close to 100% that it gets more and more difficult to construct tests for the remaining code.
You may even find code that you can prove can never be executed. You will find code on the fringes that has never been tested and you will be forced to run multi user tests to force race conditions to occur, assuming that the code had taken these into account.
On the assembly code, I had a 3000 line assembly program that took several months to test, but ran for 9 years without any bugs. Coverage testing proved its worth in that case as this code was deep inside a language interpreter.
As far as Java goes I used Clover: http://www.atlassian.com/software/clover/overview
This post: Open source code coverage libraries for JDK7? recommends Jacoco, but I've never tried it.
Thanks for the pointers #Peter Wooster. I did a lot of digging into the clover documentation but unfortunately there is not even a good indication that functional / integration is supported by clover, leave alone a good documentation.
Lucky I got to a link within the clover documentation itself which talks about this and looks promising (Thanks to Google search). I was using the ant so didn't even search in this Maven2 area. This talks about ant as well though :)
https://confluence.atlassian.com/display/CLOVER/Using+Clover+in+various+environment+configurations
I will be trying this , will update more on this soon !
I am looking for something that could visualize code coverage on the branch/statement level.
My understanding is that the test coverage tool provide by the test runner in Pharo and Squeak only tell me which methods haven't been called.
I also had a look at Hapao. First I found that the linked image was rather old, but even after loading the latest version into a fresh image, I still had the feeling that it provides me with a lot information that is not immediately helpful to answer my questions.
From the C world, I am used to things like gcov and lcov, where I can see immediately which branches haven't been covered.
Is there something similar available in the Pharo or Squeak universe?
Reading a recent thread on the Pharo mailing list gives me the impression: 'not yet'?
I know there is branch level code coverage in VisualWorks extensions (found in Cincom public store)
SUnitToo(verage)
http://www.cincomsmalltalk.com/publicRepository/SUnitToo(verage).html
SUnitToo(lsoverage)
similar address (I'm SO noob, and can't post more than 2 links)
See this blog too http://www.cincomsmalltalk.com/userblogs/mls/blogView?entry=3346628542
I don't know the license, nor if it would be easily portable though...
Stefan Reichhart implements and describes various cool tools to improve test quality in his master thesis. The tool Christo (described in Appendix B) provided sub-method level test-coverage and visualized coverage in the browsers. Unfortunately the code is unlikely to work in more recent Squeak/Pharo images, as it builds on top of libraries that got lost over the years (NewCompiler and ByteSurgeon).
I only recently started working on my latest Haskell project, and would really like to test it. I was wondering where testing currently stands, with regards to the cutting edge frameworks, test running procedures and test code organization. It seems that previously tests were just a separate binary that returned a different exit code if tests passed or failed - is this still the currently adopted setup, or are there other ways to integrate with cabal now?
Quickcheck may not be cutting edge anymore (at least for Haskell practitioners).
But in combination with HUnit it's quite easy to get almost 100% coverage (I use HPC for converage analysis).
What good (free) generator can be recommended for NUnit tests?
You're going to have to say exactly what you want the generator to do. Personally I've always found automated generators to be more hassle than they're worth - my fingers work just as well :)
On the other hand, you might want to look at Pex which works in conjunction with Code Contracts to explore your code and generate tests in an intelligent way. Pex is able to generate tests in different flavours, including NUnit.
Test generators are good at giving you high code coverage. Unfortunately high code coverage doesn't always mean good tests have been generated. I tend to write them out by hand myself.
Just downloaded and installed Novel's NUnitGenAddIn. It's kind of old-ish (seems to have been last updated in 2006), but once I tweaked the NUnitGenAddIn.AddIn file (change the Assembly path and update the Visual Studio version number to 9.0), it does exactly what I wanted: right-click generation of reasonable unit-test stubs from within Visual Studio 2008. Dunno if that works for what you want, but definitely free (GNU Lesser GPL).
Dont do it. Tests with value are generated first, by a human (ideally a pair thereof) with their thinking caps on, not their brain in neutral. (That is if you're referring to generating a test per method, or anything else that's not based on some deep insight a la Pex as Jon Skeet said)
If you use resharper you can right-click on a class or method and generate a test class plus one or more test methods (Sorry this was wrong it is part of vs2010)
Am looking for a regression test framework where I can add tests to.. Tests could be any sort of binaries that poke an application..
This really depends on what you're trying to do, but one of the features of the new Test::Harness (disclaimer: I'm the original author and still a core developer) is that if your tests output TAP (the Test Anything Protocol), you can use Test::Harness to run test suites written in multiple languages. As a result, you don't have to worry about getting "locked in" to a particular language because that's all your testing software supports. In one of my talks on the subject, I even give an example of a test suite written in Perl, C, Ruby, and HTML (yes, HTML -- you'd have to see it).
Just thought I would tell you guys what I ended up using..
QMtest ::=> http://mentorembedded.github.io/qmtest/
I found QMTest to full fill my needs. Its extensible framework, allows you to write very flexible test classes. Then, these test classes could be instantiated to large test suites to do regression testing.
QMTest is also very forward thinking, it allows for weak test dependencies and the creation of test resources. After a while of using QMTest, I started writing better quality tests. However, like any other piece of complex software, it requires some time to learn and understand the concepts, the API is documented and the User Manual give a good introduction. With sometime in your hand, I think QMTest is well worth it.
You did not indicate what language you are working in, but the xUnit family is available for a lot of different languages.
/Allan
It also depends heavily what kind of application you're working on. For a commandline app, for example, its probably easy enough to just create a shell script that calls it with a whole bunch of different options and compares its result to a previously known stable version, warning you if any of the output differs so that you can check whether the change is intentional or not.
If you want something more fancy, of course, you'll probably want some sort of dedicated testing framework.
I assume you are regression-testing a web application?
There are some tools in this kb article from Microsoft
And if I remember correctly, certain editions of Visual Studio also offer its own flavor of regression testing tools as well.
But if you just want a unit testing framework, the xUnit family does it pretty well.
Here's JUnit and NUnit.