I'm looking for a good, preferably free, test planning and documentation tool. Ideally something that will keep track of which tests have been run against which software version, with reporting ability. There's a whole bunch of tools listed here but are there any others, and which ones have you had the best experience with? (You do run tests, right?)
UPDATE 2008-01-29
So far TestLink and Fitness have been mentioned. A related question yielded also a link to the ReadySet project, an open collection of software planning documentation templates.
I have used TestLink and found it okayish, but I cannot say I enjoyed using it. Has anyone had any experience with Fitnesse? Or are there any other free tools out there that you have used and found satisfactory?
You should definitely try out Klaros-Testmanagement http://www.klaros-testmanagement.com which has an free, unrestricted Community Edition.
Yes, we do run tests, but not nearly as many as I'd like !
I highly recommend TestLink - the list of tools that you linked to shows that it's had more downloads than all of the other tools put together.
I've used QualityCenter/TestDirectory for a long time.
I'm now using testlink and I must say that I prefer QualityCenter/TestDirectory by far, even if it based on some buggy ActiveX control.
QualityCenter/TestDirectory is more easier to use and the interface is quite better.
TestLink and QualityCenter/TestDirectory are mainly for manual test case (however, you can use Quick Test Pro on QualityCenter/TestDirectory to automatize your tests).
Fitnesse is another kind of tool in my mind : basically, you write your test case on a wiki and link that to a JUnit test. Another tools like that are GreenPepper, Concordion, etc.
PractiTest is a very good option. Not free but very affordable - http://www.practitest.com
A little late on this one, but I would have to suggest you try TestLodge for your manual test management. It works in a simular way to what TestLink does, but it gives off a more professional interface and is something that we also allow our clients to use for their uat phase.
We use Quality Center / Test Director stuff. Its expensive as far as I know, and it's not that great.
I've heard good things about Fitnesse but I don't know how good it's test tracking is.
I know I just recently saw a slick looking test tracker for Trac or something, but I can't find it now...
Quality Center. It's expensive, but it is the best
I'm with Patrick - good ol' office tools :)
I just write mine in Microsoft Word
This is the structure I developed: Writing a System Test Plan.
One thought, and perhaps not a good one, would be to have every test submit a ticket to your ticketing system when it's run indicating the test name, build version, and date, and test results.
That would make the results searchable later-on.
We use a home-grown Access database.
This database keeps track of our requirements, test cases, test plan and test runs. We're able to produce an up-to-date RVTM, keep track of progress against the plan, and assign tasks to testers. We integrated it with Outlook, so each tester is assigned a task from the plan by the QA lead. When they're complete, they just tick it off in Outlook and it updates the database.
For our small team of testers it works nicely, and we're free to customize it however we want.
Related
We are a couple of entrepenours who have developed a cross-browser app and a backend administration system for the app. Or actually, we paid a company to develop it. Now we want it tested profesionally, but we dont want to use the same company for this purpose.
The tests may involve
Integration Testing
Functional Testing
System Testing
Stress Testing
Performance Testing
Usability Testing
For some of the tests, we think that the actual source code is required. We dont feel komfortable giving our source code away "just like that", to unknown parties, so what are the common methods for having external companies testing ones software?
you don't really need to give the source to perform the mentioned tests. you need to provide working environments or provide binaries and instructions how to deploy them. it seems sufficient for 1,2 (i don't know what does 3 means), 4 and 5. It's way too late for usability testing. it should have been done during UI design phase (how do you want to test it right now?)
but those tests are not sufficient. you forgot about penetration testing. and above tests are black box testing and they can show you how the application works.
but if you have any real plans for this application you must be sure it's maintainable. and for this you need the whitebox testing, you have to analyze the code.
you can start with automated analyzing to check the overall quality of the code. but at the end you will still need good programmers to perform the code review. but you don't really have to give them the code. you can invite them to your office and let them review the code on your workstations. unless your idea is so simple and brilliant that only one look at the code is enough to reproduce it. in such case you will need to sign NDA or give some shares to the expert who will take care of the quality
Our company is small and in a project only 1 or 2 testers are assigned. And our all test related things are maintained on Excels sheets. And for bug tracking we are using Mantis. We create test cases on Excel sheet and execute them via same.
Is TestLink or any other test management tool will be helpful to us or not. As number of testers are less so there are no merging of test cases are done. Only one QA develop test cases and execute it. Please suggest me if it will be any help to us or not.
If so please suggest only free application
I am working for a startup and we are pretty much using Testlink. Our QA team is always pretty small (in between 1-3). It's very helpful for organizing and keeping the test cases for your whole system. It becomes more useful when you go for a release. You can assign your testers based on a test build so that they can go through test cases one after another and mark which tests are passing or not. Finally, you can generate report based on those for your build.
Hope that helps.
Regardless of if there is only one tester or many, it is still a good practice to make use of a test management tool and using a lightweight solution will make you more productive.
There are many benefits over using a static excel file and we recently put together a short blog post which goes into a little detail of the benefits to organizing your testing process with a test management tool which may be of interest.
If you are using Mantis to track your issues, you will often find that test management tools integrate with tools like this so that when a test fails a ticket is automatically created and this is a huge time saver.
If there is some sort of debugging API that would allow someone to write his own code coverage analysis tools that would be acceptable as well. I don't believe this is a poll question because the exhaustive list of such tools is probably quite small.
There are no readymade coverage tools (afaik). However, you might be able to utilize the MONLBL facilities (here). These were introduced to be able to identify performance hotspots.
It would be interesting to look into the code of these, as the necessary hooks to create a code coverage tool are quite similar to the ones you need to create the performance monitoring. If you have access to the WRC I strongly recommend to raise this with ISC, it definitely sounds like something many people could use!
HTH
edit: come to think of it, since monlbl gives you the # of times a line is executed, it's a code coverage with benefits ;) So, the answer is: yes, there are
debuging in Cache' Studio.
there is also ^PERFMON
as of right now i'm working at place where's there's a lot of legacy codes and pretty much no useful documentation.
and most of the time we just treat business requirements as whatever that is already implemented previously.
i'm looking for any tools or useful method to keep all the requirements for future use and for regression testing mostly.
i'm thinking of maybe linking them up to tests/unit test too so that the business requirements are linked directly to the coding logic.
Any good tools or resources to get me started?
thanks~
Updates
As of now i'm making things simple on myself by writing use case and then create some simple use case diagram using this awesome tool and then convert each use case into a test plan. The test plan is meant for the end user, thus i just make it into a simple step by step flow. I had plans to automate this part using selenium but it wasn't working that well on our website and was taking too long. It's a bit TDD, but i think it create simple understandable goal for both end user and the developer, i hope.
So for now it's just excel and doc file, lugged into the project doc folder and check into cvs/svn doomed to be outdated and be forgotten :P
Business requirements can be well capture in FitNess tests. As for Unit Test they sur help, and but both together in continuous integration like Hudson to detect regression ASAP.
PS: Sorry pretty much all links go to some articles I wrote because I'm also interested in that subject.
Here are some methods/systems that I have used
HP Quality Center
Big and bulky. Not very agile but it works and has a lot of feautres.
It's used in many larger corporations and if you can afford you can get great support from HP
https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24%5E1131_4000_100__
Bugzilla-Testopia
Open Source test case management extension for Bugzilla, managed by Mozilla. Which is good enough in my book to give it a try.
http://www.mozilla.org/projects/testopia/
Excel/Open Office Calc
Just do everything in spreadsheets and link between them.
Very flexible, everybody knows how to use them and you propbably have the software in your organization already.
Other Open Source Solutions
List of 15+ Open Source Test Management Tools
http://www.jayphilips.com/2009/09/10/15-open-source-test-management-tools/
In my limited experience with them executable requirements (i.e. specifying all requirements as broken automated tests) have proven to be amazingly successful. I've worked on one project in which we placed a heavy emphasis on creating high-level automated tests which exercised all the functionality of a given use case/user story. It was really amazing to me how much easier development became after we began this practice. Implementing features became so much easier after writing a test and we were able to make major architectural changes to the system with all the confidence in the world that everything still worked the same as it did yesterday.
The biggest problem we ran into was that the tools for managing these types of tests aren't very good. We used Fitnesse quite a bit and as a result I now hate the Fit framework.
I'd like to know 1) if anyone else has experience developing using this type of test-driven requirement definition and 2) what tools you all used to facilitate this.
The primary tool I've also used was FitNesse. I've used it at several companies, with very good results. We did have test cases numbering in the many thousands, and we had to be very disciplined in how we organized and used them.
I've tried some other tools, including writing my own DSL (domain-specific language) and using things like RSpec. I really like RSpec, but it is certainly more of a developer tool than a business one.
I know Rick Mugridge has been working on a tool called ZiBreve (http://www.zibreve.com/visit.php?page=index) which is supposed to have stronger refactoring support. I haven't used it myself, but I know Rick and have talked to him several times. I know there was discussion at Agile 2008 on some different ways to deal with the Fitnesse tests in general.
Other than that, I haven't seen a lot of good tools out there. Even tools like WinRunner are fine for QA type tests, but for exploratory testing of requirements by the business, FitNesse or a custom DSL seem to be the ways to go right now.
You might want to take look at Robot Framework (http://robotframework.org). It's FIT-like but hopefully easier to integrate to different testing tools, version control and continuous integration. Different abstraction levels in the test data also make it easier to maintain the data, and when the separate test data editor gets more mature maintenance gets even easier. The quick start guide introduces the most important features of the framework and acts also as an executable demo.
I've had to use, test and set up both fitnesse and one of it's competitor, GreenPepper for my work, and what I can say is :
GreenPepper is a confluence plugin (confluence is an enterprise wiki from atlassian) and have many of the things you need in an "enterprise" level tool with little to no additional work required :
Better user friendly -rich text- wiki
syntax (makes it easier to work with
for non technical people)
It integrates very well with many
development tools : Eclipse, VB,
maven2 and Nant plugin, I tested most
and was very pleased.
User and access rights are managed by
confluence, which is to say it's good
and make uses of database of your
likin (which might be mandatory
depending on where you work)
Many other functionalities that might
or might not be required : ssl support, remote execution (install the wiki on unix, execute on windows if you are working on a C# project, or reverse)
Looks way better :D
Big downs for GreenPepper are : Configuration is quite hard and documentation is poor (although they seem to be working on it and they answer quite fast on their forum) and also it is not free, you have to pay for both confluence and GreenPepper, which might add up to quite a lot.
Fitnesse is very basic in my opinion, very easy to set up, it works but that's it, you can use some of the fitnesse plugins developed by the open source community, and even some Fit plugins, such as the Eclipse plugin (build the skeletton of the fixture from a fitnesse test file, provided it's in a .fit extension, very usefull). Integration is not ideal, authentification and access rights management is poor, but it's FREE and if you need something, you can do it because it's open source.
I've found that using contracts is a great approach. Metaprogramming contracts are generally lower-level than the types of integration tests you describe, but the two are certainly not mutually exclusive. I find contracts help keep documentation, implementation, and testing all in sync -- this is a major problem of TDD (not that it isn't a problem in non-TDD).
I've tried Fitnesse and its really awful (particularly integration with SVN).
And our company develop similar open-source tool with fit engine: FitPro
Another brilliant tool I've used is Concordion. It has the only disadvantage - requrements in html format
My experience is limited to personal projects and found much the same advantages you mentioned. I recommend http://metacpan.org/pod/Test::Simple::Tutorial which was my inspiration for trying out testing-based development. The perl testing modules seem pretty useful and flexible, though I have nothing to compare them to.
I also believe tests are vital for the maintenance period of a project. If you have good tests to begin with, it saves a lot of time and mistakes later on. I wish I had put more work into tests on my current project.