I'm writing a tool to run a series of integration tests on my product. It will install it and then run a bunch of commands against it to make sure its doing what it is supposed to. I'm exploring different options for how to markup the commands for each test case and wondering if folks had insight to share on this. I'm thinking of using YAML and doing something like this (kinda adapted from rails fixtures):
case:
name: caseN
description: this tests foo to make sure bar happens
expected_results: bar should happen
commands: |
command to run
next command to run
verification: command to see if it worked
Does anyone have another, or better idea? Or is there a domain specific language I'm unaware of?
Go and have a look at the XUnit suite of test tools. This framework was originally designed for Smalltalk by Kent Beck and, I think, Erich Gamma, and it has now been ported to a whole stack of other languages, e.g. CUnit
You might want to check out CPAN. It does for Perl scripts exactly what it sounds like your utility will do for your app.
Did you take a look at RSpec?
Related
We need to be able to execute automation testing of Mumps code as a business requirement.
Is this possible? Any suggestions on where to start? I am a Developer, but I have no experience with Mumps.
Many thanks!
I am one of the maintainers of M-Unit. I used it extensively to run regression tests on M code. M-Unit is relatively easy to use.
There is integration of M-Unit with CMake if you want to get fancy... but that's hard to run and has lots of hardcoded variables. If you are interested in that, take a look at code in the OSEHRA VistA repository
I am doing an assignment (implementing an algorithm), and I wanted to try, this time, implementing tests first. However, I simply do not know where to put them! Do I need to create a new project?
All the tutorials I have found mentioned what to write, but not the general method of proceeding when building a test suite.
I expected Monodevelop to have some kind of predefined structure (like a big "add test suite" button), but I could not find anything for FSharp.
Monodevelop seems to have many tools to deal with tests in a clean and principled way (it does have a big "run tests" button), therefore I thought I would structure my project so that Monodevelop "sees" my tests, so that I could use the tools from the graphical interface. It seems the most common way to write tests is to use NUnit, what if I use something else, like FsCheck?
I have stumbled upon a Github project called FSharpKoans, it seems to suggest that I should create a project called "MySolution.Test" inside a solution, is this the standard way?
What should be the type of the project then, is it a separate console application?
Thanks.
Yes, creating a separate "X.Test" project is the standard way of adding tests in .NET, F# is no exception here.
For testing framework, pick one that Monodevelop supports well if what you're looking for is IDE integration - NUnit sounds like a safe choice. You could conceivably use any framework in F# tests, so I would think what Monodevelop supports should dictate your choice here.
FsCheck is not a testing framework, it's a library for property-based testing that can be used in conjunction with any testing framework. You might want to look into it as it advocates a particularly interesting approach to testing, but it's by no means the only way or the required way of doing testing in F#.
I was wondering what place scripting has in today's world of IDEs and GUIs.
I'm new to programming and am wondering at what point I should, if at all, open up the PowerShell terminal for a particular task. What do people here use scripting for and how important is it to a modern developer working full time with C++/C#/Java?
In general, you will want to write a script if you ever expect to do the task more than about twice. Making a script has the following advantages:
it's not dependent on human operation (humans are fallible)
it's repeatable
scripts can be stored in source control
This is well suited to tasks such as build, deployment, and automated testing.
IDEs and GUIs don't handle the testing or deployment parts of the development process well quite yet.
Scripting is also great when you want to do something quickly just to see. Personally, I use my Python interpreter as a super-calculator all the time. I also use it to parse files, combined with regular expressions.
Scripting is also often used to allow clients to easily extend a system's behavior. Using a script language, a compiler (or even an IDE) is not necessary.
See Extending Packages with Scripting for an example in SQL Server 2008.
Another point to add is that several servers which are highly used do not have any GUI. Everything is run from a terminal. Almost everything is run through a script. While GUIs are nice, they are not going to be used all the time.
I look for a tool/framework to make automatic acceptance-testing. The interface to create new tests should be so easy, that a non-programmer (customer, boss) will be able to add specifications for which will be tested automatically.
It should be some way to execute the tests from command-line, to include a run of the tests in automatic builds.
I prefer Java and Open-Source, but my question isn't restricted in that way.
What do you can recommend and please explain why your tool/framework is the best in the world.
http://fitnesse.org/ appears to meet all of the qualifications you want. It is one I have used with success.
I think that several of the options are very good and you should test them to see which fits your team :
Cucumber (Ruby)
Fitnesse
Robot framework (Python/Java)
Behave for Java
SpecFlow (.net)
Another framework you may want to look at is Robot Framework. To see how test cases look like, take a look at the Quick Start Guide.
I've found a framework named Concordion that may fulfill my needs.
What you ask for appears to be for a very well-defined system with a very specific sets of inputs and a high degree of automation built-into the system or developed for your system.
Commercial applications such as HP Quick Test Pro isn't non-technical enough and requires an additional framework such as one from Sonnet, which is a step in the right direction, but neither is open source or java-based.
Even with a framework in place, it's quite a bit of work to make this work in an automated way. I'd like you to consider the time needed to develop the framework vs the time to manually run these tests and verify that you are using your time well.
How about Cucumber:
Feature: Acceptance testing framework
Scenario: an example speaks volumes
Given a text example
When it is read
Then the simplicity will be appreciated
You would need a developer to discuss with the boss what each of those lines really means and implement the step definition to drive it:
Given /^a text example$/ do
file.open("example.txt", "w") { |file| file.write "text example" }
end
When /^it is read$/ do
SystemUnderTest.read("example.txt")
end
Then /^the simplicity will be appreciated$/ do
SystemUnderTest.simplicity.should be_appreciated
end
Am looking for a regression test framework where I can add tests to.. Tests could be any sort of binaries that poke an application..
This really depends on what you're trying to do, but one of the features of the new Test::Harness (disclaimer: I'm the original author and still a core developer) is that if your tests output TAP (the Test Anything Protocol), you can use Test::Harness to run test suites written in multiple languages. As a result, you don't have to worry about getting "locked in" to a particular language because that's all your testing software supports. In one of my talks on the subject, I even give an example of a test suite written in Perl, C, Ruby, and HTML (yes, HTML -- you'd have to see it).
Just thought I would tell you guys what I ended up using..
QMtest ::=> http://mentorembedded.github.io/qmtest/
I found QMTest to full fill my needs. Its extensible framework, allows you to write very flexible test classes. Then, these test classes could be instantiated to large test suites to do regression testing.
QMTest is also very forward thinking, it allows for weak test dependencies and the creation of test resources. After a while of using QMTest, I started writing better quality tests. However, like any other piece of complex software, it requires some time to learn and understand the concepts, the API is documented and the User Manual give a good introduction. With sometime in your hand, I think QMTest is well worth it.
You did not indicate what language you are working in, but the xUnit family is available for a lot of different languages.
/Allan
It also depends heavily what kind of application you're working on. For a commandline app, for example, its probably easy enough to just create a shell script that calls it with a whole bunch of different options and compares its result to a previously known stable version, warning you if any of the output differs so that you can check whether the change is intentional or not.
If you want something more fancy, of course, you'll probably want some sort of dedicated testing framework.
I assume you are regression-testing a web application?
There are some tools in this kb article from Microsoft
And if I remember correctly, certain editions of Visual Studio also offer its own flavor of regression testing tools as well.
But if you just want a unit testing framework, the xUnit family does it pretty well.
Here's JUnit and NUnit.