I am doing an assignment (implementing an algorithm), and I wanted to try, this time, implementing tests first. However, I simply do not know where to put them! Do I need to create a new project?
All the tutorials I have found mentioned what to write, but not the general method of proceeding when building a test suite.
I expected Monodevelop to have some kind of predefined structure (like a big "add test suite" button), but I could not find anything for FSharp.
Monodevelop seems to have many tools to deal with tests in a clean and principled way (it does have a big "run tests" button), therefore I thought I would structure my project so that Monodevelop "sees" my tests, so that I could use the tools from the graphical interface. It seems the most common way to write tests is to use NUnit, what if I use something else, like FsCheck?
I have stumbled upon a Github project called FSharpKoans, it seems to suggest that I should create a project called "MySolution.Test" inside a solution, is this the standard way?
What should be the type of the project then, is it a separate console application?
Thanks.
Yes, creating a separate "X.Test" project is the standard way of adding tests in .NET, F# is no exception here.
For testing framework, pick one that Monodevelop supports well if what you're looking for is IDE integration - NUnit sounds like a safe choice. You could conceivably use any framework in F# tests, so I would think what Monodevelop supports should dictate your choice here.
FsCheck is not a testing framework, it's a library for property-based testing that can be used in conjunction with any testing framework. You might want to look into it as it advocates a particularly interesting approach to testing, but it's by no means the only way or the required way of doing testing in F#.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
Guys I am looking for Web Testing Generic Automation Framework which can be used to do automation testing of various web based applications .Looking for C# based framework as that is the language I am more familiar with. But any other language framework will also do and it should not use any proprietary/licensed language.
Framework should have some open source and free of cost license model.
I searched for selenium based framework on Google and SO. But could not come with any which have source code available. It will be good if the framework encapsulates all the functionality provided by Selenium WebDriver and/or Selenium RC and empower the functional tester to create and maintain test in human readable scripts.
Requirements of the framework:
The framework code should avoid hard coding of test steps. My idea is to maintain the test scripts outside the automation framework code , so that they can be easily be modified if needed. The framework should read through the step tables and the data tables and run the test accordingly.
If there is no such framework available now right then we can collectively build such a framework in a open source community model.
P.S.
I have read a little about Hermes Framework and Robot Framework, but not yet tried them, any help is welcome.
The good side of this problem: there are a lot of flexible tools and approaches, you can get together and build a flexible, reliable and robust test automation framework.
The hard part is: yes, there is no “out of box” solution, and you’ll need to find and put together lots of tools in order to solve this test automation puzzle.
What I would recommend:
First you need to choose a unit-test test framework. This is a tool which helps to identify separate methods in code as tests, so you can run them together or separately and get the run results, such as pass or fail.
My personal opinion, is that the testing tool – MS-Test – which ships with Visual Studio 2013 (and also Express Edition) is good enough. Another alternatives are: NUnit or Gallio Icarus
All unit-testing frameworks includes a mechanism for doing assertions inside the test. The capability of assertions class depends on given unit-testing framework. Here, I would like to recommend a popular library which works great for the entire unit testing framework.
This is Fluent Assertions (also available from NuGet repository).
That’s a hard moment. You need to decide: are you going to use the PageObject approach in order to build your test automation framework, or you are going to choose simpler approach, without heavy utilization of the Object Oriented Programming.
Properly designed Page Objects makes your test automation code much maintainable. Utilizing the OOP – you can do a magic in your code: write less to do more. Although, such approach requires more skill.
Here are a good articles on this topic:
Maintainable Automated UI Tests
And this one:
Tips to Avoid Brittle UI Tests
The alternative to the PageObject is a scripted approach. This approach can be also successful and requires less time to start.
Coypu is a good and usable example of such framework for Selenium Web Driver.
All the popular unit-testing frameworks support data-driven tests. The best support is in NUnit – you can run/re-run and see the tests generated for individual data row in the tests tree.
MS-Test supports reading data from different data-sources: text files, excel, mssql etc., but it is not possible to re-run the test for individual data row. Although, there is a hack for this – Ms-Test Rows.
For my data-driven tests, I am using a great library – Linq to Excel
I have a lot more to say. There are so many approaches to build test automation framework – and there is no ready solution yet.
I am trying to build one according to my testing methodology – SWD.Starter .
This project is still on its early development stages. But, at least, probably you’ll find a few tips how to build and organize the test automation code.
I've implemented https://github.com/leblancmeneses/RobustHaven.IntegrationTests based on my prior experience on large projects "trying" to implement full end to end testing.
I've been using this and and have a lot of useful extensions for general selenium, angularjs, and kendo ui work. Since this framework is not obtrusive you could just use these extensions without using anything else.
I'm using this on my latest project and everyone is loving it.
There are a lot of bdd/spec frameworks (specflow, mspec, nspec, storyq) to help wire the behavior of your system to tests.
What I've learned:
make it frictionless for any .net developer/tester to begin writing/running tests.
Most fail here because it requires installing additional pluggins into visual studio.
mine uses the standard nunit
Logically you would think that a feature is a class file and scenarios are [Test] methods - to support some of these frameworks they make each scenario a class file.
use the original spec to create stubs of your tests - hopefully readable code
I used spec flow back in 2010 - so things might have changed. I generated my tests from my bdd document. A year later when I went to add more tests and update existing tests, I felt I wasted a lot of time with ceremony than writing code I really wanted - I stopped using it.
My approach uses t4 to generate stubs - developer has a choice to generate from feature file, for a specific scenario or don't use generated code at all.
how is state shared across steps / nested steps
most use dictionary<string,object> to help you separate data from being hardcoded in your tests accessed from a context object.
mine uses viewmodels and pointers to those viewmodels - if your using something like angularjs you are using viewmodels in your server side display/editor templates and in angularjs controller so why not reuse these in your tests!
start early with CI - make development transparent
My project has ResultDiff that given the nunit testresult.xml file, folder location to your gherkin feature files, and output json file; Read description on why this is important on the screenshot: https://github.com/leblancmeneses/RobustHaven.IntegrationTests#step-5-ci-setup-resultdiff
Example:
Modified means business and developers have a mismatch of Gherkin statements - did something change that we need to talk about?
What is missing? a dashboard to render the .json file created by ResultDiff. It's on my backlog.....
With a centralized dashboard that supports multiple environments(branches of your code) this dashboard will serve all stakeholders (business, developers) what is the status of features being developed.
There is a framework named "omelet" which is built in java on top of testng for selenium,
For cross browser multi-parallel testing , it easily blends with your CI tools and have some cool reporting features with step level reports
Running your test cases on BrowserStack and Grid was never so easy as with omelet with few config changes.
if you want to give it a try then do follow the 5 min tutorial available on the website, there is archetype available on maven central + there are many more features available
Stable version is 1.0.4 and we are currently looking for people to contribute to project.
Documentation over here
Github link
I am writing a winforms application and eventually I would like to write unit test for this application from the DAL, and Biz Objects layers etc.
Does someone know of a FREE tool that can recieve the path to an assembly and then output unit test stubs with matching signatures for the assembly.
Any configurable options like "public interfaces only", "test framework choice", "language choice" would be a plus.
I at least would need this tool to emit vb.net against nunit.
Thanks.
Seth
Last I heard, the recommended method of unit testing was to write them as you develop the functionality in a test first style. Auto-generating unit test stubs, in my mind, would just result in a whole bunch of unimplemented unit tests which add no value and will most likely have very awful generic names that don't describe the behavior being tested.
On the other hand, maybe I'm just misunderstanding your question...
Take a look at Pex from Microsoft Research.
MS Unit test build into VS2008 can create stubs (using Reflection) in your behalf.
I found it very useful in most cases.
There is a tool called Pex that not only makes the stubs, but also fills in tests for you. There's also a video online.
edit: Mark Seemann beat me to it! Hopefully the links are still useful.
I am responsible for rewriting an internal tool for my company. I am currently reworking the most time consuming step to run faster which should give me time to re-think the design of the application for a full rewrite as the interstitial version will meet the current needs.
I really want to take this opportunity to implement this code using BDD/TDD but I am new to this method of programming in general and especially within the context of .NET. Are there BDD/TDD tools available for .NET? What resources should I look at?
Thank you very much in advance!
Ashish
As one of the members joked when we discussed BDD at one of our alt.net oresund meetings: "there are more BDD frameworks for .Net, than there are people using them" ;-)
When starting with BDD, or having executable acceptance tests, I recommend to first understand "what's in a story" (i.e. how to define the requirements.
Having nailed a few of those, then you can go find a tool that fits your particular needs in your particular context.
At my current project we chose StoryQ as our BDD tool since we were already using NUnit and TestDriven.Net, and didn't have to add anything but an assembly reference (no separate test runner, etc).
At the aforementioned alt.net meeting, one of the members demonstrated using Cucumber under IronRuby for an app written in C#.
Many people appreciate the way Cucumber turns plain English into requirements, why that might be worth investigating.
Resharper, Nunit, xUnit, MBUnit, are just some of the things that spring to mind.
Resharper because of the nice testrunner and the neat refactorings it can do. (not free)
NUnit because I use that now. (free)
NCover because sometimes you miss a spot. (not free anymore).
Structuremap because IoC/Di is a given. (free)
Rhino mocks because you need to stub and mock. (free)
ASP.Net MVC if you need webdevelopment.
That is just my current stack.
I forgot a few others. But most things that work in C# will now work in VB.Net. And they will work even better in VB.Net 10
This answer to a question about C++ unit test frameworks suggests a possibility that had not occurred to me before: using C++/CLI and NUnit to create unit tests for native C++ code.
We use NUnit for our C# tests, so the possibility of using it for C++ as well seems enticing.
I've never used managed C++, so my concern is are there any practical limitations to this approach? Are many of you doing this? If so, what was your experience like?
We do this all of the time. We have many assemblies written with C++/CLI and use C# and NUnit to test them. Actually, since our goal is to provide assemblies that work well with C#, doing this makes sure that we have accomplished that.
You can also write NUnit tests in C++/CLI and call unmanaged C++. Probably the best way is the keep your pure unmanaged C++ in a lib, and then make a test assembly that uses NUnit and links to the lib.
It works very well and gives you the benefit of having parameterised tests as well as a common test runner and framework if you're in a mixed environment.
There are two downsides, neither of which is serious for most cases:
If you're being really picky, the tests are no longer being run in a purely native environment so there's an outside possibility that something may work under test but fail at runtime. I think you'd have to be doing something fairly exotic for this to matter.
You rely on your C++ code being able to be included into a C++/CLI program. Sometimes this can have clashes with headers and it forces your code to build OK with UNICODE. In general, this is a good thing as it uncovers crufty bits of code (like inconsistent use of Ansi variants of Win32 calls). Bear in mind that it's only the headers being included so what it may well show is that you are exposing headers at too high a level - some of your includes should probably be within your cpp implementation files.
My experience is that it is not possible to use NUnit to test C++ native code through C++/CLI because you will have trouble loading and using native code.
I have tried using nunit to load a basic c++/cli test dll linked against "just thread" which is an implementation of the c++ standard thread library.
Test dlls won't even load with the latest version of NUnit (2.6.2).
So definitely not the way to go!
I never used one, but isn't there a port? Perhaps http://cunit.sourceforge.net/documentation.html would work for you.
The biggest concern is the learning curve of the C++/CLI language (formerly Managed C++) itself, if the tests need to be understood or maintained by non-C++ developers.
It takes a minimum of 1-2 years of C++ OOP experience in order to be able to make contributions into a C++CLI/NUnit test project and to solve the various issues that arise between the managed-native code interfaces. (By contribution, I mean being able to work standalone and able to make mock objects, implement and consume native interfaces in C++/CLI, etc. to meet all testing needs.)
Some people may just never grasp C++/CLI good enough to be able to contribute.
For certain types of native software libraries with very demanding test needs, C++/CLI/NUnit is the only combination that will meet all of the unit testing needs while keeping the test code agile and able to respond to changes. I recommend the book xUnit Test Patterns: Refactoring Test Code to go along this direction.
Am looking for a regression test framework where I can add tests to.. Tests could be any sort of binaries that poke an application..
This really depends on what you're trying to do, but one of the features of the new Test::Harness (disclaimer: I'm the original author and still a core developer) is that if your tests output TAP (the Test Anything Protocol), you can use Test::Harness to run test suites written in multiple languages. As a result, you don't have to worry about getting "locked in" to a particular language because that's all your testing software supports. In one of my talks on the subject, I even give an example of a test suite written in Perl, C, Ruby, and HTML (yes, HTML -- you'd have to see it).
Just thought I would tell you guys what I ended up using..
QMtest ::=> http://mentorembedded.github.io/qmtest/
I found QMTest to full fill my needs. Its extensible framework, allows you to write very flexible test classes. Then, these test classes could be instantiated to large test suites to do regression testing.
QMTest is also very forward thinking, it allows for weak test dependencies and the creation of test resources. After a while of using QMTest, I started writing better quality tests. However, like any other piece of complex software, it requires some time to learn and understand the concepts, the API is documented and the User Manual give a good introduction. With sometime in your hand, I think QMTest is well worth it.
You did not indicate what language you are working in, but the xUnit family is available for a lot of different languages.
/Allan
It also depends heavily what kind of application you're working on. For a commandline app, for example, its probably easy enough to just create a shell script that calls it with a whole bunch of different options and compares its result to a previously known stable version, warning you if any of the output differs so that you can check whether the change is intentional or not.
If you want something more fancy, of course, you'll probably want some sort of dedicated testing framework.
I assume you are regression-testing a web application?
There are some tools in this kb article from Microsoft
And if I remember correctly, certain editions of Visual Studio also offer its own flavor of regression testing tools as well.
But if you just want a unit testing framework, the xUnit family does it pretty well.
Here's JUnit and NUnit.