I'm looking for a way to automate testing for our TIBCO BW6 applications.
There are 2 test methods I would like to use: Unit tests and BDD.
For unit testing I would like to be able to call a subprocess with some in- and output, and assert the processed data.
For BDD I would like to use Cucumber, and test the entire flows from start to end.
To accomplish this, it would be nice if I could call (sub)processes from Java with some in- and output.
Does anyone know if this is possible?
Since in tibco bw 6 the paradigm is to have every process build as a soap/rest service, you may call these processes from java called the same way you would call a soap/rest service.
In case you have a subprocess implemented without a binding you may easily refactor this and test it from java code.
The BW 6.4.0 release introduced Unit Testing within BW6!
https://docs.tibco.com/pub/activematrix_businessworks/6.4.0/relnotes/GUID-10ED9A26-AED6-4A2D-B274-2E7A558DC98D.html
"You can now take the Test-Driven Design(TDD) approach for developing
ActiveMatrix BusinessWorks applications with the new Unit Testing
feature. You can define assertions for individual activities in the
BusinessWorks Process, test the assertions through the debugger and
see the test results in the Debug perpsective. Refer to Unit Testing
in the Application Development guide."
BW6 (6.4.2) Unit Testing Documentation :
https://docs.tibco.com/pub/activematrix_businessworks/6.4.2/doc/html/GUID-20758669-A9B4-4D84-95DB-0BFDA225E068.html
Related
I found it over a few references that it is recommended to go with the same stack for creating automation suite that is being used for the development of the application, i.e., using Java for writing tests if application is Java based.
Is there any specific reason for the same? Or is there any specific hindrance if not followed.
I would personally say that developers making an application in one language will probably feel more familiar writing the tests in the same language. And Integration tests should preferably be written by the developer who wrote the feature.
So using the same language for your test suite and your application development increases the odds your developers will be able to write and fix tests for features they create and fix themselves.
I think that should be team decision, the same as what testing frameworks to use. But it's totally fine to use for example Spock which is groovy based framework if your application is java based.
is there any tool out there that i can used to set-up run automatically and i was goggling and i found selenium test runner? there are so many tools out there its hard to figured out which is best
I'm using C# and using MSTest as a test framework and I'm looking forward to see if I can get a way from testing in MSTEST
any help?
This is very subjective question. Every requirement will have its own correct answer. Anyhow I will try to address few requirements and will be updating as I learn more.
If you are automating web app browser tests (sans flash player and silverlight) I would say that selenium is the way to go. There are ways to automate flash and silverlight too, but that is answer for another question.
Selenium is anyways an automation too and your choice will rather is of which test framework to select. So here are few options:
1. Integrating with CI tools:
If you want to organize your tests as segregated atomic units and want them to be integrated to some CI server (e.g. TeamCity). I will recommend using NUnit to run your selenium tests.
2. Behavioral Tests
It is a new trend in the software development and how we test our products. Using behavioral (i.e. business specification) like language. In my experience it is also a very good format to write up acceptance tests. You can use selenium with something like Nbehave or SpecFlow
3. Centralize Test management and Execution
Now this might not fit for everyone but I have found FitNesse (and its c# binding) to be very useful in maintaining and executing selenium test cases.
Please note this answer may not be right and is certainly not complete given the scope of the question. I have nevertheless tried provide few pointers.
Our codebase contains code in multiple languages, ranging from Python over C# to MATLAB and LaTeX. Currently we have unit tests in each individual language (using language-specific frameworks). This makes test automation cumbersome, especially collecting and checking all the different reports.
I am therefore looking for a test automation tool that
is portable (at least Windows + Linux)
is language-independent
can be extended (custom reporting, additional languages, ...)
Ideally the tool would connect via plugins to the existing tool-specific frameworks like Python's unittest, C#'s NUnit, etc.
Is there such a tool? If not, how do you handle test scenarios like this?
Perhaps you're looking for something like the Test Anything Protocol?
The Test Anything Protocol (TAP) is a protocol to allow communication between unit tests and a test harness. It allows individual tests (TAP producers) to communicate test results to the testing harness in a language-agnostic way.
It doesn't avoid the need for language-specific tests, but it does make sure that your test results are all in a uniform format, so that you can build cross-langauge reports. It's also fairly simple, so if your language of choice doesn't have a TAP producer, it is not very difficult to write one (and maybe share it back to the world).
If you go with TAP as already suggested, then a complete test infrastructure around that is Tapper.
The central server assumes some external Linux tools when you need its automation layer but besides that everything else is language and OS independent. It can receive TAP reports regardless how they were started: manual tests, crontab-driven, or the built-in automation which can set up machines from scratch or start tests via ssh.
See above link for more info, especially have a look at the linked presentations.
Summary:
TAP as protocol for writing tests
Tapper for the infrastructure
Is anyone aware of any ongoing open source project that integrates robotframework with a load testing tool such as grinder, jmeter, funkload etc?
Thanks
Yes. There is a Python library for integration of Robot Framework and JMeter: Robot Framework JMeter Library . It can be used for running JMeter and parsing and converting results. I am author of this library so I might not be objective.
No, and that's likely not to happen. Robot Framework is for functional not load testing. How would you deem a load test as pass/fail and how long does it run?
Robot Framework and functional tests have a finite set execution time (takes as long as it needs to complete testing the particular feature or times out before doing so in case it hung, etc.), and has strict criteria as to what is pass/fail when test runs.
With load testing, you at least during exploratory runs and design of test, you don't run for fixed time, or even if fixed, it's usually not short (except trial runs and scalable burst increases). And criteria for pass/fail is usually within ranges rather than yes/no.
So it's harder to integrate and design a test library that can offer pass/fail and run within some set time for load testing. Unless someone can define a good architectural design of a test and test library for how to do so with Robot Framework.
I think the idea would be that a test case is created only once and can be used in both functional tests as in load tests and even in end user monitoring. In this (utopic) way a test case can be used during the whole lifecycle of an application. With a tag (for instance) a test case can be promoted to be also a loadtesting test case with another type of response validation. Would be nice to run Robot framework and create a Loadrunner-TrueClient (or another browser-driven loadtesttool) script. Main purpose of the integration would be to automate the scripting.
Are there any books or articles that show you how to use NUnit to test entire features of a program? Is there a name for this type of testing?
This is different from the typical use of NUnit for unit testing where you test individual classes. This is similar to acceptance testing except that it is written by the developer to discern that the program does what they interpreted as being what the customer wants the program to do. I don't need it to be readable by non-programmers or to produce a readable specification for non-programmers.
The problem I am having is keeping this feature testing code maintainable. I need help in organizing my feature testing code. I also need help organizing the program code to be drivable in this way. I am having a hard time being able to issue commands to the program while still having good code design.
Currently I have a class called Program with a single public method called Run. With every test I start at the beginning of the program like the user would and then get to the desired point in the program where a particular feature would be available. I then use that feature in some way and verify it did what I want. I have a class called Commands that exposes different features of the program as methods. An instance of the Commands object is passed to the program and it eventually gets passed to every Form class. These will subscribe to events from the Commands class that are called by the methods of the command class(one matching event per method). The events are subscribed to by pointed to the method that is called when a certain part of the user interface is used, thus allowing the entire program to be drivable by my tests. If you call a method on the Command object for an event that is currently not subscribed to, a FeatureMissingException is thrown.
All of this works but I don't like the Command class. It is getting too large with too many responsibilities (every feature of the program). The Commands class is also a dependency magnet (all the Form classes have an instance of it but only subscribe to the events that represent features that can be activated through their UI).
It's called integration testing. Integration tests are much more difficult to make automated, and are very often done by hand. Many simpler tests can still be done using NUnit though - you don't have to do anything special, just don't use Mocks (like you should be doing for unit tests) so you can test how the modules actually fit together.
Context/specification is a good way of organizing these tests.
What you want to do is integration testing, like the other answer suggests. This will allows you to functional/feature testing. The most common framework for this for StoryQ or SpecFlow. This allows you to develop your tests in a BDD style and can be mostly be automated against the spec that you want.
Tools like Selenium allow you to do functional testing in a browser to do what the end user would do. All of these can be driven with NUnit since NUnit is purely a framework for running tests be them Unit tests to large functional tests