What is the best way to test SOA services? Should I write my own tests using WCF or should I be using a testing framework such as SOAPUI. What are the limitations to each method and are there better tools?
You definitely should be using SoapUI. Especially in a mixed environment. i.e. in a mixed environment (java, delphi, WCF, etc..) SoapUI will be your common tool that can confirm what works and what doesn't. It can also be used to set up mock services so you can test against a service that isn't yet built. i.e. from the WSDL you can build something in minutes that will log requests and give responses. That's hugely beneficial. Down the road, you'll be able to verify what works and what doesn't using the common tool, rather than fighting about "works here in technology x, so it must be a problem at YOUR end".
Look into the mockservices demo where they show how to do simple canned responses based on xpath. Very simple, and effective. You can send a response and return a variety of predictable responses. for example, you send updates for emps Tom, Dick, Harry. Configure your SoapUI mockservice to return success for Tom, soft error for Dick, catastrophic error for Harry.
IMO, the best place to start before building any web service is to build a mockservice in SoapUI. Then you can test with sample payloads and see if everybody is seeing what they expect. i.e. HR sends a new employee to Payroll, using the WSDL that everyone agreed to. The Payroll dev hasn't even coded his part yet, but by looking at the transaction in SoapUI, he sees that the EmpID format is "totally not going to work on our end". Now HR can make a change. The Payroll dev also sees that the Termination Dates are 12/31/1889 for employees that haven't been fired yet. He expected ''. Now a discussion can ensue between the devs and analysts, instead of later on during integration or startup, when the discussion would likely involve several layers of PMs, "situation leads", etc..
I suggest you also take a look of the brand new SO-Aware from Tellago Studios; http://www.tellagostudios.com/ . One of the features is automatic service testing.
Soa testing just ensures that all independent services behave in the expected manner, all the while adhering to the input and output contract established by these services. The tool should not just limit itself to webservices testing.
SOA testing tools:
Soap UI
SOArite.
Related
I know this question is a little subjective but I am lost on what to do here. At the moment I am using Go + Go-kit to write some microservices. I'd like to test the endpoints of these microservices in an integration test type fashion but I am unsure how to go about it. The only thing I can think of is to have shell scripts that hit the endpoints and check for responses. But this seems like kludge and not a real smart practice. I feel like there should be a better way to do this. Does anyone have any suggestions?
An alternative approach to end-to-end testing is Consumer-Driven Contract (CDC).
Although is useful to have some end-to-end tests, they have some disadvantages like:
the consumer service must know how to start the provider service. This sounds like unnecessary information, likely difficult to maintain when the number of services start ramping up;
starting up a service can be slow. Even if we’re only talking a few seconds, this is adding overhead to build times. If a consumer depends on multiple services, this all starts adding up;
the provider service might depend on a data store or other services to work as expected. It means that now not only the Provider needs to be started but also a few other services, maybe a database.
The idea of CDC is described shortly as:
The consumer defines what it expects from a specific request to a service
The provider and the consumer agree on this contract
The provider continuously verifies that the contract is fulfilled
This information is taken from here. Read more on this article, it can be useful even if it is specific to Java.
You can do this in a standard Go unit test using the httptest package. This allows you to create mock Request and ResponseWriter objects that can be passed to any Handler or HandleFunc. You create the appropriate Request, pass it to your handler, then read the response out of the ResponseRecorder and check it against the expected response.
If you're using the default mux (calling http.Handle() to register handlers) you can test against http.DefaultServeMux. I've used it for microservices in the past with good results. Works for benchmarking handlers, routing, and middleware as well.
You should always use golang's native unit testing framework to test each individual service (please, no shell script!). httptest seems fine, but I would argue it is helpful to have finer-grained test boundaries -- you should really have one _test.go for each functional block of your code. Smaller tests are easier to maintain.
In terms of overall integration tests that involve multiple microservices, you shouldn't do them at development time. Set up a staging area and run the tests over there.
My 2 cents.
We have a .Net component that provides functionality. We have a Restful Web API service that the world will use to call that functionality. We wrote tests that use OWIN to call into our Web API controllers.
In the past, I had always called these "integration tests", because the service is a separate component. However, another developer, who I respect, told me that this is not an integration test, but is instead a "functional test".
I looked at the Defintion of Functional Testing and the definition of Definition of Integration Test and neither one was a clear winner to me for what a test of a RESTful service test should be called.
Is it integration or functional? Are there any authoritative sources that can be used to definitively answer this question (because I don't want my question to be closed for "soliciting debate")?
Functional testing is a black box exercise, as stated in the Functional testing definition link you posted. This means that the testing occurs without knowledge of the internal workings of the system (such as the code). You would be interested in whether the API works as expected end-to-end; does the user receive the correct response when they use the API? If the REST API is being tested outside of your system then it is a functional test.
If the API is being tested from your code base then this is testing the integration between two or more modules and thus classifies as an integration test. This might involve testing whether a dependent module is sending the correct data, receiving the expected data and in the correct format.
For more information check out anything from ISTQB which is a recognised body for software testing. Here is a link to an ISTQB glossary: http://www.software-tester.ch/PDF-Files/ISTQB%20Glossary%20of%20Testing%20Terms%202.4.pdf
When you call a rest service to verify that the service returns what it was designed to return, that is a functional test. You are testing the functionality of the service.
If you had a second service or UI that depended on this service, and your tests interacted with this second service or UI to verify that it can properly call the REST API and consume it's data, that would be an integration test.
We have an established WCF SOAP service. Its interface is defined in WSDL, from which C# classes are generated for our server (customers generate client-side bindings in various languages, from the same WSDL). The WSDL has a current version, which we can change a bit, and old versions, which we can't change or drop without a deprecation period, consultation etc. The SOAP requests tend to be complicated, having multiple XML namespaces within the same request.
The WCF SOAP service has a lot of "smarts" in it, and provides exactly the kinds of fetching and reporting facilities that we need for a new Web application that we need to make. We hope to use AngularJS for the client side of that. But these complex SOAP requests aren't easy to make in JavaScript world. If only we had a REST service, we could use angular Resource service. If not that, then a server that spoke JSON, albeit in an RPC style like SOAP, would run a fairly close second.
I've had various ideas for how the impedance mismatch between our server and client might be mitigated. But nothing sounds quick or easy.
I've thought of: -
Write a new REST service. Exactly what the client-side wants, but a serious piece of new development.
WebHttpBinding looks to offer something. But seems to me like it requires C# markup of custom attribute (how to achieve when our C# is generated from WSDL) and possibly wouldn't support our complex types
Obtain or write loads of client-side JS to abstract away calling SOAP services. But, unless this can be auto-generated from the WSDL, it's a huge amount of client-side code to write.
Write an IDispatchMessageFormatter for the server, to accept some JSON format of messages that I invent. Sounds hard, especially as good examples of people implementing and integrating IDispatchMessageFormatter seem hard to come by.
Write a MessageEncoder to swap between JSON and XML. But this isn't really an encoding operation, as became very clear when I tried to write it!
I'm searching for suggestions.
Generally, I recommend a REST service for any AngularJS development and have wrapped a number of legacy systems with Node.js API servers. Of course there is a massive amount of "it depends", but generally most projects will be happier and more productive following that route.
Some Things To Think About
How well does your current SOAP API fit the user interface requirements?
Are you experienced with Express, Sinatra, Flask or other micro-framework that allow rapid development of REST APIs? I find I can build a solid Node.js Express API server in a couple of hours and then extend it as I build the AngularJS application out.
How experienced are you with AngularJS? It's a more advanced project to build a complex data layer client-side.
Six Reasons Why REST is Important for AngularJS
It's much faster to write Angular code using $resource and $http. Get the API right is a good recommendation for effective AngularJS development. Indeed, you could argue that AngularJS is designed for REST, and that's why plain JavaScript works for the model (see 2).
Angular's plain-old JavaScript object data model works well with a REST API that speaks JSON that matches the user interface. However, issues arise when there isn't a good fit- Angular doesn't have a formal data model so you end up writing an lot of code trying to rationalize your API to work well with Angular. 3rd party libraries like breeze.js may offer some solution, but it's still awkward.
You can scale easily with caching. It's easy to add Redis or memcache or Varnish or other common HTTP caching solutions into the mix. Resource-based abstractions are perfect for caching strategies due to the transparency and idempotency of a REST api.
Loose coupling of front-end and server- it will be easier to support changes to the backend if you migrate off SOAP or need to integrate with other services.
It's generally easier to test JSON APIs separately from AngularJS logic, so your test suites will be simpler and more effective.
Your new REST API will be easier to leverage for future AngularJS and JSON-oriented projects.
I hope that helps.
Cheers,
Nick
I work on an application that uses a WCF service (which in fact is a service-client solution).
The problem that came up was that when we did a bit of refactoring, it turned out that some of the service contracts became invalid. This was not shown until the service and application were running.
Now, I would like to write test cases that simply tests the service contracts such that when they missmatch the test cases fail. Is this possible to do without having to run the actual service?
That is, can I some how simulate the service part and call the client calls at the same time in the test case?
I would say this is not possible, because there are very much traps which can happen if you execute functions over the net.
E.g. timeouts, connection failures, authorization problems and so on.
Write offline unit tests for the code in the methods itself should be possible, but this is IMHO just one little part of the work.
I'm just speaking my mind here. Would .Net Reflection solve your problem? I mean, inspect the binaries that contain the service(s) contracts that you want to verify.
I came across this question recently, could anyone please help me what should be my approach as a tester.
Suppose, there is a webservice whose functionality have been changed and there is no documentation available of the same. What will be your approach to test the same?
Update: Does the same answer hold if Database functionality changed and no documentation.
It seems you might be asking one of two different questions:
1) How to discover the API of a black-box web service.
In this case, the best source would be the source of the web-service (with the existence failure of the documentation), alternatively look at existing clients, or the ?wsdl of the service.
2) How to discover what are correct and incorrect responses from the web service.
For this you need either requirements, or documentation, or correct clients. Probably the most likely to exist in this case is a client. Alternatively the web-service might be implementing some function the results of which can be confirmed externally.
You can't test something with no documentation. How would you know what results to expect?
Maybe you're looking for "documentation" in the wrong place. Somebody made these changes. They had some information telling them what changes to make to the database and to the service. There may even be a requirements document, but maybe also some design documents.
Get those, and use them to figure out what changed. Use that information to decide how to change your tests.
If you are using the service in a useful way, then presumably you have some calls which return some known results, even though this may not be documented. If this is the case then I would write tests which validate my expectations of the service as it is currently. Then at least if changes are made you'll have more chance of knowing which bits have changed that affect you.
Generally speaking, a web service provides a consistent contract between the providing service and callers. It specifies that whilst the back-end implementation might change, the interface for the service will remain consistent.
If you are interested in discovering what functions are available for the service, it may well provide metadata that documents it's available functions and message types. Usually, this is accessible by appending "?wsdl" to the web service URL, although other schemes exist.
Once you have a good idea of the available functions, you can begin to invoke them through your testing framework and evaluating the responses in accordance with your usual test processes.