We have a .Net component that provides functionality. We have a Restful Web API service that the world will use to call that functionality. We wrote tests that use OWIN to call into our Web API controllers.
In the past, I had always called these "integration tests", because the service is a separate component. However, another developer, who I respect, told me that this is not an integration test, but is instead a "functional test".
I looked at the Defintion of Functional Testing and the definition of Definition of Integration Test and neither one was a clear winner to me for what a test of a RESTful service test should be called.
Is it integration or functional? Are there any authoritative sources that can be used to definitively answer this question (because I don't want my question to be closed for "soliciting debate")?
Functional testing is a black box exercise, as stated in the Functional testing definition link you posted. This means that the testing occurs without knowledge of the internal workings of the system (such as the code). You would be interested in whether the API works as expected end-to-end; does the user receive the correct response when they use the API? If the REST API is being tested outside of your system then it is a functional test.
If the API is being tested from your code base then this is testing the integration between two or more modules and thus classifies as an integration test. This might involve testing whether a dependent module is sending the correct data, receiving the expected data and in the correct format.
For more information check out anything from ISTQB which is a recognised body for software testing. Here is a link to an ISTQB glossary: http://www.software-tester.ch/PDF-Files/ISTQB%20Glossary%20of%20Testing%20Terms%202.4.pdf
When you call a rest service to verify that the service returns what it was designed to return, that is a functional test. You are testing the functionality of the service.
If you had a second service or UI that depended on this service, and your tests interacted with this second service or UI to verify that it can properly call the REST API and consume it's data, that would be an integration test.
Related
I'm a automation engineer and I was wondering if pact could help us in any way to increase test coverage
I'm new to PACT, but I was thinking if it could be possible to generate a pact file with existing selenium web tests?
It is that the contract can be used in different ways. 2 different web apps using the same API, can use it in a different way.
Our web app uses a certain set of API and these also results in http get, post, etc. So in principle a pact file could be generated in that way.
It by-passes the original intention of PACT, where consumer and producer devs work together on contract refinement.
Any thoughts?
This would not be a good fit for the Pact tool. Let me copy/paste a section of the Best Practices information below.
Use Pact for isolated (unit) tests
as a mock (calls to mocks are verified after a test) not a stub (calls to stubs are not verified). Using Pact as a stub defeats the purpose of using Pacts.
for isolated tests (ie. unit tests) of the class(es) that will be responsible for making the HTTP calls from your Consumer application to your Provider application, not for integrated tests of your entire consumer codebase.
carefully, for any sort of functional or integrated tests within your consumer codebase.
Why?
If you use Pact with exact matching for integrated tests, you will drive yourself nuts. You will have very brittle Consumer tests, as Pact checks every outgoing path, JSON node, query param and header. You will also end up with a cartesian explosion of interactions that need to be verified on the Provider side. This will increase the amount of time you spend getting your Provider tests to pass, without usefully increasing the amount of test coverage.
Pact is a contract testing tool, rather than a general purpose stubbing or mocking tool. The best use of Pact involves testing as little of the code that does not concern itself with the HTTP request/response as possible. This will keep your tests fast and maintainable.
I know this question is a little subjective but I am lost on what to do here. At the moment I am using Go + Go-kit to write some microservices. I'd like to test the endpoints of these microservices in an integration test type fashion but I am unsure how to go about it. The only thing I can think of is to have shell scripts that hit the endpoints and check for responses. But this seems like kludge and not a real smart practice. I feel like there should be a better way to do this. Does anyone have any suggestions?
An alternative approach to end-to-end testing is Consumer-Driven Contract (CDC).
Although is useful to have some end-to-end tests, they have some disadvantages like:
the consumer service must know how to start the provider service. This sounds like unnecessary information, likely difficult to maintain when the number of services start ramping up;
starting up a service can be slow. Even if we’re only talking a few seconds, this is adding overhead to build times. If a consumer depends on multiple services, this all starts adding up;
the provider service might depend on a data store or other services to work as expected. It means that now not only the Provider needs to be started but also a few other services, maybe a database.
The idea of CDC is described shortly as:
The consumer defines what it expects from a specific request to a service
The provider and the consumer agree on this contract
The provider continuously verifies that the contract is fulfilled
This information is taken from here. Read more on this article, it can be useful even if it is specific to Java.
You can do this in a standard Go unit test using the httptest package. This allows you to create mock Request and ResponseWriter objects that can be passed to any Handler or HandleFunc. You create the appropriate Request, pass it to your handler, then read the response out of the ResponseRecorder and check it against the expected response.
If you're using the default mux (calling http.Handle() to register handlers) you can test against http.DefaultServeMux. I've used it for microservices in the past with good results. Works for benchmarking handlers, routing, and middleware as well.
You should always use golang's native unit testing framework to test each individual service (please, no shell script!). httptest seems fine, but I would argue it is helpful to have finer-grained test boundaries -- you should really have one _test.go for each functional block of your code. Smaller tests are easier to maintain.
In terms of overall integration tests that involve multiple microservices, you shouldn't do them at development time. Set up a staging area and run the tests over there.
My 2 cents.
Say, I have a feature in my app that relies on an external API - I provide an interface, which makes calls to my server, and the server, relying on that, makes some calls to some external API and responds something to client. If I wanna write an acceptance test with cucumber for that, how can I stub the calls to that external API, so, e.g. any GET call to https://www.cool-api.io/foo would just immediately return some-predefined JSON response with some predefined headers, any POST request to that url would return a response with some predefined status and headers, etc. How do you do it for acceptance tests if you're using cucumber?
We use WireMock or MockServer for this. You can implement them to stub API calls.
Also, I'd recommend using a framework like Jackson to generate json from domain objects. The benefits of this are not having to maintain json Strings/docs in your code base, and compile time checks on whether you created valid domain objects in your test.
I would probably write my own stub that was able to fake an implementation of the response with the expected content and headers set. If the response object is defined with an interface, then have your hand rolled stub to implement that interface.
Using Mockito for this would probably be to cumbersome in my opinion. Mockito is great, but setting up a complex return value like this may be messy. Hard coding the responses in an implementation of a response interface may be easier.
I would check the actual integration towards the external service using other tooling than Cucumber.
When writing unit tests for an API wrapper, should I be making real calls to the REST API endpoints or should I use mocl responses that simulate successful and erroneous calls?
Unit tests mean testing only your unit (API wrapper), nothing else. Thus, unfortunately, you should mock the entire API.
On the other hand this never really gives me enough confidence, so I go for system tests (also known as component tests). In that case you should run your API wrapper against existing API, maybe embedded and started along with your test. In integration test, the ultimate scenario, you run your API wrapper against real, but most likely test instance of the API (sandbox, dev environment).
In well established area of database testing: unit tests mock entire DAO level, component tests run against in-memory database while integration tests connect to real database with some fake data.
What is the best way to test SOA services? Should I write my own tests using WCF or should I be using a testing framework such as SOAPUI. What are the limitations to each method and are there better tools?
You definitely should be using SoapUI. Especially in a mixed environment. i.e. in a mixed environment (java, delphi, WCF, etc..) SoapUI will be your common tool that can confirm what works and what doesn't. It can also be used to set up mock services so you can test against a service that isn't yet built. i.e. from the WSDL you can build something in minutes that will log requests and give responses. That's hugely beneficial. Down the road, you'll be able to verify what works and what doesn't using the common tool, rather than fighting about "works here in technology x, so it must be a problem at YOUR end".
Look into the mockservices demo where they show how to do simple canned responses based on xpath. Very simple, and effective. You can send a response and return a variety of predictable responses. for example, you send updates for emps Tom, Dick, Harry. Configure your SoapUI mockservice to return success for Tom, soft error for Dick, catastrophic error for Harry.
IMO, the best place to start before building any web service is to build a mockservice in SoapUI. Then you can test with sample payloads and see if everybody is seeing what they expect. i.e. HR sends a new employee to Payroll, using the WSDL that everyone agreed to. The Payroll dev hasn't even coded his part yet, but by looking at the transaction in SoapUI, he sees that the EmpID format is "totally not going to work on our end". Now HR can make a change. The Payroll dev also sees that the Termination Dates are 12/31/1889 for employees that haven't been fired yet. He expected ''. Now a discussion can ensue between the devs and analysts, instead of later on during integration or startup, when the discussion would likely involve several layers of PMs, "situation leads", etc..
I suggest you also take a look of the brand new SO-Aware from Tellago Studios; http://www.tellagostudios.com/ . One of the features is automatic service testing.
Soa testing just ensures that all independent services behave in the expected manner, all the while adhering to the input and output contract established by these services. The tool should not just limit itself to webservices testing.
SOA testing tools:
Soap UI
SOArite.