I am exploring individual microservice testing & came across a good guide https://martinfowler.com/articles/microservice-testing/#testing-component-in-process-diagram. This post suggest there is a way to test the individual microservices "in-process".
Which means the test & service will execute in same process, it also provides the lib which can be used. However I am not able to understand following things
1) How exactly "in-process" testing will work in microservice architecture
2) Any pointers on how to use "inproctester" lib.
Thanks
This can be accomplished by hosting your API in process, interacting with it - in process - whilst using test doubles at the edges of you API, or (put another way) using test doubles for external collaborators.
This allows an API (and its core behaviour) to be tested in a test (unit/integration depending on opinion) without ever hitting a network or database or filesystem.
This testing will provide coverage but obviously not all the coverage you need
In .NET core you can use TestServer included in the Microsoft.AspNetCore.TestHost package.
If your API is simple enough you can perhaps reduce your types of testing (N.B. I am not suggesting reducing coverage)
This approach is alluded to in the slide deck you mention on page 8/25
Paraphrasing. If the logic in the API is small, component tests may be favourable to isolated and sociable unit tests
This approach is also recommended in the Spotify "Honeycomb" style testing. They call component testing of an API Integration testing (which coincidentally Microsoft also call it when they are using TestServer in their tests (second link for concrete examples)
https://labs.spotify.com/2018/01/11/testing-of-microservices/
https://learn.microsoft.com/en-us/aspnet/core/testing/integration-testing
I think that the thrust of the approach (regardless of its specific name) is that in process testing is fast and external collaborators are replaed with test doubles.
One thing worth noting is that "out of process" component tests are basically end to end tests but with all external collaborators swapped out with test doubles.
These may be a bit slower than in process component tests but exercise a bit more of the "stack". I'd suggest that if you are going to end-end test anyway, I'd try and do in process component testing as much as possible.
Related
Wanting to open a discussion about testing approaches.
Context
I'm creating a new project and my main focus has been on efficiency and clean structure (not necessarily the most STANDARDISED code, but easiest to read, consistent, and quick to understand). Building my server-side code, and have mapped it from a loose outline of website UX designs. I want to create some integration tests in preparation for building FE.
I experimented with using newman/postman to automate some of these integration tests. Pros for this:
Not reinventing the wheel, by that - the postman requests would exist anyway, it's not a whole new suite I'm having to build and maintain
Consistency with project state, during the manual testing phase these postman requests would be updated so by the time integration tests happens, it's up to date automatically with the project state.
But had some issues with running. Then had the idea...
Why not build out the FE API library which connects to my server, and then use that as my testing suite?
Easier to enforce contract testing as it can be coupled with unit testing on a real consumer
Not a whole new suite I'm having to build and maintain, makes de-coupling near impossible and therefore reliability of testing increases
Efficient use of time as this will be built anyway, the only additional coding would be scripts that schedule test calls
Considering integrating the two repos (FE + BE) into a monorepo, using a management tool like NX (which can check for differences and only do builds / deploys when affected areas are touched). That way types can be consistent across touch points as they're both running of TypeScript.
Ideal conversation anchors:
Experimented with something similar? How did it go?
Considerations I haven't made which are likely to cause problems down the line?
Anything easier to achieve the goal that I haven't considered?
For instance, I assume this is what SDETs do?
They don't actually write the functional code but they're able to write integration/unit tests, am i correct?
But can someone learn to read code and then start writing tests?
This is actually a good question. I have been at the same place when I was working only on Manual Testing. Here is how I experienced things when I transitioned to automation. To answer your question, yes someone can read code and start writing tests on it but you need to understand the code that you are going to test.
There are different types of testing methodologies that are used when testing an application. These tests are done in layers so that the application is properly tested. Here is how the layering looks like:
1) Unit testing: This part is usually written by developers. This is because they have written the code and know the functionalities of the code and is easier for them to write. I am an SDET and I have written unit tests. There was only one opportunity that presented itself and it was when we were refactoring our code and there was a lot of room to write Unit Tests. In unit tests, you test functions in isolation by giving it some values and verifying an expected value. This is not something an SDET does, but should be able to do if provided the chance.
2) Integration testing: This part is also usually written by developers but the definition of integration testing is a bit vague. It means testing multiple modules in isolation. This can be modules in backend or modules in frontend but not together. Frameworks that help achieve this are code level integration tests for the technology you are using. Like for Angular application, there are deep integration tests that test the HTML and CSS of a component and there are shallow integration tests that just test two component's logic together. This can be written by an SDET but is usually written by the developer.
3) API testing (contract based testing): Pact helps us achieve this. There are other tools like rest assured, postman and jmeter that help in testing API end points. Pact helps test the integration of APIs on the frontend and verifies that integration in backend. This is very popular with microservices. This is something that can be written either by the developer or by the SDET.
4) End to End testing: This is something that is the sole responsibility of SDET. This covers testing of user flows depending on user stories. It is testing the entire stack together. Backend and frontend. This allows SDETs to automate how a user would use the application. This is also called as blackbox testing. There are different frameworks that help achieve this. Selenium, Protractor, Cypress, Detox etc. This is the sole responsibility of an SDET.
5) Load testing: This is again something that an SDET does. Using tools like hey, jmeter, loadrunner etc. These tests allow the SDET to initiate a heavy load on the system and check for breaking points of the system.
6) Performance testing: Testing the performance of the webpage for an end user depending on the page load time, the SEO optimisation and the weight of elements of page. This is something achieved by google's lighthouse tool that is an amazing tool to use. I am not aware of anything else that is as amazing as lighthouse because it gives us a lot of data that we can use to improve our website. This is a primary job of an SDET.
7) CI/CD: Continuous Integration and Continuous Deployments is something that requires architectural knowledge of the system. This is something you can do when you are an SDET3 or a lead QA engineer. For systems like AWS and GCP, using CI build tools like Jenkins and CircleCI, one can set up a pipeline that runs all the above tests when ever a branch is merged into master or whenever a pull request is created. Creating the pipeline will require you to have knowledge of Docker, Kubernetes, Jenkins and your test frameworks. First you dockerize your tests, then you build the image and push it to a directory in cloud, then you use the image to create a kubernetes job that runs everytime a change is presented in your code.
These are the levels of work that an SDET does. It takes time and hard work to have an understanding of all testing frameworks and how everything fits together. An SDET should have knowledge of the server, http protocols, frontend, backend, browsers, caching, pipeline management and orchestration of tests.
Yes, sure. You can write unit tests increasing test coverage of the codebase. That's very qualified work from software test engineering since you need to be aware what is going on in the code. This skill is definitely great!
I advise you to take a look on so called "mutation coverage". Usage of mutation coverage as a better metric than simple unit test coverage. Mutation tests are changing logical operators in the different parts of the codebase (making so called "mutants") and then are running unit tests to find out how many unit tests will fail showing their effectiveness (if after mutants were injected the result is the same as without them - unit tests quality is low and they won't catch any new injected issues to the codebase).
Is it overkill to try TDD and BDD in applications in which we consume web services, collect data based on some conditionals, and then show that data on a web page? I am trying to convince my team to use TDD and BDD, but they don't seem to understand the necessity for this.
Especially for communication across system borders I find automated tests extremely helpful.
Setting up the other system in a way so that it shows the behavior which is required for the test is often rather tedious. With automated tests you'd mock those webservices. With this approach development becomes much faster and easier (after overcomming the learning curve for TDD).
Throw in some tests that test the actual webservices to behave as expected and you get early notification if a change in the webservices is going to break your system.
We have Eclipse UI in the frontend and have a non Java based backend.
We generally write Unit tests separately for both frontend and backend.
Also we write PDE tests which runs Eclipse UI against a dummy backend.
My question is do we need to have integration tests which test end to end.
One reason i might see these integration tests are useful are when i upgrade my frontend /backend i can run end to end tests and i find defects.
I know these kind of questions are dependent on particular scenario.
But would like to what is the general and best practice followed by all here.
cheers,
Saurav
As you say, the best approach is dependant on the application. However, in general it is a good idea to have a suite of integration tests that can test your application end-to-end, to pick up any issues that may occur when you upgrade only one layer of the application without taking those changes into account in another layer. This sounds like it would be definitely worthwhile in your case, given that you have system components written in different languages, which naturally creates more chance of issues arising due added complexity around the component interfaces.
One thing to be aware of when writing end-to-end integration tests (which some would call system tests) is that they tend to be quite fragile when compared to unit tests, which is a combination of a number of factors, including:
They require multiple components to be available for the tests, and for the communication between these components to be configured correctly.
They exercise more code than a unit test, and therefore there are more things that can go wrong that can cause them to fail.
They often involve asynchronous communication, which is more difficult to write tests for than synchronous communication.
They often require complex backend data setup before you can drive tests through the entire application.
Because of this fragility, I would advise trying to write as few tests as possible that go through the whole stack - the focus should be on covering as much functionality as possible in the fewest tests possible, with a bias towards your most important functional use-cases. A good strategy to get started would be:
Pick one key use-case (which ideally touches as many components in the application as possible), and work on getting an end-to-end test for this (even just having this single test will bring a lot of value). Focus on making this test as realistic as possible (i.e. use a production-like deployment), as reliable as possible, and as automated as possible (ideally it should run as part of continuous integration). Even just having this single test brings a lot of value.
Build out tests for other use-cases one test at a time, again focusing on your most important use-cases at first.
This approach will help to ensure that your end-to-end tests are of high quality, which is vital for their long-term health and usefulness. Too many times I have seen people try to introduce a comprehensive suite of such tests to an application, but ultimately fail because the tests are fragile & unreliable, people lose faith in them, don't run or maintain them, and eventually they forget they even had the tests in the first place.
Good luck and have fun!
I'm using a twitter gem which basically accesses twitter and lets me grab tweets, timeline etc. Its really good but I have a lot of my code that uses the stuff it returns and I need to test it. The things the gem returns aren't exactly simple strings, there pretty complex objects (scary as well) so im left scratching my head.
So basically I'm looking for an answer, book, blog, open-source project that can show me the rights and wrongs of testing around external services.
answers that are either not language centric or ruby/rails centric would most greatly be appreciated.
What you are really talking about are two different kinds of testing that you would want to accomplish - unit tests and integration tests.
Unit tests will test the validity of the methods, independently of any external data. You should look into some sort of mocking framework, based on whatever language it is that you are using. You are basically looking to say, with the tests, something equivalent to "if these assumptions are qualified, then this test should yield..." The making framework will define your assumptions, in terms of saying that certain classes/objects are set in a particular way and can be assumed to be valid. These are the tests that will not rely on Twitter being alive, or the third part library/API being responsive.
Integration tests will perform tests live against the data source, consuming the library/API to perform actual actions. Where it gets tricky, since you are using a third party service, is in writing out to the service (i.e. if you are creating new Tweets). If you are, you could always create an account on Twitter that could be used just for write operations. Generally, if you were testing against a local database - for example - you could then, instead, use transactions to test similar operations; rolling back the transactions instead of committing them.
Here are a couple of non-language specific, high-level definitions:
Wikipedia (Software Testing)
Wikipedia (Mock Object)
I am from a .NET stack, so I won't pretend to know much about Ruby. A quick Google search, though, did reveal the following:
Mocha (Ruby Mocking Framework)
You can easily stub at the http layer using something like wiremock http://wiremock.org/ I've used this on a few projects now and it's quite powerful and fast. This will eliminate all the set up code of code based mocking - just fire up the jar with related mappings and bob's your uncle.