I have a project which is composed of several modules each module holds it's region and repository classes.
The problem with that is that each module has it's own Gemfire gfe:cache on it's own spring context.
So my problem is that when I run my mvn test every module starts it's own Gemfire and closes it after it's tests and this makes my test runt to take almost 10 minutes, every instance of Gemfire takes 40s to start.
So I would like to know what is the best way to avoid that?
I was thinking about having the parent module (holds all repositories and regions) holding and creating the regions and then using lookups on the children modules to use them. But I also need the individual modules to run by themsleves in case I would like to run only one of the modules tests.
Is there a way to use lookup and in case it fails create a cache with that same region lookedup? Or having the cache to be created once (first test) and get the regions added to it while the other contexts are started instead of being closed?
Thanks
In certain cases, you cannot avoid it, especially if you dirty the Spring context or your GemFire instance in such a way that it would lead to conflicts when running subsequent tests in your test suite.
I.e. it maybe more work to try to manage isolation/separation in each new test class and/or test case you write, having to keep track of what test touches what (e.g. Region), rather then perhaps just restarting the GemFire instance per test class (or in the worse case, per test case). E.g., think of a OQL Query pulling in unexpected results due to a previous test's actions, or something of that nature.
The Spring Data GemFire test suite is very similar in that it starts up GemFire instances and stops them, either per test case, or in most cases, per test class. The entire build with tests (900+ tests) runs on avg in ~15 minutes. GemFire's own test suite (unit + integration/distributed tests, etc) runs in about 8-12 hours depending on how effectively you can "parallelize" the tests, o.O
I am a firm believer in the 10 minute build, but GemFire is a complex beast and writing tests, especially distributed tests, effectively takes careful planning.
Mosts of the tests in Spring Data GemFire are peer cache applications (i.e. the test JVM is embeds the GemFire instance, for example... RegionDataPolicyShortcutsIntegrationTest and it's associated Spring context configuration file).
I have found that if you set of a few GemFire properties (such as setting log-level to "warning" and particularly mcast-port to 0) you can reduce the runtime substantially, for example. Setting mcast-port to 0 sets up a "loner" GemFire node and substantially improves startup time.
There are other Spring Data GemFire tests, which are "ClientCaches", that even spawn a separate JVM with a GemFire Server process to test the client/server interaction. You can imagine those taking even longer to start/stop, and in fact they do. For example, the ClientCacheFunctionExecutionWithPdxIntegrationTest and the associated Spring context configuration files... (server) and of course the (client). Note, the test is the GemFire ClientCache VM in this case.
Now, you might be able to get a way with use mocks in certain tests. Many of the Spring Data GemFire tests mock the interactions between SDG and GemFire using the mocks provided in the org.springframework.data.gemfire.test package.
The test classes in SDG that use mocks declare a special kind of Spring ApplicationContextInitializer like so in the GemfireTemplateTest, which uses the SDG GemfireTestApplicationContextInitializer. The real magic is in the GemfireTestBeanPostProcessor though. If you trace the code through, you will get the idea.
In time, I hope to formalize these mocks a bit and create a separate project with these mocks, the GemfireTestApplicatonContextInitializer and associated GemfireTestBeanPostProcessor for developers testing purposes involving Spring and GemFire (and also Apache Geode).
Hope this gives you a few ideas to ease the pain of the test setup and runtime.
Cheers!
Related
I have a integration test that should test the creation of a new account in a CRM software.
The account creation triggers several things:
Creates the basic profile of the company
Creates every user (you can define the number of users on the registration)
Initialize the basic configuration of the account
Sends a welcome email with the starting information
etc
The test checks every aspect with several asserts, but I don't know if this is correct or if I should do a separate test for every one.
If I go for separate tests, the setup would be the same for all, so I feel like it would be a waste of time.
What you explain there sounds more like an end-to-end test. It's ok to have some end-to-end tests, but they are usually very expensive to write, to maintain, and brittle.
For me, the tests in a service should give you confidence that the software you are delivering will work in production. So maybe it's ok to have a very small number of end-to-end tests that check that everything is glued together properly, but most of the actual functionality should be in normal tests. An example of what I would try to avoid is to is have an end-to-end test that checks what happens when a downstream service is down.
Another very important aspect is that tests are written for other developers, they are not written for the compiler, so keeping a tests simple is important for maintainability. I want to stress this because if a test has 10 lines of assertions, that will be unreadable for most developers. even a test of 10 lines of code is difficult to grok.
Here's how I try to build services:
If you are familiar with ATDD and hexagonal architecture, most of the features should be tested stubbing the adaptors, which allows the tests to run super fast and fiddle with the adapters using test doubles. These tests shouldnt interact with anything outside the JVM, and give one a good level of confidence that the features will work. If the feature has too many side effects, I try to pick the assertions carefully. For example if a feature is to create an account, I won't check that the account is actually on the DB (because the chances of that breaking are minuscle), but I would check that all messages that need to be triggered are sent. Sometimes I do create multiple tests if the test starts to become unclear. For example one tests that checks the returned value and another tests that verifies the side effects (e.g. messages being produced).
Having as minimum a good coverage of the critical code with unit tests and integration tests (here I mean test classes that interact with external services) builds up the confidence that the classes work as expected. So end-to-end tests don't need to cover the miriad of combinations.
And last a very small number of end-to-end tests to ensure everything is glued together nicely.
Bottom line: create multiple test with the same setup if it helps understanding the code.
edit
About integration tests: It's just terminology. I call integration test a class or small group of classes that interact with an external service (database, queue, files, etc); A component test is something that verifies a single service or module; and end-to-end test something that tests all the services or modules working together.
What you mentioned about stored procs changes the approach. Do you have unit tests for them? Otherwise you could write some sort of integration tests that verify the stored procs work as expected.
About readability of the test: for me, the real test is to ask someone from another team or a product owner and ask them if the test name, the setup, what is asserted and the relatioship between those things is clear. If they struggle, it means that the test should be simplified.
I am exploring individual microservice testing & came across a good guide https://martinfowler.com/articles/microservice-testing/#testing-component-in-process-diagram. This post suggest there is a way to test the individual microservices "in-process".
Which means the test & service will execute in same process, it also provides the lib which can be used. However I am not able to understand following things
1) How exactly "in-process" testing will work in microservice architecture
2) Any pointers on how to use "inproctester" lib.
Thanks
This can be accomplished by hosting your API in process, interacting with it - in process - whilst using test doubles at the edges of you API, or (put another way) using test doubles for external collaborators.
This allows an API (and its core behaviour) to be tested in a test (unit/integration depending on opinion) without ever hitting a network or database or filesystem.
This testing will provide coverage but obviously not all the coverage you need
In .NET core you can use TestServer included in the Microsoft.AspNetCore.TestHost package.
If your API is simple enough you can perhaps reduce your types of testing (N.B. I am not suggesting reducing coverage)
This approach is alluded to in the slide deck you mention on page 8/25
Paraphrasing. If the logic in the API is small, component tests may be favourable to isolated and sociable unit tests
This approach is also recommended in the Spotify "Honeycomb" style testing. They call component testing of an API Integration testing (which coincidentally Microsoft also call it when they are using TestServer in their tests (second link for concrete examples)
https://labs.spotify.com/2018/01/11/testing-of-microservices/
https://learn.microsoft.com/en-us/aspnet/core/testing/integration-testing
I think that the thrust of the approach (regardless of its specific name) is that in process testing is fast and external collaborators are replaed with test doubles.
One thing worth noting is that "out of process" component tests are basically end to end tests but with all external collaborators swapped out with test doubles.
These may be a bit slower than in process component tests but exercise a bit more of the "stack". I'd suggest that if you are going to end-end test anyway, I'd try and do in process component testing as much as possible.
We have Eclipse UI in the frontend and have a non Java based backend.
We generally write Unit tests separately for both frontend and backend.
Also we write PDE tests which runs Eclipse UI against a dummy backend.
My question is do we need to have integration tests which test end to end.
One reason i might see these integration tests are useful are when i upgrade my frontend /backend i can run end to end tests and i find defects.
I know these kind of questions are dependent on particular scenario.
But would like to what is the general and best practice followed by all here.
cheers,
Saurav
As you say, the best approach is dependant on the application. However, in general it is a good idea to have a suite of integration tests that can test your application end-to-end, to pick up any issues that may occur when you upgrade only one layer of the application without taking those changes into account in another layer. This sounds like it would be definitely worthwhile in your case, given that you have system components written in different languages, which naturally creates more chance of issues arising due added complexity around the component interfaces.
One thing to be aware of when writing end-to-end integration tests (which some would call system tests) is that they tend to be quite fragile when compared to unit tests, which is a combination of a number of factors, including:
They require multiple components to be available for the tests, and for the communication between these components to be configured correctly.
They exercise more code than a unit test, and therefore there are more things that can go wrong that can cause them to fail.
They often involve asynchronous communication, which is more difficult to write tests for than synchronous communication.
They often require complex backend data setup before you can drive tests through the entire application.
Because of this fragility, I would advise trying to write as few tests as possible that go through the whole stack - the focus should be on covering as much functionality as possible in the fewest tests possible, with a bias towards your most important functional use-cases. A good strategy to get started would be:
Pick one key use-case (which ideally touches as many components in the application as possible), and work on getting an end-to-end test for this (even just having this single test will bring a lot of value). Focus on making this test as realistic as possible (i.e. use a production-like deployment), as reliable as possible, and as automated as possible (ideally it should run as part of continuous integration). Even just having this single test brings a lot of value.
Build out tests for other use-cases one test at a time, again focusing on your most important use-cases at first.
This approach will help to ensure that your end-to-end tests are of high quality, which is vital for their long-term health and usefulness. Too many times I have seen people try to introduce a comprehensive suite of such tests to an application, but ultimately fail because the tests are fragile & unreliable, people lose faith in them, don't run or maintain them, and eventually they forget they even had the tests in the first place.
Good luck and have fun!
What's the best practice fort creating webdriver instances in Selenium-webdriver? Once per test method, per test class or per test run?
They seem to be rather (very!) expensive to spin up, but keeping it open between tests risks leaking information between test methods.
Or is there an alternative - is a single webdriver instance a single browser window (excluding popups), or is there a method for starting a new window/session from a given driver instance?
Thanks
Matt
I've found that reusing browser instances between test methods has been a huge time saver when using real browsers, e.g. Firefox. When running tests with HtmlUnitDriver, there is very little benefit.
Regarding the danger of indeterministic tests, it's a trade-off between totally deterministic tests and your time. Integration tests often involve trade-offs like these. If you want totally deterministic integration tests you should also be worrying about clearing the database/server state in between test runs.
One thing that you definitely should do if you are going to reuse browser instances is to clear/store the cookies between runs.
driver.manage().deleteAllCookies();
I do that in a tearDown() method. Also if your application stores any data on the client side you'd need to clear that (maybe via JavascriptExecutor). To your application which is under test, it should look like a completely unrelated request after doing this, which really minimizes the risk of indeterministic behaviour.
If your goal of automated integration testing is to have reproducible tests, then I would recommend a new webdriver instance for every test execution.
Each test should stand alone, independent from any other test or side-effects.
Personally the only thing I find more frustrating than a hard to reproduce bug, is a non-deterministic test that you don't trust.
(This becomes even more crucial for managing the test data itself, particularly when you look at tests which can modify persistent application state, like CRUD operations.)
Yes, additional test execution time is costly, but it is better then spending the time debugging your tests.
Some possible solutions to help offset this penalty is to roll your testing directly into your build process, going beyond Continuous Build to Continuous Integration approach.
Also try to limit the scope of your integration tests. If you have a lot of heavy integration tests, eating up execution time, try to refactor. Instead, increase the coverage of your more lightweight unit tests of the underlying service calls (where your business logic is).
I am starting out with automated testing and I would like to test one of my data access methods. I am trying to test what the code does if the database returns no records.
Is this something that should be done in a unit test or an integration test?
Thanks
If your test code connects to an actual database and relies on the presence of certain data (or lack of data) in order for the test to pass, it's an integration test.
I ususally prefer to test something like this by mocking out the component that the "data access method" used to get the actual data, whether that's a JDBC connection or web service proxy or whatever else. With a mock, you say "when this method is called, return this" or "make sure that this method is called N times", and then you tell the class under test to use the mock component rather than the real component. This then is a "unit test", because you are testing how the class under test behaves, in a closed system where you've declared exactly how the other components will behave. You've isolated the class under test completely and can be sure that your test results won't be volatile and dependent on the state of another component.
Not sure what language/technology you are working with, but in the Java world, you can use JMock, EasyMock, etc for this purpose.
I think more time has been wasted arguing about what is a unit vs. what is an integration test than value has been added.
I don't care.
Let me put it a different way: If I were testing it, I'd see two ways to do it - fake out the database returning zero rows, or actually connect to a database that has no data for the select. I'd probably start testing with whatever was easiest to do and simplest to implement - if it ran fast enough for me to get meaningful feedback. Then I'd consider the other if I needed it to run faster or thought there would be some advantage.
For example, I'd probably start connecting to the actual test DB at my work. But if the software needed to work with many different databases - Oracle, PostGres, MySQL, SQL server and DB, or if the test DB at work was down for 'refreshes' a lot, I'd probably write the 'pure/unit' test that existed totally in isolation.
In my old age, I prefer to use the term 'developer-facing' vs. 'customer facing' more often, and do the kind of testing that makes more sense. I find using terms like "unit" extensively, then getting a definition-weenie about it leads to people doing things like mocking out the filesystem or mocking getters and setters - activity that I find unhelpful.
I believe this strongly; I've presented before google on it.
http://www.google.com/url?sa=t&source=web&oi=video_result&ct=res&cd=1&url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPHtEkkKXSiY&ei=9-wKSobjEpKANvHT_MEB&rct=j&q=heusser+GTAC+2007&usg=AFQjCNHOgFzsoVss50Qku1p011J4-UjhgQ
good luck! Let us know how it goes!
Do your test and let other people spend time with taxonomy.
My perspective is that you should categorize the test based on scope:
A unit test can be run standalone
without any external dependencies
(File IO, Network IO, Database,
External Web Services).
An integration test can touch external systems.
If the test requires a real database to run then call it an integration test and keep it separate from the unit tests. This is important because if you mix integration and unit tests than you make your code less maintainable.
A mixed bag of tests mean that new developers may need a whole heap of external dependencies in order to run the test suite. Imagine that you want to make a change to a piece of code that is related to the database but doesn't actually require the database to function, you're going to be frustrated if you need a database just to run the tests associated with the project.
If the external dependency is difficult to mock out (for example, in DotNet, if you are using Rhino Mocks and the external classes don't have interfaces) then create a thin wrapper class that touches the external system. Then mock out that wrapper in the unit tests. You shouldn't need a database to run this simple test so don't require one!
There are those (myself included) who have strict rules about what constitutes a unit test vs an integration test.
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can’t run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config files) to run it
Which may be one way to make a distinction between what a unit test will be doing for you using mocking for example, rather than any of the real resource providers - filesystem, db etc.
An integration test can be viewed as a test of very coupling of systems/application layers, so the fundamentals are tested in the unit and the system interoperability is the focus of an integration test.
Its still a grey area though because one can often pinpoint certain exceptions to these sorts of rules.
I think the important question is "What SHOULD I be doing?"
In this case I think you should be unit testing. Mock the code that talks to the DB and have it return a reliable result (no rows), this way your test checks what happens when there are no rows, and not what happens when the DB returns whatever is in the DB at the point you test.
Definitely unit test it!
[TestMethod]
public void ForgotMyPassword_SendsAnEmail_WhenValidUserIsPassed()
{
var userRepository = MockRepository.GenerateStub<IUserRepository>();
var notificationSender = MockRepository.GenerateStub<INotificationSender>();
userRepository.Stub(x => x.GetUserByEmailAddressAndPassword("me#home.com", "secret")).Return(new User { Id = 5, Name = "Peter Morris" });
new LoginController(userRepository, notificationSender).ResendPassword("me#home.com", "secret");
notificationSender.AssertWasCalled(x => x.Send(null),
options => options.Constraints(Text.StartsWith("Changed")));
}
I believe that it is possible to test that as a unit test, without a real database. Instead of using a real interface to the database, replace it with a mock/stub/fake object (better visualized PDF is here).
If writing it as a unit test proves to be too hard, and you are not able to refactor the code that testing it would be easy, then you better write it as an integration test. It will run slower, so you might not be able to run all the integration tests after ever code change (unlike unit tests which you can run hundreds and thousands per second), but as long as they are run regularly (for example as part of continous integration), they produce some value.
Most likely a unit test ... but there is a blurred line here. It really depends upon how much code is being executed - if it is contained to a library or class then its unit test, if it spans multiple components then it's more of an integration test.
I believe that should be done in a unit test. You aren't testing that it can connect to the database, or that you can call your stored procedures... you are testing the behavior of your code.
I could be wrong, but that's what I think unless someone gives me a reason to think otherwise.
that is a unit test, by definition: you are testing a single isolated element of the code on a specific path