I'm designing a web service running on Google App Engine that scrapes a number of websites and presents their data via a RESTful interface. Based on some background reading, I think I'd like to attempt Test Driven Development (TDD) and develop my tests before I write any business code.
My problem is caused by the fact that my list of scraped elements includes timetables and other records that change quite frequently. The limit of my knowledge on TDD is that you write tests that examine the results of code execution and compare these results to a hardcoded result set. Seeing as the data set changes frequently, this method seems impossible. Assuming that this is true, what would be the best approach to test such an API? How would a large-scale web API be tested (Twitter, Google, Netflix etc.)?
You have to choose the type of test:
Unit tests just test proper operation of your modules (units). You provide input data and test that code outputs proper results. If there are system dependent classes you try to mock them or in case of GAE services, you use google provided local services. Unit tests can be run locally on your machine or on CI servers. There are two popular unit test libs for java: Junit & TestNG.
Integration tests check that various modules (internal & external) work together - they basically check that APIs between modules are working. They are usually run on real servers and call real external services. They are technology specific and are harder to run.
In your case, I'd go with unit tests and provide sets of different input data which you logic should parse and act upon. Since your flow is pretty simple (load data from fixed Url, parse it) you could also embed loading of real data into unit tests (we do this when we parse external sources).
From what you are describing you could easily find yourself writing integration tests. If your aim is to test the logic for processing what is returned from the scraped data (e.g. you know that you are going to get a timetable in a specific format coming in and you now have logic to process that data) you will need to create a SEAM between your web services logic and your processing logic. Once you have done this you should be able to mock the data that is returned from the web service call to always return the same table data and then you can write consistent unit tests against it.
public class ScrapingService : IScrapingService
{
public string Scrape(string url)
{// scraping logic}
}
public interface IScrapingService
{
string Scrape(string url);
}
public class ScrapingProcessor
{
private IScrapingService _scrapingService
// inject the dependency
pubilc ScrapingProcessor(IScrapingService scrapingService)
{
_scrapingService = scrapingService;
}
public void Process(string url)
{
var scrapedData = _scrapingService.Scrape(url)
// now process the scrapedData
}
}
To test you can now create a FakeScrapingService that implements the IScrapingService interface and then return whatever data you like from the Scrape method. There are some very good Mocking frameworks out there that make this type of thing easy. My personal favorite is NSubstitue.
I hope this explanation helps.
Related
What is the difference between functional test and end-to-end test?
Techopedia says that end-to-end test is
a methodology used to test whether the flow of
an application is performing as designed from start to finish. The
purpose of carrying out end-to-end tests is to identify system
dependencies and to ensure that the right information is passed
between various system components and systems.
Techopedia also says the following about functional test:
Functional testing is a software testing process used within software
development in which software is tested to ensure that it conforms
with all requirements. Functional testing is a way of checking
software to ensure that it has all the required functionality that's
specified within its functional requirements.
After reading the above two paragraphs, I'm still confused about the difference between them.
I have a node.js application which accepts requests, then parses the request, then sends the parsed data to a Database.
requests parse requests and send data to the database
Client ---------> node.js app --------------------------------------------> Database
How can I write end-to-end test and functional test for the node.js app I mentioned?
I think in both types of the tests, I should treat the node.js app as a black box. And send requests to it. Then check if the output of the black box is correct or not.
It seems that in my case, there's no difference between functional test and end-to-end test.
As I understand it, the biggest difference between the two is that an end-to-end test requires the test to setup the system components as they are in production. Real database, services, queues, etc. The reason for this is to see that your system is wired correctly (database connections, configuration and such).
A functional test can setup the system with in-memory implementations of your application ports, which would make the test run faster and perhaps allow tests to run in parallel (in some cases). The only thing the test cares about is that a feature works as expected. This can reduce the overhead of setting up certain tests, since preparing 3rd party systems with data can be difficult or time consuming.
I think the definitions of functional and end to end testing could vary based on the context of your project. I have seen different people use these terms to describe different things. That being said, usually this is what the 2 terms mean-
Functional testing - This refers to testing the functionality of system based on the requirements. This usually focuses on different requirements of the system and ensure it is working properly. For example - Logging into an application - could be one requirement and then a person could test this functionality manually or in an automated way. Similarly, adding a product to the cart could be one functionality, then, able to make a payment to purchase a product could be a functionality.
End to end testing - This refers to testing the system based on end to end user flows, instead of testing the system has separate components like in unit testing or story level testing. For example - Logging into the application, then adding a product to the shopping cart, then going to the check out screen and then placing an order and then logging out of the application could be one user flow.
What we follow is slightly different and of course difference in just how your team treats each of them. for further clarity,
Functional Test : Tests a feature say login, verify from database if login data is correct, verify if intended event received, or send to a message bus or any external activity in a Prod like environment like staging environment. You test a particular functionality in a real environment.
End to End testing : Test complete feature like, login to app, view product on view page, select product, checkout and do payment. This could cover multiple microservices as well, or maybe multiple teams. If this flow breaks, we can pin point which of the functional tests failed.
Integration Test: Test integration between multiple components, from a wide spectrum of multiple classes to multiple system. Like can UI connect to some external login service, can backend connect to database. If a functional test breaks, we can watch which Int Test failed and so on with unit test.
I've been seraching for a while how should I test my data access layer with not a lot of success. Let me list my concerns:
Unit tests
This guy (here: Using InMemoryConnection to test ElasticSearch) says that:
Although asserting the serialized form of a request matches your
expectations in the SUT may be sufficient.
Does it really worth to assert the serialized form of requests? Do these kind of tests have any value? It doesn't seem likely to change a function that should not change the serialized request.
If it does worth it, what is the correct way to assert these reqests?
Unit tests once again
Another guy (here: ElasticSearch 2.0 Nest Unit Testing with MOQ) shows a good looking example:
void Main()
{
var people = new List<Person>
{
new Person { Id = 1 },
new Person { Id = 2 },
};
var mockSearchResponse = new Mock<ISearchResponse<Person>>();
mockSearchResponse.Setup(x => x.Documents).Returns(people);
var mockElasticClient = new Mock<IElasticClient>();
mockElasticClient.Setup(x => x
.Search(It.IsAny<Func<SearchDescriptor<Person>, ISearchRequest>>()))
.Returns(mockSearchResponse.Object);
var result = mockElasticClient.Object.Search<Person>(s => s);
Assert.AreEqual(2, result.Documents.Count()).Dump();
}
public class Person
{
public int Id { get; set;}
}
Probably I'm missing something but I can't see the point of this code snippet. First he mocks an ISearchResponse to always return the people list. then he mocks an IElasticClient to return this previous search response to any search request he makes.
Well it doesn't really surprise me the assertion is true after that. What is the point of these kind of tests?
Integration tests
Integration tests do make more sense to me to test a data access layer. So after a little search i found this (https://www.nuget.org/packages/elasticsearch-inside/) package. If I'm not mistaken this is only about an embedded JVM and an ES. Is it a good practice to use it? Shouldn't I use my already running instance?
If anyone has good experience with testing that I didn't include I would happily hear those as well.
Each of the approaches that you have listed may be a reasonable approach to take, depending on exactly what it is you are trying to achieve with your tests. you haven't specified this in your question :)
Let's go over the options that you have listed
Asserting the serialized form of the request to Elasticsearch may be a sufficient approach if you build a request to Elasticsearch based on a varying number of inputs. You may have tests that provide different input instances and assert the form of the query that will be sent to Elasticsearch for each. These kinds of tests are going to be fast to execute but make the assumption that the query that is generated and you are asserting the form of is going to return the results that you expect.
This is another form of unit test that stubs out the interaction with the Elasticsearch client. The system under test (SUT) in this example is not the client but another component that internally uses the client, so the interaction with the client is controlled through the stub object to return an expected response. The example is contrived in that in a real test, you wouldn't assert on the results of the client call as you point out but rather on the output of the SUT.
Integration/Behavioural tests against a known data set within an Elasticsearch cluster may provide the most value and go beyond points 1 and 2 as they will not only incidentally test the generated queries sent to Elasticsearch for a given input, but will also be testing the interaction and producing an expected result. No doubt however that these types of test are harder to setup than 1 and 2, but the investment in setup may be outweighed by their benefit for your project.
So, you need to ask yourself what kinds of tests are sufficient to achieve the level of assurance that you require to assert that your system is doing what you expect it to do; it may be a combination of all three different approaches for different elements of the system.
You may want to check out how the .NET client itself is tested; there are components within the Tests project that spin up an Elasticsearch cluster with different plugins installed, seed it with known generated data and make assertions on the results. The source is open and licensed under Apache 2.0 license, so feel free to use elements within your project :)
I have an API that I've written and now I'm in the middle of writing an SDK for 3rd parties to more easily interact with my API.
When writing tests for my SDK, it's my understanding that it's best not to simply call all of the API endpoints because:
The tests in the API will be responsible for making sure that the API works.
If the SDK tests did directly call the API, my tests would be really slow.
As an example, let's say my API has this endpoint:
/account
In my API test suite I actually call this endpoint to verify that it returns the proper data.
What approach do I take to test this in my SDK? Should I be mocking a request to /account? What else would I need to do to give my SDK good coverage?
I've looked to other SDKs to see how they're handling this (Stripe, Algolia, AWS), but in some cases it does look like they're calling a sandbox version of the actual API.
(I'm currently working with PHPUnit, but I'll be writing SDKs in other languages as well.)
I ended up taking this approach:
I have both unit tests AND integration tests.
My integration tests call the actual API. I usually run this much less frequently — like before I push code to a remote. (Anyone who consumes my code will have to supply their own API credentials)
My unit tests — which I run very frequently — just make sure that the responses from my code are what I expect them to look like. I trust the 3rd party API is going to give me good data (and I still have the integration tests to back that up).
I've accomplished this by mocking Guzzle, using Reflection to replace the client instance in my SDK code, and then using Mock Handlers to mock the actual response I expect.
Here's an example:
/** #test */
public function it_retrieves_an_account()
{
$account = $this->mockClient()->retrieve();
$this->assertEquals(json_decode('{"id": "9876543210"}'), $account);
}
protected function mockClient()
{
$stream = Psr7\stream_for('{"id": "9876543210"}');
$mock = new MockHandler([new Response(
200,
['Content-Type' => 'application/json'],
Psr7\stream_for($stream)
)]);
$handler = HandlerStack::create($mock);
$mockClient = new Client(['handler' => $handler]);
$account = new SparklyAppsAccount(new SparklyApps('0123456789'));
$reflection = new \ReflectionClass($account);
$reflection_property = $reflection->getProperty('client');
$reflection_property->setAccessible(true);
$reflection_property->setValue($account, $mockClient);
return $account;
}
When writing tests for the SDK you assume that your api DOES work exactly like it should (and you write tests for your api to assure that).
So using some kind of sandbox or even a complete mock of your api is sufficient.
I would recommend to mock your API using something like wiremock and then write your unit tests around that mock API to make sure everything works as desired.
This way when your production app with break, you can at-least make sure (by running unit tests) that nothing broken in your application side but there can be problem with actual API (i.e response format being changed).
I'm a QA who decided to use SpecFlow for my test automation after some consideration. I think it's brilliant, but missing one feature which I did use often with other test runners such as NUnit - something similar to the TestCaseSource property from NUnit to specify a potentially dynamic set of data for tests to be ran against at run time.
I would often have different data in each environment the test should run in, so cannot specify hardcoded values for test parameters. A trivial example is for checking that each type of user account is able to login, the user account credentials can be retrieved using a DB query to populate each test case dynamically in NUnit:
public List<User> GetTestData()
{
List<User> testData = new List<User>();
testData = MyDatabase.GetAllUsersInfo().ToList();
return testData;
}
[Test, TestCaseSource("GetTestData")]
public void CallLoginService(User user)
{
var response = LoginController.TryLogin(User.UserName, User.Password);
if (response.Error != null)
{
Assert.Fail("Failed to Login: {0}", response.Error);
}
Assert.AreEqual("Logged in ok", response.Message, "Login message not as expected");
}
Obviously this is a simple example of that feature, but I think it describes it well enough. I know we have the ability in SpecFlow to use a Scenario Outline and table of test run input data, but that is still static, so doesn't fit the bill.
I've been looking for a while and have not found anything in SpecFlow like this yet, does anybody know of anything similar to the above which can be used (or planned if anyone who works on the project reads this)?
Thanks :)
I have no idea if anything like this is planned but for now the problem is that there is a background code generation step when you edit your feature file via Visual Studio.
When it is saved in Visual Studio it is parsed and converted into the feature.cs file and that is the one that is compiled and used for testing.
So your process would become
edit your data source
export to feature file
get specflow's VS plugin to convert to feature.cs
run msbuild
run tests via Nunit or similar
I wouldn't do this. Instead I'd focus on getting my tests to be better examples. It sounds like you are to trying to exhaustively cover every possibility. Don't come up with examples to cover every possible case, but instead cover as much logic as possible with fewer tests.
I see several other questions about load testing web services. But as far as I can tell those are all synchronous load testing tools. (Meaning they send a ton of requests but the go one at a time.)
I am looking for a tool where I can say, "I want 100 requests to be launched at the exact same time".
Now, I am new to the whole load testing thing, so it is possible that those tools are asynchronous and I am just missing it.
Anyway, in short my question is: Is there a good tool for load testing WCF Web Services asynchronously (ie lots of threads).
In general, I recommend you look at soapUI, for anything to do with testing web services. They do have load testing features in the Professional edition (I haven't used these yet).
In addition, they've just entered beta with a loadUI product. If it's anywhere near as good as the parent product, then it's worth a hard look.
you can use the Visual Studio load testing agent components to run on multiple client machines and that will allow you to run as asynchronously as you have machines to load.
There is a licence requirement for using this feature.
There are no tools that will allow you to apply a load at exactly the same instant (i.e. within milliseconds), but this is not necessary to load test an application correctly.
For most needs a single load test server running Visual Studio Ultimate edition will be more than enough to get an understand of how your webservice performs under load.
Visual Studio and most other tools I imagine will apply load in an asynchronous manner, but I think in your view you want to apply a set load all at once.
This is not really necessary as in practice load is not applied to a service in this manner.
The best bet for services expecting high load is to load your service until a given number of "requests per second" is reached. Finding what level your application should expect is a bit trickier, but involves figuring out roughly how many users you would expect and the amount they will be using it over a given period.
The other test to do is to setup a load test harness and run the load up until either the webservice starts to perform badly or the test harness runs out of "oomph" and cannot create any more load.
For development time you can use NLoad (http://nload.github.io)
to run load tests on your development machine or testing environment.
For example
public class MyTest : ITest
{
public void Initialize()
{
// Initialize your test, e.g., create a WCF client, load files, etc.
}
public void Execute()
{
// Send http request, invoke a WCF service or whatever you want to load test.
}
}
Then create, configure and run a load test:
var loadTest = NLoad.Test<MyTest>()
.WithNumberOfThreads(100)
.WithDurationOf(TimeSpan.FromMinutes(5))
.WithDeleyBetweenThreadStart(TimeSpan.Zero)
.OnHeartbeat((s, e) => Console.WriteLine(e.Throughput))
.Build();
var result = loadTest.Run();