I am working on a project in which we keep one wiki platform in sync with the content of other. The way we do this is a document edit on 'Wiki A' kicks of a data flow pipeline that transforms data from format of of 'Wiki A' to format of 'Wiki B' and sends this data to 'Wiki B' for import.
I have 3 components.
'Wiki A' which is in PHP
Translation Service which is a Ruby-on-rails service
'Wiki B' which is in Java
I want to build an automated end-to-end testing framework which should ideally be able to test the following: The main need for the testing is my unit tests for each product cannot test the communication between the products and do not test the whole end-to-end data flow.
Edit a page on 'Wiki A'
Test that it kicks of the data flow
Test that the TranslationService transformed the data
Test that 'Wiki B' imports the transformed data
Based on initial research, my options are a recording tools such as Selenium. Selenium can handle the multiple products I want to test, but from what I have seen the tests are fragile.
The other option is some development testing tool like Cucumber/Capybara with which I can write robust tests, but I am not sure how it works in a multiple product architecture, each written in a different language.
Am I looking at it in the correct way? Am I too ambitious to attempt a singular end-to-end testing framework spanning multiple products?
It is possible to write end-to-end tests spanning multiple products written in different languages as long as the products provide some kind of proper interface. Ideally this is some messaging interface (e.g. Http REST). I would suggest to use the Wiki interface directly instead of accessing the UI over the browser.
I assume that 'Wiki A' provides such an interface for adding and changing content. Your integration test first of all uses this interface in order to change some data and trigger the whole process. Then you need to make sure that the content change has been processed. You can do that by verifying the change in 'Wiki B'. Also ideally 'Wiki B' offers some kind of interface to get some content, too. So your test should just use the messaging interfaces of 'Wiki A' and 'Wiki B'.
1) Trigger 'Wiki A' change
2) Verify content on 'Wiki B'
Maybe you need to wait some time between step 1 and 2 for the translation and import. You can write these kind of integration tests fully automated with test frameworks like Citrus (http://citrusframework.org)
Related
I try to setup an integration/API test suite with Karate and consider to use Karate Netty for mocking required services. For the test setup the system under test A (a Spring Boot app) is started up completely. The Karate tests are then executed by a Maven test run against this instance.
The service A depends on multiple other services these needs to be mocked away for the tests. To do so my idea was to configure a running Karate Netty standalone instance as HTTP proxy (done by JVM args of the service A).
Now my idea was to create one test feature file: xyz-test.feature
And the required mocks for this file are defined in an associated mock feature file: xyz-mock.feature
(The test scenarios are rather complex and the responses of the external services could vary)
This means for a full test run I need to load up a couple of mock feature files. So:
What is the matching strategy for multiple mock feature files? Which scenario wins, so to say.
Is there any way to ensure, that the right mock file is used for the associated test file?
(Clearly I can reconfigure the running standalone instance and advice it to use xyz-mock.feature next.
But this would stop me from using parallel execution for my API tests, right?)
I already thought about reusing the Correlation-Id which I can send in for each test and then match against this in the mock file (it is also sent to all called services). But:
Is there a way to define a global matcher per mock file?
It sounds like you need only one mock file. You could boot 2 on different ports if you wanted, but there is no way to "merge" them into one port - if that is what you were looking for.
In my experience, you will be able to have a single mock take care of all your edge cases. This is because Karate's approach is un-conventional: you pretty much write a stateful server. But by keeping variables in memory and some clever JSON-path, you can simulate CRUD with very few lines of code: https://github.com/intuit/karate/tree/master/karate-netty#background
You can use only one at a time, by design
Given the above limitation, here's an interesting idea: add something like an extra pathMatches('/__test/reset') scenario that cleans-up your state and sets the Background variables to things like * def cats = []. Now in each feature, just call the special "reset" URL at the start. The good thing is Karate is thread-safe. Another idea as you said is you can maintain two or three different variables and use some logic to "route" based on a header, again very easy IMO. Use a map of maps, e.g:
def data = { cats1: {}, cats2: {}, cats3: {} }
And you can get the header, e.g. if it is mode: cats1
* def mode = karate.get('requestHeaders.mode[0]')
* def cats = data[mode]
not sure if this answers your question, but if the last Scenario has an "empty" description, it is a "catch all" and can in theory delegate to another server (or mock): https://github.com/intuit/karate/tree/develop/karate-netty#proxy-mode
Your question is a little confusing, so you may have to edit and re-word it if I haven't understood.
EDIT: using multiple mock files should be possible in 1.1.0 onwards: https://github.com/intuit/karate/issues/1566
I have a few Nifi process groups which I want to run integration tests on before promoting to production. The issue is that I can't seem to find any documentation on how to do so.
Data Provenance seems like a promising tool to accomplish what I want, however, over the course of the flowfile's lifecycle, data is published to/from kafka or the file system. As a result, the flowfile UUID changes so I cannot query for it using the nifi-api.
Additionally, I know that Nifi offers a TestRunner library to run tests, however, this seems to only be for processors/processor groups generated via code and not the UI.
Does anyone know of a tool, framework, or pattern for integration and unit testing nifi process groups. Ideally this would be a solution where you can programatically compare input/output of the processor/processor group without modifying the existing workflow.
With the introduction of the Apache NiFi Registry, we have seen users promote flows from a development/sandbox environment to a test/QE environment where there are existing "test harness" flows surrounding the "flow under test" so that they can send repeatable and deterministic (or an anonymized sample of real production data) through the flow and compare the results to an expected value.
As you point out, there is a TestRunner class and a whole testing framework provided for unit tests. While it can be difficult to manually translate a UI-constructed flow to the programmatic construction, you could also create something like a translator to accept a flow template or flow.xml.gz file and convert it into something processable by the test framework.
Maybe plumber will help you with flow testing.
We also wanted to test whole NiFi flows, not just single processor, so we created this library and decided to open-source it.
Simple example in Scala:
// read flow previously exported from NiFi
val template = TemplateDeserializer.deserialize(this.getClass.getClassLoader.getResourceAsStream("exported-flow.xml"))
val flow = NifiTemplateFlowFactory(template).create()
// enqueue some data to any processor
flow.enqueueByName("csv row,12,another value,true", "CsvParserProcessor")
// run entire flow once
flow.run(1)
// get the results from any processor
val records = flow.resultsFromProcessorRelation("LastProcessorInFlow","successRelation")
records should have size 1
This library is still under development so improvements and ideas are welcomed! :)
What is the difference between functional test and end-to-end test?
Techopedia says that end-to-end test is
a methodology used to test whether the flow of
an application is performing as designed from start to finish. The
purpose of carrying out end-to-end tests is to identify system
dependencies and to ensure that the right information is passed
between various system components and systems.
Techopedia also says the following about functional test:
Functional testing is a software testing process used within software
development in which software is tested to ensure that it conforms
with all requirements. Functional testing is a way of checking
software to ensure that it has all the required functionality that's
specified within its functional requirements.
After reading the above two paragraphs, I'm still confused about the difference between them.
I have a node.js application which accepts requests, then parses the request, then sends the parsed data to a Database.
requests parse requests and send data to the database
Client ---------> node.js app --------------------------------------------> Database
How can I write end-to-end test and functional test for the node.js app I mentioned?
I think in both types of the tests, I should treat the node.js app as a black box. And send requests to it. Then check if the output of the black box is correct or not.
It seems that in my case, there's no difference between functional test and end-to-end test.
As I understand it, the biggest difference between the two is that an end-to-end test requires the test to setup the system components as they are in production. Real database, services, queues, etc. The reason for this is to see that your system is wired correctly (database connections, configuration and such).
A functional test can setup the system with in-memory implementations of your application ports, which would make the test run faster and perhaps allow tests to run in parallel (in some cases). The only thing the test cares about is that a feature works as expected. This can reduce the overhead of setting up certain tests, since preparing 3rd party systems with data can be difficult or time consuming.
I think the definitions of functional and end to end testing could vary based on the context of your project. I have seen different people use these terms to describe different things. That being said, usually this is what the 2 terms mean-
Functional testing - This refers to testing the functionality of system based on the requirements. This usually focuses on different requirements of the system and ensure it is working properly. For example - Logging into an application - could be one requirement and then a person could test this functionality manually or in an automated way. Similarly, adding a product to the cart could be one functionality, then, able to make a payment to purchase a product could be a functionality.
End to end testing - This refers to testing the system based on end to end user flows, instead of testing the system has separate components like in unit testing or story level testing. For example - Logging into the application, then adding a product to the shopping cart, then going to the check out screen and then placing an order and then logging out of the application could be one user flow.
What we follow is slightly different and of course difference in just how your team treats each of them. for further clarity,
Functional Test : Tests a feature say login, verify from database if login data is correct, verify if intended event received, or send to a message bus or any external activity in a Prod like environment like staging environment. You test a particular functionality in a real environment.
End to End testing : Test complete feature like, login to app, view product on view page, select product, checkout and do payment. This could cover multiple microservices as well, or maybe multiple teams. If this flow breaks, we can pin point which of the functional tests failed.
Integration Test: Test integration between multiple components, from a wide spectrum of multiple classes to multiple system. Like can UI connect to some external login service, can backend connect to database. If a functional test breaks, we can watch which Int Test failed and so on with unit test.
I was tasked with creating a health check for our production site. It is a .NET MVC web application. There are a lot of dependencies and therefore points of failure e.g. a document repository, Java Web services, Site Minder policy server etc.
Management wants us to be the first to know if ever any point fails. Currently we are playing catch up if a problem arises, because it is the the client that informs us. I have written a suite of simple Selenium WebDriver based integration tests that test the sign in and a few light operations e.g. retrieving documents via the document api. I am happy with the result but need to be able to run them on a loop and notify IT when any fails.
We have a TFS build server but I'm not sure if it is the right tool for the job. I don't want to continuously build the tests, just run them. Also it looks like I can't define a build schedule more frequently than on a daily basis.
I would appreciate any ideas on how best achieve this. Thanks in advance
What you want to do is called a suite of "Smoke Tests". Smoke Tests are basically very short and sweet, independent tests that test various pieces of the app to make sure it's production ready, just as you say.
I am unfamiliar with TFS, but I'm sure the information I can provide you will be useful, and transferrable.
When you say "I don't want to build the tests, just run them." Any CI that you use, NEEDS to build them TO run them. Basically "building" will equate to "compiling". In order for your CI to actually run the tests, it needs to compile.
As far as running them, If the TFS build system has any use whatsoever, it will have a periodic build option. In Jenkins, I can specify a Cron time to run. For example:
0 0 * * *
means "run at 00:00 every day (midnight)"
or,
30 5 * 1-5 *
which means, "run at 5:30 every week day"
Since you are making Smoke Tests, it's important to remember to keep them short and sweet. Smoke tests should test one thing at a time. for example:
testLogin()
testLogout()
testAddSomething()
testRemoveSomething()
A web application health check is a very important feature. The use of smoke tests can be very useful in working out if your website is running or not and these can be automated to run at intervals to give you a notification that there is something wrong with your site, preferable before the customer notices.
However where smoke tests fail is that they only tell you that the website does not work, it does not tell you why. That is because you are making external calls as the client would, you cannot see the internals of the application. I.E is it the database that is down, is a network issue, disk space, a remote endpoint is not functioning correctly.
Now some of these things should be identifiable from other monitoring and you should definitely have an error log but sometimes you want to hear it from the horses mouth and the best thing that can tell you how you application is behaving is your application itself. That is why a number of applications have a baked in health check that can be called on demand.
Health Check as a Service
The health check services I have implemented in the past are all very similar and they do the following:
Expose an endpoint that can be called on demand, i.e /api/healthcheck. Normally this is private and is not accessible externally.
It returns a Json response containing:
the overall state
the host that returned the result (if behind a load balancer)
The application version
A set of sub system states (these will indicate which component is not performing)
The service should be resilient, any exception thrown whilst checking should still end with a health check result being returned.
Some sort of aggregate that can present a number of health check endpoints into one view
Here is one I made earlier
After doing this a number of times I have started a library to take out the main wire up of the health check and exposing it as a service. Feel free to use as an example or use the nuget packages.
https://github.com/bronumski/HealthNet
https://www.nuget.org/packages/HealthNet.WebApi
https://www.nuget.org/packages/HealthNet.Owin
https://www.nuget.org/packages/HealthNet.Nancy
I'm designing a web service running on Google App Engine that scrapes a number of websites and presents their data via a RESTful interface. Based on some background reading, I think I'd like to attempt Test Driven Development (TDD) and develop my tests before I write any business code.
My problem is caused by the fact that my list of scraped elements includes timetables and other records that change quite frequently. The limit of my knowledge on TDD is that you write tests that examine the results of code execution and compare these results to a hardcoded result set. Seeing as the data set changes frequently, this method seems impossible. Assuming that this is true, what would be the best approach to test such an API? How would a large-scale web API be tested (Twitter, Google, Netflix etc.)?
You have to choose the type of test:
Unit tests just test proper operation of your modules (units). You provide input data and test that code outputs proper results. If there are system dependent classes you try to mock them or in case of GAE services, you use google provided local services. Unit tests can be run locally on your machine or on CI servers. There are two popular unit test libs for java: Junit & TestNG.
Integration tests check that various modules (internal & external) work together - they basically check that APIs between modules are working. They are usually run on real servers and call real external services. They are technology specific and are harder to run.
In your case, I'd go with unit tests and provide sets of different input data which you logic should parse and act upon. Since your flow is pretty simple (load data from fixed Url, parse it) you could also embed loading of real data into unit tests (we do this when we parse external sources).
From what you are describing you could easily find yourself writing integration tests. If your aim is to test the logic for processing what is returned from the scraped data (e.g. you know that you are going to get a timetable in a specific format coming in and you now have logic to process that data) you will need to create a SEAM between your web services logic and your processing logic. Once you have done this you should be able to mock the data that is returned from the web service call to always return the same table data and then you can write consistent unit tests against it.
public class ScrapingService : IScrapingService
{
public string Scrape(string url)
{// scraping logic}
}
public interface IScrapingService
{
string Scrape(string url);
}
public class ScrapingProcessor
{
private IScrapingService _scrapingService
// inject the dependency
pubilc ScrapingProcessor(IScrapingService scrapingService)
{
_scrapingService = scrapingService;
}
public void Process(string url)
{
var scrapedData = _scrapingService.Scrape(url)
// now process the scrapedData
}
}
To test you can now create a FakeScrapingService that implements the IScrapingService interface and then return whatever data you like from the Scrape method. There are some very good Mocking frameworks out there that make this type of thing easy. My personal favorite is NSubstitue.
I hope this explanation helps.