I was tasked with creating a health check for our production site. It is a .NET MVC web application. There are a lot of dependencies and therefore points of failure e.g. a document repository, Java Web services, Site Minder policy server etc.
Management wants us to be the first to know if ever any point fails. Currently we are playing catch up if a problem arises, because it is the the client that informs us. I have written a suite of simple Selenium WebDriver based integration tests that test the sign in and a few light operations e.g. retrieving documents via the document api. I am happy with the result but need to be able to run them on a loop and notify IT when any fails.
We have a TFS build server but I'm not sure if it is the right tool for the job. I don't want to continuously build the tests, just run them. Also it looks like I can't define a build schedule more frequently than on a daily basis.
I would appreciate any ideas on how best achieve this. Thanks in advance
What you want to do is called a suite of "Smoke Tests". Smoke Tests are basically very short and sweet, independent tests that test various pieces of the app to make sure it's production ready, just as you say.
I am unfamiliar with TFS, but I'm sure the information I can provide you will be useful, and transferrable.
When you say "I don't want to build the tests, just run them." Any CI that you use, NEEDS to build them TO run them. Basically "building" will equate to "compiling". In order for your CI to actually run the tests, it needs to compile.
As far as running them, If the TFS build system has any use whatsoever, it will have a periodic build option. In Jenkins, I can specify a Cron time to run. For example:
0 0 * * *
means "run at 00:00 every day (midnight)"
or,
30 5 * 1-5 *
which means, "run at 5:30 every week day"
Since you are making Smoke Tests, it's important to remember to keep them short and sweet. Smoke tests should test one thing at a time. for example:
testLogin()
testLogout()
testAddSomething()
testRemoveSomething()
A web application health check is a very important feature. The use of smoke tests can be very useful in working out if your website is running or not and these can be automated to run at intervals to give you a notification that there is something wrong with your site, preferable before the customer notices.
However where smoke tests fail is that they only tell you that the website does not work, it does not tell you why. That is because you are making external calls as the client would, you cannot see the internals of the application. I.E is it the database that is down, is a network issue, disk space, a remote endpoint is not functioning correctly.
Now some of these things should be identifiable from other monitoring and you should definitely have an error log but sometimes you want to hear it from the horses mouth and the best thing that can tell you how you application is behaving is your application itself. That is why a number of applications have a baked in health check that can be called on demand.
Health Check as a Service
The health check services I have implemented in the past are all very similar and they do the following:
Expose an endpoint that can be called on demand, i.e /api/healthcheck. Normally this is private and is not accessible externally.
It returns a Json response containing:
the overall state
the host that returned the result (if behind a load balancer)
The application version
A set of sub system states (these will indicate which component is not performing)
The service should be resilient, any exception thrown whilst checking should still end with a health check result being returned.
Some sort of aggregate that can present a number of health check endpoints into one view
Here is one I made earlier
After doing this a number of times I have started a library to take out the main wire up of the health check and exposing it as a service. Feel free to use as an example or use the nuget packages.
https://github.com/bronumski/HealthNet
https://www.nuget.org/packages/HealthNet.WebApi
https://www.nuget.org/packages/HealthNet.Owin
https://www.nuget.org/packages/HealthNet.Nancy
Related
What is the difference between functional test and end-to-end test?
Techopedia says that end-to-end test is
a methodology used to test whether the flow of
an application is performing as designed from start to finish. The
purpose of carrying out end-to-end tests is to identify system
dependencies and to ensure that the right information is passed
between various system components and systems.
Techopedia also says the following about functional test:
Functional testing is a software testing process used within software
development in which software is tested to ensure that it conforms
with all requirements. Functional testing is a way of checking
software to ensure that it has all the required functionality that's
specified within its functional requirements.
After reading the above two paragraphs, I'm still confused about the difference between them.
I have a node.js application which accepts requests, then parses the request, then sends the parsed data to a Database.
requests parse requests and send data to the database
Client ---------> node.js app --------------------------------------------> Database
How can I write end-to-end test and functional test for the node.js app I mentioned?
I think in both types of the tests, I should treat the node.js app as a black box. And send requests to it. Then check if the output of the black box is correct or not.
It seems that in my case, there's no difference between functional test and end-to-end test.
As I understand it, the biggest difference between the two is that an end-to-end test requires the test to setup the system components as they are in production. Real database, services, queues, etc. The reason for this is to see that your system is wired correctly (database connections, configuration and such).
A functional test can setup the system with in-memory implementations of your application ports, which would make the test run faster and perhaps allow tests to run in parallel (in some cases). The only thing the test cares about is that a feature works as expected. This can reduce the overhead of setting up certain tests, since preparing 3rd party systems with data can be difficult or time consuming.
I think the definitions of functional and end to end testing could vary based on the context of your project. I have seen different people use these terms to describe different things. That being said, usually this is what the 2 terms mean-
Functional testing - This refers to testing the functionality of system based on the requirements. This usually focuses on different requirements of the system and ensure it is working properly. For example - Logging into an application - could be one requirement and then a person could test this functionality manually or in an automated way. Similarly, adding a product to the cart could be one functionality, then, able to make a payment to purchase a product could be a functionality.
End to end testing - This refers to testing the system based on end to end user flows, instead of testing the system has separate components like in unit testing or story level testing. For example - Logging into the application, then adding a product to the shopping cart, then going to the check out screen and then placing an order and then logging out of the application could be one user flow.
What we follow is slightly different and of course difference in just how your team treats each of them. for further clarity,
Functional Test : Tests a feature say login, verify from database if login data is correct, verify if intended event received, or send to a message bus or any external activity in a Prod like environment like staging environment. You test a particular functionality in a real environment.
End to End testing : Test complete feature like, login to app, view product on view page, select product, checkout and do payment. This could cover multiple microservices as well, or maybe multiple teams. If this flow breaks, we can pin point which of the functional tests failed.
Integration Test: Test integration between multiple components, from a wide spectrum of multiple classes to multiple system. Like can UI connect to some external login service, can backend connect to database. If a functional test breaks, we can watch which Int Test failed and so on with unit test.
Can you have automated regression/integration tests for Azure Logic Apps?
And if you can, how? ... especially in the context of CI/CD builds and deployments
... and if you can't, why not!!
There isn't any out-of-the-box tooling yet to provide automated testing of Azure Logic Apps. We have a few customers who have followed one of the following patterns. There is also this article that goes into detail on how to create a Logic App deployment template:
After deployment (using a release management tool like Visual Studio Release Management), a series of unit tests are run (writtin in something like C#) to test the Logic App.
Since a logic app could have any kind of trigger (on queue item, on HTTP request), the code usually performs the action and asserts the result.
A logic app in the resource group that can run a series of basic tests in a workflow. This one requires a bit more chewing on, but idea being you have a workflow that makes use of connectors or "calling nested apps" to perform basic validation tests (ensure connections are active, etc.)
It's something we have had discussions on from time-to-time, but would love to know if you have any thoughts on what types of tooling/configuration you'd want to configure for an app (remember that some apps "trigger" on something like a message in a queue or a file in FTP).
I would like to share one of the approach for LogicApp testing that my team has followed.
First level of validation is the ARM template deployment status (ProvisioningState) which should not have any errors.
After that we have developed test automation using the logic app sdk which does the following
Get auth token.
Execute a specific logic app trigger with a synthetic transaction.
Waits till the execution is completed.
Gets logic app & its action status (succeed, failed or skipped), validates it as per the expected scenario.
Gets the outputs from each action execution, validates them against an expected scenario.
Repeat above steps for all the various cases that logic app might go through.
Hook this all-in CI/CD :)
Deployed an LA, ran a synthetic transaction & validated the results.
Hope this helps.
I have Jenkins project that perform some sort of sanity check on couple of independent documents. Check result is written in JUnit XML format.
When one document test fails, entire build fails. Jenkins can be simply configured to send email to commiter in this situation. But I want to notify commiters only when new test failed or any failed test was fixed with the commit. They are not interested in failed tests for documents they have not edited. Email should contain only information of changes in tests, not full test report. Is it possible to send this kind of notification with any currently available Jenkins plugins? What could be the simplest way to achieve this?
I had the same question today. I wanted to configure Jenkins sending notifications only when new tests fail.
What I did was to install email-ext plugin.
You can find there a special trigger that is called Regression (An email will be sent any time there is a regression. A build is considered to regress whenever it hasmore failures than the previous build.)
Regarding fixed tests, there is Improvement trigger (An email will be sent any time there is an improvement. A build is considered to have improved wheneverit has fewer failures than the previous build.)
I guess that this is what you are looking for.
Hope it helps
There's the email-ext plugin. I don't think it does exactly what you want (e.g. sending only emails to committers who have changed a file that is responsible for a failure). You might be able to work around that/extend the plugin though.
Also have a look at the new Emailer, which talks about new email functionality in core hudson that is based on aforementioned plugin.
I was wondering if there was a way to do this in Hudson (or with any of the various plugins). My IDEAL scenario:
I want to trigger a build based on a job through a REST-like API, and on that build, I want it to return me a job ID. After-wards, I would like to poll this ID to see its status. When it is done, I would like to see the status, and the build number.
Now, since I can't seem to get that working, here is my current solution that I have yet to implement:
When you do a REST call to do a build, its not very REST-ful. It simply returns HTML, and I would have to do a kind of parsing to get the job ID. Alternatively, I can do a REST call for all the history listing all the jobs, and the latest one would be the one I just built. Once I have that, I can poll the console output for the output of the build.
Anyone know a way I can implement my "ideal" solution?
Yes, you can use the Hudson Remote API for this (as #Dan mentioned). Specifically, you need to configure your job to accept Remote Triggers (Job Configuration -> Build Triggers -> Trigger builds remotely) and then you can fire off a build with a simple HTTP GET to the right url.
(You may need to jump through a couple additional hoops if your Hudson requires authentication.)
I'm able to start a Hudson job with wget:
wget --auth-no-challenge --http-user=test --http-password=test "http://localhost:8080/job/My job/build?TOKEN=test"
This returns a bunch of HTML with a build number #20 that you could parse. The build number can then be used to query whether the job is done / successful.
You can examine the Hudson Remote API right from your browser for most of the Hudson web pages that you normally access by appending /api (or /api/xml to see the actual XML output), e.g. http://your-hudson/job/My job/api/.
Update: I see from your question that you probably know much of what I wrote. It is worth exploring the built-in Hudson API documentation a bit. I just discovered this tidbit that might help.
You can get the build number of the latest build (as plain text) from the URL: http://your-hudson/job/My job/lastBuild/buildNumber
Once you have the build number, I think the polling and job status is straightforward once you understand the API.
And what if you don't want the latest build number, but you want the build number of the build that was triggered by hitting the build URL ?
As far as I can tell, hitting that URL returns a 302 that redirects you to the job's mainpage, with no indication whatsoever of what the build number is of the one that you triggered.
I see several other questions about load testing web services. But as far as I can tell those are all synchronous load testing tools. (Meaning they send a ton of requests but the go one at a time.)
I am looking for a tool where I can say, "I want 100 requests to be launched at the exact same time".
Now, I am new to the whole load testing thing, so it is possible that those tools are asynchronous and I am just missing it.
Anyway, in short my question is: Is there a good tool for load testing WCF Web Services asynchronously (ie lots of threads).
In general, I recommend you look at soapUI, for anything to do with testing web services. They do have load testing features in the Professional edition (I haven't used these yet).
In addition, they've just entered beta with a loadUI product. If it's anywhere near as good as the parent product, then it's worth a hard look.
you can use the Visual Studio load testing agent components to run on multiple client machines and that will allow you to run as asynchronously as you have machines to load.
There is a licence requirement for using this feature.
There are no tools that will allow you to apply a load at exactly the same instant (i.e. within milliseconds), but this is not necessary to load test an application correctly.
For most needs a single load test server running Visual Studio Ultimate edition will be more than enough to get an understand of how your webservice performs under load.
Visual Studio and most other tools I imagine will apply load in an asynchronous manner, but I think in your view you want to apply a set load all at once.
This is not really necessary as in practice load is not applied to a service in this manner.
The best bet for services expecting high load is to load your service until a given number of "requests per second" is reached. Finding what level your application should expect is a bit trickier, but involves figuring out roughly how many users you would expect and the amount they will be using it over a given period.
The other test to do is to setup a load test harness and run the load up until either the webservice starts to perform badly or the test harness runs out of "oomph" and cannot create any more load.
For development time you can use NLoad (http://nload.github.io)
to run load tests on your development machine or testing environment.
For example
public class MyTest : ITest
{
public void Initialize()
{
// Initialize your test, e.g., create a WCF client, load files, etc.
}
public void Execute()
{
// Send http request, invoke a WCF service or whatever you want to load test.
}
}
Then create, configure and run a load test:
var loadTest = NLoad.Test<MyTest>()
.WithNumberOfThreads(100)
.WithDurationOf(TimeSpan.FromMinutes(5))
.WithDeleyBetweenThreadStart(TimeSpan.Zero)
.OnHeartbeat((s, e) => Console.WriteLine(e.Throughput))
.Build();
var result = loadTest.Run();