I want to start a pet project to get confident with creating Web Applications from scratch and I want to use BDD and TDD.
I read in "Growing Object-Oriented Software: Guided by Tests" that we should start from the thinnest slice of the system that allows us to have a walking skeleton of the entire application so we have can have a quick feedback and start wondering about the production/deployment procedures.
After writing the acceptance test in BDD I would move to finer granularity tests such as Unit tests.
Imagining that the web application is about finding the superhero who is more similar to you, I would write a BDD scenario such as:
When I insert my information
Then the system should tell that the superhero I'm more similar to is "Batman"
I'm ignoring the authentication on purpose so we can focus on the main functionality of the system. This scenario assumes that there is a working infrastructure behind, so that the scenario above can be replicated automatically, end-to-end.
Assuming that I want to lay out the web application in different layers (Web Server, Application Server and Database), how can I implement this test?
Supposing I want to use Selenium WebDriver to simulate the user, what layers must be mocked and what layer is going to be tested first?
I also thought about starting from testing the API only, but that wouldn't be a end-to-end test, but we would test the application only partially.
Cucumber is a good framework for BDD. UI/Selenium tests can be a bit flakey so I would suggest less of them (see Testing Pyramid).
Personally I think I would work at the API level as you suggested for starters, then you have decoupled the UI from the app.
Write some Cucumber Integration Tests of the Application (BDD) at the API level, describing high level features. I wouldn't mock at this level, as you want to see the whole app working at this point.
Then as you implement these, use TDD, here you will want to mock out externals like File IO/Database. Once all the TDD Unit Tests are done, the overall feature should then pass. Continue until all features are completed.
Continue like this and then you have a solid set of unit tests and a solid set of Integration (feature) tests at API level.
Once done (or in parallel to get a vertical slice) start the UI using 'happy day' scenario selenium tests, all that really needs testing here is the UI code and that is correctly hooked up to the App, as the main App will be well tested already.
Just my view - Hope it helps.
I'd use Cucumber for this (but I'm terribly biased), and I would start by writing some scenarios around the core part of your application.
Some important rules to follow when writing your scenarios
Use your domain language
Only talk about WHAT you are doing and WHY its important
Don't talk about HOW you do things.
So with your superhero site you can start with thinking about things you need to do to get started.
Generally when you have discovered a resource (superhero) you will have some basic CRUD operations you can explore, so
creating superhero
editing a superhero
deleting a superhero
COMPARING a superhero ** this is your key starting point **
Lets start working on the comparing a superhero scenario
Given there are superheroes Batman and Superman
And I am like Batman
When I find my superhero
Then I should see I am like Batman
Now lets start implementing this scenario. One important thing here is to not put all your code in your step definitions. Instead make each step definition a call to a helper method.
note: all following code will be ruby
Given 'there are superheroes Batman and Superman' do
create_superhero batman
create_superhero superman
end
And now we can get into the BDD red/green testing cycle.
So we will start getting errors like No method found create_superhero and undefined constant batman. We can address these with a helper module in our step definitions.
module CreateSuperHeroStepHelper
def create_superhero(attrs={})
Superhero.create(attrs)
end
def batman
{
...
}
end
...
end
And from this code we can start working on the domain objects for our application e.g. Superhero.
To get this step working you'll have to interact with your underlying web framework. Once you have it working, in its most basic form you can move onto the next step.
Along the way you'll have all sorts of choices about what you do next. If you stick to working on this core scenario, you'll have to be clever about how you implement things. If you decide to work on scenarios that seem like pre-requisites e.g. user accounts, create superhero, enter your superhero characteristics, add a superhero characteristic. Then you should end up with tools which you can use for your core scenario (e.g. you'll have create_superhero) but it will take you longer to get to the core of your application.
There is no need to be thinking about unit tests, layers or mocking here. Thats low-level detail about HOW you do things which comes much later. So long as you have some basic competency in your web framework you should be to start exploring your domain and use BDD to create simple tools that you can use to explore your World of superheroes.
Good luck
Related
For instance, I assume this is what SDETs do?
They don't actually write the functional code but they're able to write integration/unit tests, am i correct?
But can someone learn to read code and then start writing tests?
This is actually a good question. I have been at the same place when I was working only on Manual Testing. Here is how I experienced things when I transitioned to automation. To answer your question, yes someone can read code and start writing tests on it but you need to understand the code that you are going to test.
There are different types of testing methodologies that are used when testing an application. These tests are done in layers so that the application is properly tested. Here is how the layering looks like:
1) Unit testing: This part is usually written by developers. This is because they have written the code and know the functionalities of the code and is easier for them to write. I am an SDET and I have written unit tests. There was only one opportunity that presented itself and it was when we were refactoring our code and there was a lot of room to write Unit Tests. In unit tests, you test functions in isolation by giving it some values and verifying an expected value. This is not something an SDET does, but should be able to do if provided the chance.
2) Integration testing: This part is also usually written by developers but the definition of integration testing is a bit vague. It means testing multiple modules in isolation. This can be modules in backend or modules in frontend but not together. Frameworks that help achieve this are code level integration tests for the technology you are using. Like for Angular application, there are deep integration tests that test the HTML and CSS of a component and there are shallow integration tests that just test two component's logic together. This can be written by an SDET but is usually written by the developer.
3) API testing (contract based testing): Pact helps us achieve this. There are other tools like rest assured, postman and jmeter that help in testing API end points. Pact helps test the integration of APIs on the frontend and verifies that integration in backend. This is very popular with microservices. This is something that can be written either by the developer or by the SDET.
4) End to End testing: This is something that is the sole responsibility of SDET. This covers testing of user flows depending on user stories. It is testing the entire stack together. Backend and frontend. This allows SDETs to automate how a user would use the application. This is also called as blackbox testing. There are different frameworks that help achieve this. Selenium, Protractor, Cypress, Detox etc. This is the sole responsibility of an SDET.
5) Load testing: This is again something that an SDET does. Using tools like hey, jmeter, loadrunner etc. These tests allow the SDET to initiate a heavy load on the system and check for breaking points of the system.
6) Performance testing: Testing the performance of the webpage for an end user depending on the page load time, the SEO optimisation and the weight of elements of page. This is something achieved by google's lighthouse tool that is an amazing tool to use. I am not aware of anything else that is as amazing as lighthouse because it gives us a lot of data that we can use to improve our website. This is a primary job of an SDET.
7) CI/CD: Continuous Integration and Continuous Deployments is something that requires architectural knowledge of the system. This is something you can do when you are an SDET3 or a lead QA engineer. For systems like AWS and GCP, using CI build tools like Jenkins and CircleCI, one can set up a pipeline that runs all the above tests when ever a branch is merged into master or whenever a pull request is created. Creating the pipeline will require you to have knowledge of Docker, Kubernetes, Jenkins and your test frameworks. First you dockerize your tests, then you build the image and push it to a directory in cloud, then you use the image to create a kubernetes job that runs everytime a change is presented in your code.
These are the levels of work that an SDET does. It takes time and hard work to have an understanding of all testing frameworks and how everything fits together. An SDET should have knowledge of the server, http protocols, frontend, backend, browsers, caching, pipeline management and orchestration of tests.
Yes, sure. You can write unit tests increasing test coverage of the codebase. That's very qualified work from software test engineering since you need to be aware what is going on in the code. This skill is definitely great!
I advise you to take a look on so called "mutation coverage". Usage of mutation coverage as a better metric than simple unit test coverage. Mutation tests are changing logical operators in the different parts of the codebase (making so called "mutants") and then are running unit tests to find out how many unit tests will fail showing their effectiveness (if after mutants were injected the result is the same as without them - unit tests quality is low and they won't catch any new injected issues to the codebase).
I'm writing thesis and my supervisor told me I have to make chapter about testing but I have no idea what it should include. Can you give me some hints?
Thank you
In my experience the foundations of testing, in application development, rely mainly onto a good architecture design.
Your application must be modular, structured in layers (n-tier architecture) and/or in services.
In order to test the whole application you have to test each module with some kind of unit testing approach. In order to do this you have to isolate each module and treat it as a black box.
Then your unit tests for the specific module should run against the interface module and all the interactions of the module towards other application components must be replaced with mock components, that simulates real behaviours. This is important because lets you controll what to expect in terms of successful results or failures.
After unit testing each component you can go on designing integration tests, that test the interactions between each module.
Then there are the functional tests that check that the application user experience and use cases meets the client requirements.
As a final step the appliation testing must be configured as a step of DevOps routines such as continous integration in order to keep your development workflow clean and secure from regressions and bugs.
Although the application testing is a HUGE argument i hope that this could help you find more informations for your thesis. Good luck!
I'm using a twitter gem which basically accesses twitter and lets me grab tweets, timeline etc. Its really good but I have a lot of my code that uses the stuff it returns and I need to test it. The things the gem returns aren't exactly simple strings, there pretty complex objects (scary as well) so im left scratching my head.
So basically I'm looking for an answer, book, blog, open-source project that can show me the rights and wrongs of testing around external services.
answers that are either not language centric or ruby/rails centric would most greatly be appreciated.
What you are really talking about are two different kinds of testing that you would want to accomplish - unit tests and integration tests.
Unit tests will test the validity of the methods, independently of any external data. You should look into some sort of mocking framework, based on whatever language it is that you are using. You are basically looking to say, with the tests, something equivalent to "if these assumptions are qualified, then this test should yield..." The making framework will define your assumptions, in terms of saying that certain classes/objects are set in a particular way and can be assumed to be valid. These are the tests that will not rely on Twitter being alive, or the third part library/API being responsive.
Integration tests will perform tests live against the data source, consuming the library/API to perform actual actions. Where it gets tricky, since you are using a third party service, is in writing out to the service (i.e. if you are creating new Tweets). If you are, you could always create an account on Twitter that could be used just for write operations. Generally, if you were testing against a local database - for example - you could then, instead, use transactions to test similar operations; rolling back the transactions instead of committing them.
Here are a couple of non-language specific, high-level definitions:
Wikipedia (Software Testing)
Wikipedia (Mock Object)
I am from a .NET stack, so I won't pretend to know much about Ruby. A quick Google search, though, did reveal the following:
Mocha (Ruby Mocking Framework)
You can easily stub at the http layer using something like wiremock http://wiremock.org/ I've used this on a few projects now and it's quite powerful and fast. This will eliminate all the set up code of code based mocking - just fire up the jar with related mappings and bob's your uncle.
There are a lot of testing methods out there i.e. blackbox, graybox, unit, functional, regression etc.
Obviously, a project cannot take on all testing methods. So I asked this question to gain an idea of what test methods to use and why should I use them. You can answer in the following format:
Test Method - what you use it on
e.g.
Unit Testing - I use it for ...(blah, blah)
Regression Testing - I use it for ...(blah, blah)
I was asked to engage into TDD and of course I had to research testing methods. But there is a whole plethora of them and I don't know what to use (because they all sound useful).
1. Unit Testing is used by developers to ensure unit code he wrote is correct. This is usually white box testing as well as some level of black box testing.
2. Regression Testing is a functional testing used by testers to ensure that new changes in system has not broken any of existing functionality
3. Functional testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. Functionality testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic
.
This Test-driven development and Feature Driven Development wiki articles will be of great help for you.
For TDD you need to follow following process:
Document feature (or use case) that
you need to implement or enhance
in your application that
currently does not exists.
Write set of functional test
cases that can ensure above
feature (from step 1) works. You may need to
write multiple test cases for
above feature to test all different
possible work flows.
Write code to implement above feature (from step 1).
Test this code using test cases you
had written earlier (in step 2). The actual
testing can be manual but I would recommend to create automated tests
if possible.
If all test cases pass, you are good to
go. If not, you need to update code (go back to step 3)
so as to make the test case pass.
TDD is to ensure that functional test cases which were written before you coded should work and does not matter how code was implemented.
There is no "right" or "wrong" in testing. Testing is an art and what you should choose and how well it works out for you depends a lot from project to project and your experience.
But as a professional Test Expert my suggestion is that you have a healthy mix of automated and manual testing.
(Examples below are in PHP but you can easily find the correct examples for what ever langauge/framework you are using)
AUTOMATED TESTING
Unit Testing
Use PHPUnit to test your classes, functions and interaction between them.
http://phpunit.sourceforge.net/
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.webinade.com/web-development/functional-testing-in-php-using-selenium-ide
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
This answer is (almost) identical to one that I gave to another question. Check out that question since it had some other good answers that might help you.
How can we decide which testing method can be used?
I usually do the following things:
Page consistency in case of multi-page web sties.
Testing the database connections.
Testing the functionalities that can be affected by the change I just made.
I test functions with sample input to make sure they work fine (especially those that are algorithm-like).
In some cases I implement features very simply hard-coding most of the settings then implement the settings later, testing after implementing every setting.
Most of these apply to applications, too.
Well before going to the answer i would like to clear testing concept about multiple methods.
There are six main testing types which cover all most all testing methods.
Black Box Testing
White Box Testing
Grey Box Testing
Functional Testing
Integration Testing
Usability Testing
Almost all Testing methods lies under these types, you can also use some testing method in multiple types like you can use Smoke testing in black box or white box approach on the basis of resources available to test.
So for testing a web site completely you need to use at least following testing methods on the basis of resources available to test. These are at least methods which should be used to test a web site, but there may be some more imp methods on the basic of nature of website.
Requirement Testing
Smock Testing
System Testing
Integration Testing
Regression Testing
Security Testing
Performance & Load Testing
Deployment Testing
You should at least use all of above (8) testing methods to test a web site no matter what testing type you are focusing. You can automate you test in some areas and you can do this manually it all depends upon the resources availability.
There is specifically no hard and fast rule to follow any testing type or any method. As you know "Testing Is An ART" so art don't have rules or boundaries. Its totally up to you What you use to test and how.......
Hope you got the answer of question.
Selenium is very good for testing websites.
The answer depends on the Web framework used (if any). Django for example has built-in testing functions.
For PHP (or functional web testing), SimpleTest is pretty good and well... simple. It support Unit Testing (PHP only) and Web Testing. Tests can run in the IDE (Eclipse), or in the browser (meaning on your server).
The other answers posted so far focus on unit/functional/performance/etc. testing, and they are all reasonable.
However, one the key questions you should ask is, "how effective is my testing?".
This is often answered with test coverage tools, that determine which parts of your application actually get exercised by some set of tests. The ideal test coverage tool lets you test your application by any method you can imagine (including all the standard answers above) and will then report what part and what percentage of your code was exercised. Most importantly, it will tell you what code you did not exercise. You can then inspect that code and decide if more testing is warranted, or if you don't care. If the untested code has to do with "disk full error handling" and you belive that 1TB disks are common, you might decide to ignore that. If the untested code is the input validation logic leading to SQL queries, you might decide that you must test that logic to ensure that no SQL injection attacks can occur.
What test coverage tools let you do it to make a rational decision that you have tested adequately, using data about what parts of your code has been exercised. So regardless of how you test, best practices indicates you should also do test coverage analysis.
Test coverage tools can be obtained from a variety of sources. SD provides a family of test coverage tools that handle C, C++, Java, C#, PHP and COBOL, all of which are used to support web site testing in various ways.
Recently I've came up with the question is it worth at all to spent development time to generate automatic unit test for web based projects? I mean it seems useless at some point because at some point those projects are oriented on interactions with users/clients, so you cannot anticipate the whole possible set of user action so you be able to check the correctness of content showed. Even regression test can hardly be done. So I'm very eager to know to know the opinion of other experienced developers.
Selenium have a good web testing framework
http://seleniumhq.org/
Telerik are also in the process of developing one for web app testing.
http://www.telerik.com/products/web-ui-test-studio.aspx
You cannot anticipate the whole
possible set of user action so you be
able to check the correctness of
content showed.
You can't anticipate all the possible data your code is going to be handed, or all the possible race conditions if it's threaded, and yet you still bother unit testing. Why? Because you can narrow it down a hell of a lot. You can anticipate the sorts of pathological things that will happen. You just have to think about it a bit and get some experience.
User interaction is no different. There are certain things users are going to try and do, pathological or not, and you can anticipate them. Users are just inputting particularly imaginative data. You'll find programmers tend to miss the same sorts of conditions over and over again. I keep a checklist. For example: pump Unicode into everything; put the start date after the end date; enter gibberish data; put tags in everything; leave off the trailing newline; try to enter the same data twice; submit a form, go back and submit it again; take a text file, call it foo.jpg and try to upload it as a picture. You can even write a program to flip switches and push buttons at random, a bad monkey, that'll find all sorts of fun bugs.
Its often as simple as sitting someone down who's unfamiliar with the software and watching them use it. Fight the urge to correct them, just watch them flounder. Its very educational. Steve Krug refers to this as "Advanced Common Sense" and has an excellent book called "Don't Make Me Think" which covers cheap, simple user interaction testing. I highly recommend it. It's a very short and eye opening read.
Finally, the client themselves, if their expectations are properly prepared, can be a fantastic test suite. Be sure they understand its a work in progress, that it will have bugs, that they're helping to make their product better, and that it absolutely should not be used for production data, and let them tinker with the pre-release versions of your product. They'll do all sorts of things you never thought of! They'll be the best and most realistic testing you ever had, FOR FREE! Give them a very simple way to report bugs, preferably just a one button box right on the application which automatically submits their environment and history; the feedback box on Hiveminder is an excellent example. Respond to their bugs quickly and politely (even if its just "thanks for the info") and you'll find they'll be delighted you're so responsive to their needs!
Yes, it is. I just ran into an issue this week with a web site I am working on. I just recently switched-out the data access layer and set up unit tests for my controllers and repositories, but not the UI interactions.
I got bit by a pretty obvious bug that would have been easily caught if I had integration tests. Only through integration tests and UI functionality tests do you find issues with the way different tiers of the application interact with one another.
It really depends on the structure and architecture of your web application. If it contains an application logic layer, then that layer should be easy to unit test with automating tools such as Visual Studio. Also, using a framework that has been designed to enable unit testing, such as ASP.NET MVC, helps alot.
If you're writing a lot of Javascript, there have been a lot of JS testing frameworks that have come around the block recently for unit testing your Javascript.
Other than that, testing the web tier using something like Canoo, HtmlUnit, Selenium, etc. is more a functional or integration test than a unit test. These can be hard to maintain if you have the UI change a lot, but they can really come in handy. Recording Selenium tests is easy and something you could probably get other people (testers) to help you create and maintain. Just know that there is a cost associated with maintaining tests, and it needs to be balanced out.
There are other types of testing that are great for the web tier - fuzz testing especially, but a lot of the good options are commercial tools. One that is open source and plugs into Rails is called Tarantula. Having something like that at the web tier is a nice to have run in a continuous integration process and doesn't require much in the form of maintenance.
Unit tests make sense in TDD process. They do not have much value if you don't do test-first development. However the acceptance test are a big thing for quality of the software. I'd say that acceptance test is a holy grail of the development. Acceptance tests show whether the application satisfies the requirements. How do I know when to stop developing the feature --- only when all my acceptance test pass. Automation of acceptance testing a big thing because I do not have to do it all manualy each time I make changes to the application. After months of development there can be hundreds of test and it becomes unfeasible (sometime impossible) to run all the test manually. Then how do I know if my application still works?
Automation of acceptance tests can be implemented with use of xUnit test frameworks, which makes a confusion here. If I create an acceptance test using phpUnit or httpUnit is it a unit test? My answer is no. It does not matter what tool I use to create and run test. Acceptance test is the one that show whether the features is working IAW requirements. Unit test show whether a class (or function) satisfies the developer's implementation idea. Unit test has no value for the client (user). Acceptance test has a lot of value to the client (and thus to developer, remember Customer Affinity)
So I strongly recommend creating automated acceptance tests for the web application.
The good frameworks for the acceptance test are:
Sahi (sahi.co.in)
Silenium
Simpletest (I't a unit-test framework for php, but includes the browser object that can be used for acceptance testing).
However
You have mentioned that web-site is all about user interaction and thus test automation will not solve the whole problem of usability. For example: testing framework shows that all tests pass, however the user cannot see the form or link or other page element due to accidental style="display:none" in the div. The automated tests pass because the div is present in the document and test framework can "see" it. But the user cannot. And the manual test would fail.
Thus, all web-applications needs manual testing. The automated test can reduce the test workload drastically (80%), but manual test are as well significant for the quality of the resulting software.
As for the Unit testing and TDD -- it make the code quality. It is beneficial to the developers and for the future of the project (i.e. for projects longer that a couple of month). However TDD requires skill. If you have the skill -- use it. If you don't consider gaining the skill, but mind the time it will take to gain. It usually takes about 3 - 6 month to start creating a good Unit tests and code. If you project will last more that a year, I recommend studding TDD and investing time in proper development environment.
I've created a web test solution (docker + cucumber); it's very basic and simple, so easy to understand and modify / improve. It lies in the web directory;
my solution: https://github.com/gyulaweber/hosting_tests