How can i do smoke and monkey testing in ABAP? Are there tools for that ?
Smoke testing can be automatized by using eCATT. Note that the creation of eCATT tests is quite time-consuming, so you should do a cost-benefit analysis beforehand.
I don't know of any tools for monkey testing, except maybe creating a ABAP Unit test suit which uses randomly generated data.
Related
For instance, I assume this is what SDETs do?
They don't actually write the functional code but they're able to write integration/unit tests, am i correct?
But can someone learn to read code and then start writing tests?
This is actually a good question. I have been at the same place when I was working only on Manual Testing. Here is how I experienced things when I transitioned to automation. To answer your question, yes someone can read code and start writing tests on it but you need to understand the code that you are going to test.
There are different types of testing methodologies that are used when testing an application. These tests are done in layers so that the application is properly tested. Here is how the layering looks like:
1) Unit testing: This part is usually written by developers. This is because they have written the code and know the functionalities of the code and is easier for them to write. I am an SDET and I have written unit tests. There was only one opportunity that presented itself and it was when we were refactoring our code and there was a lot of room to write Unit Tests. In unit tests, you test functions in isolation by giving it some values and verifying an expected value. This is not something an SDET does, but should be able to do if provided the chance.
2) Integration testing: This part is also usually written by developers but the definition of integration testing is a bit vague. It means testing multiple modules in isolation. This can be modules in backend or modules in frontend but not together. Frameworks that help achieve this are code level integration tests for the technology you are using. Like for Angular application, there are deep integration tests that test the HTML and CSS of a component and there are shallow integration tests that just test two component's logic together. This can be written by an SDET but is usually written by the developer.
3) API testing (contract based testing): Pact helps us achieve this. There are other tools like rest assured, postman and jmeter that help in testing API end points. Pact helps test the integration of APIs on the frontend and verifies that integration in backend. This is very popular with microservices. This is something that can be written either by the developer or by the SDET.
4) End to End testing: This is something that is the sole responsibility of SDET. This covers testing of user flows depending on user stories. It is testing the entire stack together. Backend and frontend. This allows SDETs to automate how a user would use the application. This is also called as blackbox testing. There are different frameworks that help achieve this. Selenium, Protractor, Cypress, Detox etc. This is the sole responsibility of an SDET.
5) Load testing: This is again something that an SDET does. Using tools like hey, jmeter, loadrunner etc. These tests allow the SDET to initiate a heavy load on the system and check for breaking points of the system.
6) Performance testing: Testing the performance of the webpage for an end user depending on the page load time, the SEO optimisation and the weight of elements of page. This is something achieved by google's lighthouse tool that is an amazing tool to use. I am not aware of anything else that is as amazing as lighthouse because it gives us a lot of data that we can use to improve our website. This is a primary job of an SDET.
7) CI/CD: Continuous Integration and Continuous Deployments is something that requires architectural knowledge of the system. This is something you can do when you are an SDET3 or a lead QA engineer. For systems like AWS and GCP, using CI build tools like Jenkins and CircleCI, one can set up a pipeline that runs all the above tests when ever a branch is merged into master or whenever a pull request is created. Creating the pipeline will require you to have knowledge of Docker, Kubernetes, Jenkins and your test frameworks. First you dockerize your tests, then you build the image and push it to a directory in cloud, then you use the image to create a kubernetes job that runs everytime a change is presented in your code.
These are the levels of work that an SDET does. It takes time and hard work to have an understanding of all testing frameworks and how everything fits together. An SDET should have knowledge of the server, http protocols, frontend, backend, browsers, caching, pipeline management and orchestration of tests.
Yes, sure. You can write unit tests increasing test coverage of the codebase. That's very qualified work from software test engineering since you need to be aware what is going on in the code. This skill is definitely great!
I advise you to take a look on so called "mutation coverage". Usage of mutation coverage as a better metric than simple unit test coverage. Mutation tests are changing logical operators in the different parts of the codebase (making so called "mutants") and then are running unit tests to find out how many unit tests will fail showing their effectiveness (if after mutants were injected the result is the same as without them - unit tests quality is low and they won't catch any new injected issues to the codebase).
I've been asked by an interviewer the following questions about the diagram below:
Why should less effort be required for UI testing?
Is the test pyramid meant for programmers or testers?
Thanks for your help on these questions.
My answer for 1. is that it does not mean "less effort" should be put into UI testing, but that "less code should be covered" by UI testing than the larger portions.
As for 2., it should by considered by anybody who is testing the code. In some organizations that is (and should be) most often a specialized tester and in others, a developer.
My impression is that these are not very good questions, but I do not mention this during the interview.
As per Test Pyramid: the key to good automated test strategy the following diagram represents the Test Pyramid which was originally reflected in one of Martin Fowler's article which was based on a concept developed by Mike Coen.
Test Pyramid
The test pyramid is a tool to fix the problem of over-reliance on long-running Automated / Manual UI tests.
Take aways from the TestPyramid
The pyramid implies that:
Tests on the lower levels are cheaper to write and maintain, and quicker to run.
Tests on the upper levels are more expensive to write and maintain, and slower to run.
Ideally, you should have lots of unit tests, some service tests, and very few UI tests.
This concept is exactly the reverse case of the Testing Ice Cream Cone which is the most often found Test Strategy followed across the software industry where we have very few unit tests, some UI test, lots of QA tests and lots of Manual Tests.
Testing Ice Cream Cone
The Testing Ice Cream Cone approach can be easily identified where the QA department has created an automated test suite, but the development team has not. As a result, tests will be very long running and flakey because the development team has not helped build the suite or architect the application in a way that makes it easy to test. The new check-ins by the developers breaks the product functionality regularly and they are relying on the QA department to report the defects to fix it up.
And further, how they relate together or even if they do?
What would one do to understand the various pieces to a simple question, how to I properly build a testing facility for my (web or other) application?
Agile Development is a banner term for many things, too numerous to mention, including Scrum and TDD. It typically, but not always follows the Agile Manifesto.
SCRUM
This is a particular flavour of agile. This diagram from wikipedia shows the process:
See wikipedia for more info.
Unit Testing
This is the art of writing code that tests code. Failing tests indicate a problem in your solution.
Test Driven Development
This is the practice of writing tests before code, some of the advantages being that untested code isn't added to the solution, and that the code written is testable.
A proper testing facility, usually leverages something along the lines of xunit, junit, nunit, mstest depending on the framework used, these tests are typically ran via a Continuous Integration build on some kind of build server. That is a build that runs everytime the code changes, that executes tests. This way problems are identified quicker.
Unit testing is a practice of writing code tests. TDD is the practice of writing them "before". BDD is the practice of writing behavior/spec driven tests. Can I write BDD "after" or do I have to do it always "before"?
If you write BDD "after", and it's not BDD, then what it is called?
By definition of Behaviour Driven Development, you cannot write the behaviour tests after the code, however that does not mean that doing so isn't useful. You may get more benefit from writing the spec tests first, but they are still useful as regression system tests for your application. So while you're technically not practicing BDD, writing these tests is a good idea. One of the big perks of BDD is that it guides the development of the particular behaviour, so you are losing a lot of value by adding them later, but they still serve some use.
This is the same as writing unit tests after the code in TDD. It's technically not TDD, but having the tests is obviously still useful.
Behavior-Driven Development (BDD) is a variation of Test-Driven Development (TDD) and just like with TDD you should write your tests first.
Some people call BDD for TDD done right, or the way it was intended. Also, you could say that BDD is a mix of Domain-Driven Development (DDD) and TDD.
BDD after development is not BDD, and it is a case of validation rather than specification.
However as the other guys mentioned, it does not mean that adding in an acceptance test suite after-the-fact has no value. You will be building a suite of regression acceptance tests that validate behaviour, before proceeding with further development (large refactoring jobs or new features being added).
From experience I would say if you are to do this task, it is best that the key developers who wrote the production code stay well away from writing the acceptance tests (hopefully in the form of Gherkin scripts); and those that are writing them go back to the original requirements documentation (if any) and most definitely collaborate with some of the stakeholders in doing so. This will help make sure that the acceptance tests you write are closer to specification.
I like the observation that BDD-After is simply a case of writing validation. I also appreciate the comments that a developer doing BDD-After misses some of the other benefits of BDD-As-You-Go. One point that seems worth adding is that writing a secenario/test before the implementation and then having the test pass is also a type of validation that the test itself is sound. Writing a passing test for a feature that already works (BDD-After) may leave a developer wondering if their test will "fail appropriately" should a feature get broken.
There are a lot of testing methods out there i.e. blackbox, graybox, unit, functional, regression etc.
Obviously, a project cannot take on all testing methods. So I asked this question to gain an idea of what test methods to use and why should I use them. You can answer in the following format:
Test Method - what you use it on
e.g.
Unit Testing - I use it for ...(blah, blah)
Regression Testing - I use it for ...(blah, blah)
I was asked to engage into TDD and of course I had to research testing methods. But there is a whole plethora of them and I don't know what to use (because they all sound useful).
1. Unit Testing is used by developers to ensure unit code he wrote is correct. This is usually white box testing as well as some level of black box testing.
2. Regression Testing is a functional testing used by testers to ensure that new changes in system has not broken any of existing functionality
3. Functional testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. Functionality testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic
.
This Test-driven development and Feature Driven Development wiki articles will be of great help for you.
For TDD you need to follow following process:
Document feature (or use case) that
you need to implement or enhance
in your application that
currently does not exists.
Write set of functional test
cases that can ensure above
feature (from step 1) works. You may need to
write multiple test cases for
above feature to test all different
possible work flows.
Write code to implement above feature (from step 1).
Test this code using test cases you
had written earlier (in step 2). The actual
testing can be manual but I would recommend to create automated tests
if possible.
If all test cases pass, you are good to
go. If not, you need to update code (go back to step 3)
so as to make the test case pass.
TDD is to ensure that functional test cases which were written before you coded should work and does not matter how code was implemented.
There is no "right" or "wrong" in testing. Testing is an art and what you should choose and how well it works out for you depends a lot from project to project and your experience.
But as a professional Test Expert my suggestion is that you have a healthy mix of automated and manual testing.
(Examples below are in PHP but you can easily find the correct examples for what ever langauge/framework you are using)
AUTOMATED TESTING
Unit Testing
Use PHPUnit to test your classes, functions and interaction between them.
http://phpunit.sourceforge.net/
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.webinade.com/web-development/functional-testing-in-php-using-selenium-ide
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
This answer is (almost) identical to one that I gave to another question. Check out that question since it had some other good answers that might help you.
How can we decide which testing method can be used?
I usually do the following things:
Page consistency in case of multi-page web sties.
Testing the database connections.
Testing the functionalities that can be affected by the change I just made.
I test functions with sample input to make sure they work fine (especially those that are algorithm-like).
In some cases I implement features very simply hard-coding most of the settings then implement the settings later, testing after implementing every setting.
Most of these apply to applications, too.
Well before going to the answer i would like to clear testing concept about multiple methods.
There are six main testing types which cover all most all testing methods.
Black Box Testing
White Box Testing
Grey Box Testing
Functional Testing
Integration Testing
Usability Testing
Almost all Testing methods lies under these types, you can also use some testing method in multiple types like you can use Smoke testing in black box or white box approach on the basis of resources available to test.
So for testing a web site completely you need to use at least following testing methods on the basis of resources available to test. These are at least methods which should be used to test a web site, but there may be some more imp methods on the basic of nature of website.
Requirement Testing
Smock Testing
System Testing
Integration Testing
Regression Testing
Security Testing
Performance & Load Testing
Deployment Testing
You should at least use all of above (8) testing methods to test a web site no matter what testing type you are focusing. You can automate you test in some areas and you can do this manually it all depends upon the resources availability.
There is specifically no hard and fast rule to follow any testing type or any method. As you know "Testing Is An ART" so art don't have rules or boundaries. Its totally up to you What you use to test and how.......
Hope you got the answer of question.
Selenium is very good for testing websites.
The answer depends on the Web framework used (if any). Django for example has built-in testing functions.
For PHP (or functional web testing), SimpleTest is pretty good and well... simple. It support Unit Testing (PHP only) and Web Testing. Tests can run in the IDE (Eclipse), or in the browser (meaning on your server).
The other answers posted so far focus on unit/functional/performance/etc. testing, and they are all reasonable.
However, one the key questions you should ask is, "how effective is my testing?".
This is often answered with test coverage tools, that determine which parts of your application actually get exercised by some set of tests. The ideal test coverage tool lets you test your application by any method you can imagine (including all the standard answers above) and will then report what part and what percentage of your code was exercised. Most importantly, it will tell you what code you did not exercise. You can then inspect that code and decide if more testing is warranted, or if you don't care. If the untested code has to do with "disk full error handling" and you belive that 1TB disks are common, you might decide to ignore that. If the untested code is the input validation logic leading to SQL queries, you might decide that you must test that logic to ensure that no SQL injection attacks can occur.
What test coverage tools let you do it to make a rational decision that you have tested adequately, using data about what parts of your code has been exercised. So regardless of how you test, best practices indicates you should also do test coverage analysis.
Test coverage tools can be obtained from a variety of sources. SD provides a family of test coverage tools that handle C, C++, Java, C#, PHP and COBOL, all of which are used to support web site testing in various ways.