Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
As a developer I often release different versions of applications that I want tested by users to identify bugs and to confirm requirements are being met.
I give the users a rough idea of what I have changed or new features that need testing, but this seems a bit slap-dash and not very well strucutured.
I'd like to know what approaches or procedures others take when asking for UAT during iterative development.
Thanks.
I find that writing test scripts is increadibly time consuming, often longer than the time taken to put the fix into place. With the large volume of work we do here we just don't have the time to create effective testing scripts.
With our changes we push the testing through two levels, applicaiton support and business acceptance. It is our hope that with a technical approach and a business approach that most of the aspects of the change will be tested. To let them know what they should test we attach a list of actions that have been effected by the change (Adding a product, Removing a product, Editing a product).
This coupled with a strong unit testing approach is the best approach to a high volume environment in my opinion.
User Stories or Use Cases might be what you are looking for, how did you decide on the change in the first place and how did you specify it. If you write up a little story, or bigger a actual structured use case you can use it as the specification for your change and then the users can test against that story to see whether the implementation matches the description.
Generally I create a script in excel with each feature list and an "Expected Result" and "Actual Result" column, with the Expected Result column filled out with what should transpire. For my own use I include a column that is the id of the item. This corresponds with the Task Id from Team System or the WSB from the project plan created
You're seeking an efficient and effective way to conduct UAT in a structured manner. I highly recommend using a pairwise or combinatorial test design approach. I have used this approach in more than 2 dozen proof of concept projects and found that, as compared to traditional methods of identifying test cases manually, this approach consistently leads to dramatically more defects being found per tester hour. In fact, on average, as reported in a recent IEEE Computer article I co-wrote, we found 2.4 X as many defects per tester hour on average.
The approach is described in the video here. Apologies if this appears to be an "use my tool" plug. I don't mean it to be. It is the approach that will deliver dramatic benefits, not the specific tool you choose to use to design your tests. James Bach also offers a free tool called AllPairs on his satisfice.com site. My point is that using any such tool will generate dramatically superior results because these tools are designed to generate maximum coverage in a minimum number of tests. They avoid repetition; in addition, they automatically identify and close potential gaps in coverage that manual test case identification methods will fail to close.
While it might be counter-intuitive that a tool like Hexawise would be able to identify (in seconds) the UAT test cases that should be run better than testers would be able to identify and document (in days), it is nevertheless true. Try it for yourself. Have one UAT tester on your team execute 20 end-to-end "black box" or "gray box" tests that are created with Hexawise and have other testers test what they usually would. I would bet good money that the tester executing the 20 Hexawise tests would find many more defects per tester hour (and would find "important" as well as "unimportant" defects).
It is a shame that these kinds of methods aren't much better known in the testing community outside of a relatively sophisticated group of testers who take the time to read books like Lee Copeland's book on test design methods. Pairwise and combinatorial methods work consistently, they deliver enormous improvements in efficiency and effectiveness, and they are quite easy for testing teams to start using immediately.
Justin (Founder of Hexawise)
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I'm the lead developer on a new project and have the opportunity to work with the system engineers to create our template for testing functional requirements. I was wondering if anyone had input on what makes a good test procedure template or had an example of a great template.
Thanks!
This isn't a very easy one to answer. It depends on a few things:
1) The definition/interpretation of what is a functional test case
2) The role of the support staff in the acceptance tests
3) The longevity of the tests
This is purely opinion based on my own experiences.
(inserts two cents into vending machine)
1) What is a functional test case? - You and the systems engineer need to align on this one. You may find (as I did) that the system engineer will tackle things at a higher (less granular) level than you. For example, assuming that a specific requirement is for the creation of a web service, the engineer would need to know:
is the interface behave correctly?
Are the input parameters in a test case meant to induce a success/failure?
On failure, are the appropriate errors/error codes returned? Note that depending on their time, an engineer may only stick to major/important failure conditions (or negative responses) that affect the product/service as a whole (for example a "host not found/timeout error" should be in the interface but does not necessarily need to be tested, but a use-case related failure such as "client has insufficient funds" is important to the engineer.
is the transaction status recorded correctly?
Again, you and the systems engineer should be clear on what is a functional test case and what is not. Usually the functional tests are derived directly from the functional spec provided to you. For certain products, retry on timeout falls under non-functional, but you may have an engineer who wants his web service to retry 17 times on a timeout before giving up - and if he specifies this - then you include it.
2) How are these tests carried out and who signs them off? Depending on this, you may need to streamline or flesh out the functional tests.
If yourself and the systems engineer will lock yourselves up into a cosy room for half a day going through each test case, then keep it streamlined: the two of you should be pretty familiar with the requirements and the engineer would have reviewed the document and have provided comment already. On the other hand, you may have the support engineers running the tests with you instead of the engineer (that's how we run it...the systems engineer reviews the test cases, stays for a bit at the beginning and leaves when he gets bored). Where was I? Right, so in this case, your document may have to go into a bit more hand-holding where you describe the scenario that is being tested. This leads me to the last point in my long winded chat...
3) Longevity of the document
So often as is the case on my side, once a set of functional tests are over and done with, they are promptly forgotten. However, these tests validate your system and your product and the support engineers should be in the position to run them whenever they'd like to :
resolve issues ("was this sort of case even tested before go-live?")
resolve issues again ("geez did these guys even test this particular scenario?")
validate system/product integrity after a major change
learn about the as-is functionality of a product or service (so many times people forget how the product is supposed to behave, and support staff hate reading requirements specs especially the requirements specs that are out of date and the current behaviour of the system differs from what was originally specced)
(deep breath)
So now you need to make sure you cover the following:
Test set up part 1: what are the requirements to run the test? What tools do i need? network connectivity?
Test set up part 2: what test-data am i going to use? where is it if I need it or how do i generate it?
Overview of the functional requirements/tests to at least impart what the expected behaviour is.
Overview of the major system components that will be tested
An idea of the limitations of the tests - certain functional tests may only be simulated or could not be tested against a live end system etc etc - you need to decribe the limitation and show the reader how you're gonna fake it.
Also, the systems engineer will expect you to have already completed your granular tests like component tests, integration tests, etc etc as well. Depending on how awesome he is, the engineer may ask for the documentation of these component tests and run a few himself.
Hope that helps somewhat - having a template provides consistent presentation and helps you ensure that all the important content is covered - but I think the focus should be on pinning the purpose and fulfilling that purpose.
Hope I made some cents :)
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
My understanding is that it's advised testers are separate from developers, i.e you obviously have developers testing their code but then dedicated testers as well.
How does that actually work in practice on a small project, say 5 developers people or less? It seems unlikely you could keep a tester occupied full-time, and while you could bring in random short-term people I'd argue a tester should understand the app well - its intended usage, its users, its peculiarities - just like you don't want developers to be transient on the project.
You can definitely keep a tester working full time - they should be testing the product throughout the development process, not just at the end. In fact leaving testing to the end of a project is absolutely the worst thing you can do.
I have worked in a couple of companies that have typically 1 tester for every 2 developers, and there has never been an issue with them running out of things to do - in fact quite the opposite.
Both of these have been small companies with 10-20 developers and 5-10 testers.
In a small company, this is difficult because you're right: you can't just have the testers sitting idle between rounds of formal testing. Sure, they could do other things like write test cases and test plans, but even then they may have some idle time. For a small company, it might make sense to hire testers on contract when they are needed, as you might only have one product for them to test and the time between products is large. You might also look to see if you can find another company that will do the testing for you - similar to hiring contractors, but the contract would be with the parent company not the individuals.
In larger companies, there are usually (but not always) enough projects at different stages of development/testing going to keep all of full-time testers mostly occupied with work of some sort. Of course sometimes the demand exceeds the resources on hand (full-time testing staff) so contractors are sometimes brought in for a specific project. And yes, you're correct, even the contractors need to be trained to the system they are testing, even if they are ony there for the one project.
You can ask developers to test each other's parts but in general it's not a good idea and a separate tester will be the best way to go.
Another option is to find a 3rd-party company that will test the application for you. This will also force you to have a better spec on the project.
I work in a small team environment, with only rarely more than 1-2 developers on any given project. We do not have, nor could I realistically see having, a dedicated tester. Usually, I involve my customers doing the QA testing of the application in a staging environment prior to putting any release into production. This is more or less successful depending on the customer's buy in to the testing process. I also rely heavily on automated unit tests, using TDD, and significant hand testing of the UI.
While I would like to have people with specific QA test responsibilities, and sometimes my customer will designate someone as such, this rarely happens. When I do have a dedicated tester (almost always a customer representative) who is engaged in the process I feel that the entire development process proceeds better.
It's important in situations like this to utilize formalized test plans, and find whatever non-developer resources you can for testing. Often the Technical Architect or Project Manager will need to author Acceptance Criteria or full on Test Plans for new functionality, as well as test plans for regression testing. Try to get users, project managers, any stakeholders who are willing to help you test. But give them structure to ensure that all necessary test cases are reviewed.
An outside QA engineer could be very helpful in helping you architect the test plan(s), even if he/she is not doing all the testing.
Good luck
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
This question is marked as a community wiki, and is subjective, but please don't close it, I think its a good question, and I would like to know what the development community have to say about testing.
I've been a developer for over 10 years, and I've yet to work in a company that has a dedicated testing department. Over the years I've seen the attitude towards testing get steadily worse, lately management are after quick results, and quick deployment, and there are lots of teams out there that simply forget the science of development, and omit serious testing.
The end result is - management is satisfied with the speed of development initially, the app might even run stable in production for a while, but after that something is bound to snap. Depending on the complexity of the app, a lot could go wrong, and sometimes all at once. In most cases, these issues are environment driven making them hard to isolate and fix. The client is the entity who is ultimately taking on the role as stress testing, because like it or not, someone eventually HAS to test the app.
During this phase, management feels let down by the developer. The developer feels management didn't listen in the first place to the pleas for significant testing, and the customer looses faith in the software. Once order is eventually restored, if the product survives this. The developer is ultimately the one who gets blamed for not outputting a stable product, and for now going way over budget in man days, because the developer spent 2-3 times more on testing the app (eventually).
Is this view point realistic? Does anyone else feel this strain? Should developers be taking professional courses in testing? Why is testing being left behind? Or is this just my bad fortune to have had this experience over the last 10 years of my career.
Any thoughts welcome. Please don't close the question.
In my opinion developers should never test, since they test "does it work?".
A test engineer on the other hand, tests if something "does not work", which is a very important difference in my opinion.
So let other people do the testing, test engineers preferably or otherwise functional analysts, support engineers, project managers, etc...
Personally, everything I write is unit-tested if it has any significance. Once it passes that kind of testing, I usually pass it on to friends and ask them to use it. It's always the end-user who does some sort of unexpected action which breaks things, or finds that the interface you designed which was oh-so-intuitive to you is really quite complex.
Many managers really do need to focus more on testing. I personally am appalled at some of the code goes out the door without proper testing. In fact, I can think of multiple applications I use from various companies that could've used a nice unit test, let alone usability testing.
I supposed for companies it boils down to, does it cost less to have dedicated people for testing, or to fix the inevitable problems later and get a product out the door?
The last two companies I have worked for had dedicated professional testers who do both manual testing and write automated test scripts. The testers did not simply test the product at the end of the development cycle (when it is usually too late to make significant changes) but were involved from the beginning converting requirements into test cases and testing each feature as it was developed. The testers were not a separate department, but an integral part of the development teams and worked with the programmers on a daily basis.
The difference between this and the companies I have worked at without dedicated testers is huge. Without the testers I think development at both companies would have ground to a halt long ago.
Unit testing is important too but developers test that the code does things right, not that it does the right thing.
I've only worked in one organization that had dedicated testers - and that was in 1983.
Use TDD and it won't be an issue - plus your development cycles will accelerate.
For example, this week I wrote 3 automated acceptance tests for a complex application. Manually performing these tests takes about 4 hours. The automated tests run in under 3 minutes. I ran the tests over 50 times today, shaking out bugs both small and large.
End result: the application is good to go to the end-users, and the team has high confidence in its capabilities. Plus the automated tests saved about 200 man-hours of manual testing just today. They'll save even more as regression tests as future enhancements are made.
Some people claim that TDD imposes extra overhead, which is true in only the most myopic of perspectives. Writing the test scripts took about 2 hours. Fixing the twenty bugs that they found took the rest of the work day. Without the tests, I'd still be doing manual testing trying to track down (at best!) the second bug.
Like so many others here (so far you have all been too ashamed to admit it) but I have users to test my software. I have read that this is not best practice, but I'm not sure that the management have.
In ours, we have dedicated testers. However, for the developer it is implied that he does his own informal testing first before submitting to the tester for a more formal testing.
In the company i work for:
The programmers tests everything => If it compiles keep it (as development is mostly done live so it's not necessary to push changes to live environment), if it doesn't fix it until it does. Oh, and unit tests are not used as they take up too much time.
Later Bugs are usually found by the users and/or the project manager who checks if the project looks ok but has too much to do to do in-depth testing.
I currently fix parts of projects that have never worked at all which haven't been noticed/reported for a year.
Developer perform unit testing.but unit testing is just not enough for application.Because developer never accept their faults and they protect their own code. SO If you want to deliver a good quality of product let the QA team to test the application . They test the application from user's perspective which helps organization to deliver good application.
In my company, we have dedicated testers. I am one of the testers.
What I can feel and think is the Developer focuses on making sure that what they have done (with the code) is tested and working OK. But from Tester's point of view, they are trying to find bugs - so the testing is for defect identification.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
What do you do when you join a team that says they use Scrum, but only use it as a time-management tool and not the whole process?
How can I reinstate back testing and documentation?
I was thinking to start off with adding user stories specifically for testing and documenting.
Perhaps someone else has more experience with this then I do about this as I am sure its not that uncommon.
The key to scrum is that a task be identifiable as "done" before it can be classed as done. How does you company assess whether something is done without reviewing documentation and tests?
Perhaps they have an unusual, but valid, way of doing it. Or perhaps they have missed the point of "done tasks". I'd suggest you start by asking them how they measure down and whether it could be improved. Then suggest documentation and testing as the way of improving the process.
Note that neither testing nor documentation are in fact part of Scrum. Scrum is a pure project management approach - the required engineering practices, like the ones you mention, are supposed to "emerge" during the project. And most specifically, they are supposed to be identified during the heartbeat retrospectives that you do at the end of every sprint. Are you doing those? Can you bring up your concerns there - and are they actually the biggest concerns the team has?
Is the issue that they don't have any documentation and tests, or that they aren't implementing the entire Scrum methodology? Those are 2 very different problems in my mind.
I would much prefer an organization that has taken the time and effort to find and fit a development process that matches their development style as opposed to mandating down from on high the one true process. So I would not be concerned at all if they were using a process that they called Scrum but that didn't meet all the "official" guidelines. Try to determine why the process is the way it is. Chances are that if they have taken the time to tailor it, the team will be receptive to your ideas, especially if you have taken the time to determine why things are the way they are. If you simply approach it as "this isn't Scrum and so isn't right", you will probably not make much headway, but by being pragmatic about the benefits you can likely make some substantial improvements.
Alternatively, if they aren't doing testing and don't have any documentation I would consider that a fairly bad sign. And by documentation I am taking the minimalist view here - a list of features, bug tracking, etc. - I would be very concerned by the absence of these items, less concerned by the absence of items higher up the abstraction list. In the absence of support from management, I would suggest you lead by example. Take it on yourself to setup a simple bug tracking system (there are several - in a pinch, simple text lists in a central location work as well). Don't declare your features complete until someone else has tested it. This can be as simple as walking over to another developer and asking them to try it in front of you. If someone claims a feature is complete, take a few minutes to familiarize yourself with it. If you find a bug, politely mention it to the responsible developer. Slowly build an environment where the team can see the benefits of running tests and tracking features and bugs.
Most teams operate in this manner simply because of a mistaken belief that they don't have time to "do it right", or that they will get to it later. Often this will occur when a simple proof-of-concept done by a developer or two as a side-project turns into a full-on development effort. By showing that it can actually save time and effort, and reducing the initial costs to the rest of the team, you will often find that it becomes ingrained as part of the process without ever actually being officially endorsed or accepted.
If you have management support it will make it much easier, but always be careful to make sure that the team is receptive to the changes. This may mean it takes longer than you want, but so be it, without the team's support any mandated process will fail at the first sign of pressure, which is when you need the process the most.
*Disclaimer - On my last project I spearheaded the movement to tailor the SCRUM process to fit our environment. The "official" process was simply untenable for our client, but it was still an invaluable guide in tailoring our process.
"adding user stories specifically for testing and documenting"
While meta-user stories might make sense in some circles, it rarely works out well. Software folks rarely cope well with meta-user stories, they either don't get the idea that they can change their own processes by writing a story, or -- more typically -- they engineer the meta-user story to death.
When you're interviewing users, it feels like they're making the user story up. Certainly, you're making it up as you listen to them and try to capture it.
When an IT organization tries to make up its own user stories about how IT should work, the process falls apart. Until the organization has done the thing (testing, for example) a bunch of times manually, they're not really qualified to write user stories. Then, after they've done it, they don't need software development processes, they'll just automate the important bits a little at a time.
I think change has to come from a less formal direction. Actually balking at calling something "done" that hasn't been tested is a good starting point.
IT doesn't do things unless forced. So, meet the users and find out why they're not requiring testing. Coach them to require testing. Tell them the consequences and the words to use.
A lot can go wrong in an organization to lead to poor processes. It's important to know what's wrong, and create a demand for change. The best possible thing is to have your boss complaining that you're not fixing it, rather than you suggesting that perhaps it would be good to fix it.
[It doesn't feel right when your boss demands you fix the process, but it's about the only way change will happen.]
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 5 years ago.
Improve this question
If you had to do without one or the other in a software project, which would you pick?
I've had plenty of projects in which the client or PM thought they could get away without one or the other. We always paid the price.
Turn this around and repeat after me: "Tests are requirements." :-)
If you mean "formal requirements", I can and easily do without those. I would much prefer a living, breathing customer who can tell me what they want over a rigid, out-of-date document. Having switched to TDD, I wouldn't ever want to go back to a "no test" environment. I choose informal requirements -- stories, on-site customer, and customer-written acceptance tests -- over formal requirements and no tests.
I'd say you could go without Testing rather than Requirements. If you don't have requirements, how do you know what you're developing?
If the programmers are good enough, they should be able to catch most of the egregious errors that testing would find.
You have to test against the requirements, so if you don't have requirements you can't do testing. So if you have to pick one, you can only pick requirements.
But not doing testing is a path to failure. Guaranteed.
If I had to pick one, it would be requirements.
It doesn't have to be a formal, excruciatingly detailed document with twenty signatures, but you have to know exactly what the customer wants and more importantly what the customer needs.
The requirements are also your first communication to the development team. How will they know what you're asking if you're not asking it clearly? At best you're at grave risk of building the wrong thing right. I'd rather have the right thing built slightly wrong.
If I were asked to choose between requirements or testing I would choose to polish up my resume. You really can’t do without either in any projects because the basic project lifecycle is:
Define Needs/Goals (AKA Requirements)
Design & Build to the requirements
Verify that you built to spec (to requirements.)
If you dont have success criteria and goals that are verifiable (and then are verified) how can you insure that you are going to succeed? And if you dont have a chance to succeed, why start the project?
I would say requirements because there always seems to be some level of "feature creep" from the client when you are developing software. Testing is one of the crucial pieces in the SDLC.
Requirements and testing are important for most projects but if you really have to pick, you should go with requirements. One of the advantages of picking requirements over testing is that, you might save some development time since the developers know what they have to build, and if the development is done with extra time in hand, you can allocate that time for testing :)
tests (feature and integration) are more important than requirements; if you can specify the tests then you have also specified the requirements, at least implictly
comments are also the developer documentation, with unit tests being the how-to 'quickstart' examples ;-)
Not sure if the requirements are referred to as an artefact or as a process. Although it is possible to skip requirements as artefact especially for smaller teams and still deliver a product, skipping requirements as process is out of question. Requirements as artefact let you model the system at cost lower than building the entire thing, do feasibility, estimates, and for a larger and more disperse team to cut communication overheads and have a common ground under the feet. Neglect the requirements and you get louse estimates (regardless if you plan a lot up front or just do a short sprint), poor idea of feasibility and possibly very inefficient communication and a lot of miscommunication.
Requirements as a process on the other hand is going to exist regardless if it is formally acknowledged or not. You cannot really exclude it, you can pretend requirements process does not exist or integrate into the design, coding, testing or into stages as late as pilot and maintenance. Obviously treating the process in this way mean it will not get fair amount of attention and resource. Consequences normally range from delivering something that is ultimately useless to having to fix the now obvious shortcomings of the product later in the development cycle or even discovering the real requirements once the product fails in the field, increasing the cost of development, defaulting on the deadlines, ruining team’s good name, destroying user confidence etc.
Testing usually boils down to validation and verification, more recently testing technology improvements let automated testing to be used as a solid tool for achieving greater efficiency in debugging and reducing time necessary for regression testing. Validation is making sure that the team has built the right product, i.e. scoped requirements are correct, not contradictory and there are no gaps. Verification on the other hand is making sure that the product is built right: no technical defects, accidental errors etc.
As we can see testing provides a safety net in the scenario where requirements were neglected. Normally as the team starts testing they need to refine their understanding of requirements and as a result modify the software. Since both requirement artefacts and software itself just represent different levels of fidelity in modelling a solution for a real life problem, and software as a model is order of magnitude more precise the testing of application evaluates requirements as well (regardless if they are implicit or explicit, formally analysed or informally communicated).
Normally the alternative to testing is to let users report a substantially larger amount of defects and shortcomings and try and fix them as part of maintenance (meaning later in product lifecycle), increasing the cost of every fix.
So requirements versus testing? Fire the manager. Ok, skip requirements if you want the project schedule slip during the testing phase and get yourself into the mess of building not what users need, skip the testing if you just need to show utter disrespect to your users.
Without requirements you don't need testing since what you end up with is exactly what was spec'd
There are categories of software that can be developed perfectly well without requirements, at least anything more than a vaguely expressed idea the length of an email.
Thing is, if you have a specific client, and a project manager, it is unlikely your software is in one of them. It's unlikely someone is specifically paying you to, say, 'make me a fun game involving a juggling monkey'.
The only category of software that can be developed without testing is failware: where your company has managed to sucker some customer into paying whether or not the software works (or if you have a really dumb customer, pay more if it doesn't work, in support and maintenance).
That's probably more likely: contracts structured so that success is less profitable than failure are still fairly common. If you think that's the case, and you want to develop working software, then consider switching to a job where your interests and your bosses are less opposed.
Without Requirements can we make a Test Plan? So We Cant do Testing even if we pick Testing instead of Requirements.
So Requirements should be Priority even if you consider Agile Testing Environment.