Is there a method for tracking or measuring the cause of bugs which won't result in unintended consequences from development team members? We recently added the ability to assign the cause of a bug in our tracking system. Examples of causes include: bad code, missed code, incomplete requirements, missing requirements, incomplete testing etc. I was not a proponent of this as I could see it leading to unintended behaviors from the dev team. To date this field has been hidden from team members and not actively used.
Now we are in the middle of a project where we have a larger than normal number of bugs and this type of information would be good to have in order to better understand where we went wrong and where we can make improvements in the future (or adjustments now). In order to get good data on the cause of the bugs we would need to open this field up for input by dev and qa team members and I'm worried that will drive bad behaviors. For example people may not want to fix a defect they didn't create because they'll feel it reflects poorly on their performance, or people might waste time arguing over the classification of a defect for similar reasons.
Has anyone found a mechanism to do this type of tracking without driving bad behaviors? Is it possible to expect useful data from team members if we explain to the team the reasoning behind the data (not to drive individual performance metrics, but project success metrics)? Is there another, better way to do this type of thing (a more ad-hoc post mortem or open discussion on the issues perhaps)?
A lot of version control packages have things like svn blame. This is not a direct metric for tracking a bug, but it can tell you who checked in changes to a release that has a major bug in it.
There's also programs like http://www.bugzilla.org/ that help track things over time.
But as far as really digging into why bugs exist, yes, it's definitely worth looking into, though I can't give a standard metric for collecting that information. There are a number of reasons why a system might be very buggy:
Poorly written specs
Rushed timelines
Low-skill programming
Bad morale
Lack of beta or QA testing
Lack of preparing software so that it is even feasible to beta or QA test
Poor ratio of time spent cleaning up bugs vs getting new functionality out
Poor ratio of time spent making bug-free enhancements vs getting functionality out
An exceeding complex system that is easy to break
A changing environment that is outside the code base, such as the machine administration
Blame for mistakes affecting programmer compensation or promotion
That's just to name a few... If too many bugs is a big problem, then management and lead programmers and any other stake-holders in the whole process need to sit down and discuss the issue.
High bug rates can be a symptom of a schedule which is too rushed or inflexible. Switching to a zero defect approach may help. Fix all bugs before working on new code.
Assigning reasons is a good technique to see if you have a problem area. Typical metrics I have seen and encountered are an even split between:
Specification errors (missing, incorrec, etc.)
Application bugs (inccorrect code, missing code, bad data, etc.)
Incorrect tests / no error (generally incorrect expectations, or specifications not yet implemented)
Reveiwing and verifying the defect causes can be useful.
Related
I am trying to understand a testing theory and few times I have seen following, which is related to priority:
1. Critical : Bugs at this level must be resolved as soon as possible. Testing
should not progress until a known critical defect is fixed.
or
1.Immediate fix, blocks further testing, very visible
But to be honest, I dont know why. Why could I not test e.g. another part of the system until the critical-priority bug has been fixed?
I expect the reason behind this design is that these bugs intend to mark massive interdependencies of the critical parts with other parts of your system.
For example, if you have an application that deals with money pervasively, you may have a test at the base of your system that determines whether currencies are represented accurately. If they aren't, and you keep testing other parts of your system, you may find other numerical tests that report wrong results elsewhere. Unfortunately, you may also waste time looking for an error in your math or your algorithm, and spend time fixing those numerical tests while, in fact, they're doing the Right thing with the wrong currency.
If it's possible to test something else, then the bug is not priority/severity 1. Simple as that. :-) Priority 1 bugs tend to be things like "code does not compile" or "the system crashes at bootup and there is no workaround".
Scenario:
Currently we are in the process of system and integration testing. Every day we get lots of defect raised by testers. Most of these defects do not match with requirement we were given. Lots of scenarios are new to developers. Requirement we had was signed off by business.
Could someone clarify how to distinguish between Defect vs CR?
Everything that was not a requirement is a change-request.
But live is unfortunately not that easy, so please read on.
Quarrels on what is a defect and what is a change-request are very common in projects. Managing the situation is difficult because you often have to make compromises.
I have seen project managers being removed by programme managers because they insisted to much that all the defects are really change requests. They often were right but still there behavior was not helpful for the overall progress of the programme. I have also seen project managers who killed themselves by accepting every defect and built castles though never originally required and effort estimated for.
I personally always make absolutely sure that my managers know that I am building features not originally required that came in under the disguise of defects. I also make sure the client/tester knows that this is my viewpoint. But also I am very tolerant in my consideration what a defect is.
Example: I recently joint a project where we developed a financial payments system and another programmer said to me "It is outrages what they want have that is not a defect this is a CR!". I looked at it and due to my background in this business domain I thought it actually is very fundamental requirement and asking for a CR for this is really laughable. So I decided we fix it without making a fuss about it.
Also the following questions are worth to consider:
Are you in a fix price project? Do you still have resources and show real greatness by adding features without moaning that will give you a good reputation and a future contract?
Do you get penalized if you accept a CR as a defect? Is a low-number of defects a KPI (Key Performance Indicator) and affecting your career?
Was the requirement definition poor at the beginning and you accepted it? Was the requirement mentioned in the defect real obvious and could be considered implied? E.g. never specified that amount field should only allow numeric values but still it makes sense.
Have you accepted requirements without asking about the whole big picture and are partially responsible?
Is the client ripping you off and exploiting your inability to say no and reject the defect?
In projects I always try to get the best for the client but make sure I am not being penalized undue.
I have a web site I am building for a client. I now have a tester on the project with me.
I feel testers are needed. REALLY! I cannot test my own code. I also appreciate the value of a new set of eyes. But what desires reporting?
It is easy to say everything should be reported, but I don't have someone between me and the tester to filter out the unimportant requests. The tester does not know the system nor the target user well. She is assigning me tasks and not the project manager. I think this will change soon, but until it does, what do you recommend? There seems to be a believe that our users have NEVER used the interent before at all, and they are as dumb as rocks.
The problem I am having is that EVERYTHING the tester suggests is being accepted automattically and assigned to me.
I have many cases that make me drop my jaw and say "Really? Are you serious? This deserves to be a issue?"
Ex: Need to add text at top of page that says "* = Required" for required fields.
Have you ever felt this way? How did you deal with it?
For now, I am just doing as I am told, but I am making it clear I do not agree.
It sounds to me like your tester is doing the right thing. You can't assume any level of user expertise when testing an application. If a user can break something, they will.
You and your tester need to work out a severity scale. The outliers (those that anybody with Internet experience could probably work around/would never hit) would be considered low priority and sit on the back burner until you knock out the high priority items.
...never the less, those outliers should still be logged because they can definitely come back to bite you in the ass in the end.
You need to add priorities for your issues. This will allow you to do the important issues first, and cosmetic issues last. Here is example priorities from Jira:
Priority 1 - a reproducible crash; issue blocking any further testing or development of a specific feature; loss of user's persistent data; huge memory leak
Priority 2 - a major issue that must be fixed before the product is released; prevents users from using a feature; negatively affects partner; significant memory leak in frequently used functionality
Priority 3 - a minor issue that should be fixed before a product is released; does not prevent users from using a product; highly visible usability issue; small memory leak in rarely used functionality
Priority 4 - a purely cosmetic issue; doesn't affect functionality
Actually it sounds like your tester is doing the right thing (and the text for "* = required" is a very good idea).
In addition to the suggestions about prioritizing reports, I would suggest that you categorize the reports as to whether they refer to user experience or functionality.
You and the tester will never exactly agree on what "needs" to be reported. Just set the priority on issues correctly, and get on with fixing the high-priority stuff first.
One thing you absolutely do not want to do is to discourage the tester from filing bugs. That'll come back to bite you when something ships totally broken, and they say "I thought that was just how it worked".
Do make sure that you're communicating the development schedule and status properly, so they don't waste time testing features that aren't sufficiently complete.
I would report to the client what each change will cost in terms of time and money. Things that are legitimate bugs you'll probably need to fix on your own time (unless your contract says otherwise). Things that are design / subjective issues you should be able to assign a cost to. Let the client know what it is going to cost them and they can decide if they want to proceed or not.
Hopefully you've got some sort of a project specification that the client has signed off on so that you know when the project is complete and what sort of things are not included in the project scope. If not, you might have a bit of a fight on your hands. For changes that you think are outside of the project scope, you might need to compromise - maybe bill them at a cheaper rate or split the cost with them. If you're in that situation it's a good learning experience to get everything documented in the project specification so that there is no question about what falls outside of the project scope. I've been there - one experience like this is enough to teach you to put more work into your specifications.
Report everything and triage. After a bit of time, she'll start to understand what gets past triage and what doesn't. Humans can learn; teach.
On occasion I've heard people discuss the benefits of keeping track of programming mistakes, if for no other reason than it increases awareness of common errors. I've started to keep a list of bugs that I find in my code, along with what could have led to them. The main question I have is this:
What information related to my mistakes should I be keeping
track of so that I can improve as a
programmer?
And a couple other questions related to this:
How do I use this information once I start logging my mistakes?
Is tracking mistakes truly beneficial?
This is only useful if you are actually vigilant with tracking and reviewing. When I was working on a team, no matter how much documented that for example our servers in the production environment were natted and would not be able to resolve their own domain names or public IP addresses, every 6 months, I'd get a call at 4 AM from the deployment team or dev team that a new developer was responsible for, and they either forgot or were unaware.
I remember a particular engineer who was repsonsible for deploying and he had paper checklists, we built him deployment tools that forced him to record his checklist, yet he would always forgot to set the connection string (resulting in the 4 am phone call). Point is it's only worth it if your going to use it vigilantly.
I've found the best way to use this is by implementing your rules into a code analyzer like fxcop.
I think what is more useful than keeping a log of individual mistakes is making sure you come away with a real understanding of why it was a mistake in the first place. Most mistakes stem from a lack of understanding about one thing or another, correct that understanding and you eliminate an entire set of potential mistakes in the future. If I would log anything it would be what I learned from the experience that I didn't know before, not the specifics of the mistake that was made which is likely to be of limited usefulness when you look back at it later.
what was the mistake
how can it be avoided
add the latter to an appropriate checklist, and refer to it as often as appropriate
I think tracking mistakes can be worthwhile, but in my experience it helps a lot to categorize them at some level.
Every programmer is going to make enough mistakes over the course of their career to fill an encyclopaedia. If you make a huge checklist out of all of them then you're never going to get any coding done because you'll eventually be spending all of your time going over your checklist. So: categorize your mistakes in some way that makes sense to you so you can rifle through your list looking at the most important mistakes for the sort of code you're currently working on.
Also, to add to the above as far as what to collect:
what are the symptoms of the mistake (so you can find it later)
how you actually solved it
Yes, tracking your personal mistakes is beneficial. Refer to the SEI for numerous data points (here's one at random). One such methodology is the Personal Software Process (PSP). It's too long to go into here, but here's a book about it. There's also this free SEI publication on PSP.
If you balk at SEI and think Agile is the way to go, you'll probably get better mileage out of a book like Clean Code: A Handbook of Agile Software Craftsmanship (publisher website).
Bottom line: disciplined developer = good, undisciplined developer = bad.
I would also want to ask the question of how much time would be required to accurately track the mistakes, and if that time could be better spent directly on improving the software instead. If you can do this in a minimal amount of time and are able to refer back to your records to prevent future mistakes, it may be valuable. In the long run though, I think it will be better to stick with an absolutely high level list of common mistakes you make.
I'd look for trends or similar types of errors that tend to recur. Once you're aware of the types of errors you make, you can be alert for them, or you can change your coding style to avoid them.
As an example, I worked very closely with my office-mate in a previous life. At one point I was halfway through my first sentence explaining some odd behavior, and he interrupted with, "increment your counter." I still double-check my loop counters today!
Instead of keeping a log of future mistakes, maybe you could review your bugtracker history, and try to find common types of errors which keep ocurring.
I'm so inspired I think I'll try this tommorow.
"What information related to my mistakes should I be keeping track of so that I can improve as a programmer?"
Read other people blogs. Write your own. You don't have to publish it. But you do have to turn each mistake into a story.
Don't drown this in metadata. This isn't amenable to a database or even a bug tracker. A mistake is a story where someone made a bad choice and recovered from it.
Here's your outline.
Context. What was going on.
Situation. The problem you were trying to solve.
What you did. Why it was bad.
What you should have done. Why it was better.
What changed. What you learned.
You should also record your successes in almost the same form.
Context. What was going on.
Situation. The problem you were trying to solve.
What you did. Why it was good.
What you learned.
When in doubt watch a movie. Seriously. Characters a confronted with choices, make mistakes and recover from those mistakes. That story line is the essence of drama. A mistake is the same thing.
Be careful what you log. Not every bad situation is a mistake. Some situations are inevitable or so far outside your control that nothing can be done about it.
Bad planning on someone else's part which puts you into panic-mode isn't your mistake. You can meticulously log everyone else's mistakes, but what's the point? If you're tracking what other people are doing, you should seek therapy.
Bad planning which provides inadequate budget, schedule or communication about changes may be a compound problem.
You didn't provide enough planning information up front.
In the scramble to cope, you made another mistake which lead to problem resolution and recovery.
In hindsight (which is always 20/20) you can see a decision which was wrong. However, at the time, it was based on best available information. That's not a mistake. Any sentence that begins "If we'd known that X..." is useless for mistake analysis. You can try to write a checklist of all the things you need to know next time you make that decision. But next time, you won't be making exactly that decision and some other piece of information will turn up missing.
Never call other people's actions mistakes.
Find the root causes. It's a root cause when it's something you can't control.
Don't retroactively call information that turned up missing a mistake.
I have been thinking the same thing. For the time being I keep a little spreadsheet with bugs I correct. The purpose of this is to make myself aware of the more common errors being made.
Hopefully I will start to see patterns and avoid making common errors over and over again.
I find that this makes my job more interesting, probably since I am an analytical person who likes to collect data and information in a structured manner.
You should try to relate the errors to use of inadequate tools, habits, methods or knowledge. In most cases you will find there would have been ways to catch the error earlier. Things like type-safety, unit-testing, code-review, coding standards, better API design, better documentation and working environment, comes to my mind.
I use JIRA with a custom workflow. For a bug, the last step before an issue is closed is a screen that allows me to select a "root cause". I've got a history of the root cause of every bug that's occurred in the last 3 systems I've built.
How useful, if at all, is for the testers on a product team to know about the internal code details of a product. This does not mean they need to know every line of code but a good idea of how the code is structured, what is the object model, how the various modules are inter-linked, what are the inter-dependencies between various features etc.? This can argubaly help them in finding related issues or defects once they hit one. On the other side, this can potentially 'bias' their "user-centric" approach towards evaluating and certifying the product and can effect the testing results in the end.
I have not heard of any specific model for such interaction. (Lets assume a product that users, potentially non-technical consume, and not a framework or API that the testers are testing - in the latter case the testers may need to understand the code to test that because the user is another programmer).
That entirely depends upon the type of testing being done.
For functional system testing, the testers can and probably should be oblivious to the details of the implementation -- if they know the details they may inadvertently account for that in their test strategy and not properly test the product.
For performance and scalability testing it's often helpful for the testers to have some high-level knowledge of the structure of the codebase, as it's beneficial in identifying potential performance hotspots, and therefore writing targetted test cases. The reason this is important is that generally performance testing is a broad open-ended process, so anything that can be done to focus the testing to get results is beneficial to everybody.
This sounds similiar to this previous question: Should QA test from a strictly black-box perspective?
I've never seen a circumstance where a tester who knew a lot about the internals of system was disadvantaged.
I would assert that there are self justifying myths that an informed tester is as adequate or even better than a deeply technical one because:
It allows project managers to use 'random or low quality resources' for testing. The 'as uninformed as the user myth'. If you want this type of testing - get some 'real' users to test your stuff.
Testers are still often seen as cheaper and less valuable than developers. The 'anybody can do blackbox testing myth'.
Development can defer proper testing to the test team. Two myths in one 'we don't need to train testers' and 'only the test team does testing' myths.
What you are looking at here is the difference between black box (no knowledge of the internals), white box (all knowledge) and grey box (some select knowledge).
The answer really depends on the purpose of the code. For integration heavy projects then where and how they communicate, even if it is entirely behind the scenes, allows testers to produce appropriate non-functional test cases.
These test cases are determining whether or not a component will gracefully handle the lack of availability of a dependency. It can also be used to identify performance related issues.
For example: As a tester if I know that the Web UI component defers a request to a orchestration service that does the real work then I can construct a scenario where the orchestration takes a long time (high load). If the user then performs another request (simulating user impatience) and the web service will receive a second request while the first is still going. If we continually repeat this the web service will eventually die from stress. Without knowing the underlying model it would not be easy to find the problem
In most cases for functionality testing then black box is preferred, as soon as you move towards non-functional or system integration then understanding the interactions can assist in ensuring appropriate test coverage.
Not all testers are skilled or comfortable working/understanding the component interactions or internals so it is on a per tester/per system basis on whether it is appropriate.
In almost all cases we start with black box and head towards white as the need sees.
A tester does not need to know internal details.
The application should be tested without any knowledge of interal structure, development problems, externals depenedncies.
If you encumber the tester with those additional info you push him into a certain testing scheme and the tester should never be pushed in a direction he should just test from a non coder view.
There are multiple testing methodologies that require code reviewing, and also those that don't.
The advantages to white-box testing (i.e. reading the code) is that you can tailor your testing to only test areas that you know (from reading the code) will fail.
Disadvantages include time wasted from actual testing to understand the code.
Black-box testing (i.e. not reading the code) can be just as good (or better?) at finding bugs than white-box.
Normally both types of testing can happen on one project, developers white-box unit testing, and testers black-box integration testing.
I prefer Black Box testing for final test regimes
In an ideal world...
Testers should know nothing about the internals of the code
They should know everything the customer will - i.e. have the documents/help required to use the system/application.(this definetly includes the API description/documents if it's some sort of code deliverable)
If the testers can't manage to find the defects with these limitations, you haven't documented your API/application enough.
If they are dedicated testers (Only thing they do) then I think they should know as little about the code as possible that they are attempting to test.
Too often they try to determine why its failing, that is the responsibility of the developer not the tester.
That said I think developers make great testers, because we tend to know the edge cases for certain types of functionality.
Here's an example of a bug which you can't find if you don't know the code internals, because you simply can't test all inputs:
long long int increment(long long int l) {
if (l == 475636294934LL) return 3;
return l + 1;
}
However, in this case it would be found if the tester had 100% code coverage as a target, and looked at only enough of the internals to write tests to achieve that.
Here's an example of a bug which you quite likely won't find if you do know the code internals, because false confidence is contagious. In particular, it is usually not possible for the author of the code to write a test which catches this bug:
int MyConnect(socket *sock) {
/* socket must have been bound already, but that's OK */
return RealConnect(sock);
}
If the documentation of MyConnect fails to mention that the socket must be bound, then something unexpected will happen some day (someone will call it unbound, and presumably the socket implementation will select an arbitrary local address). But a tester who can see the code often doesn't have the mindset of "testing" the documentation. Unless they're really on form, they won't notice that there's an assumption in the code not mentioned in the docs, and will just accept the assumption. In contrast, a tester writing from the docs could easily spot the bug, because they'll think "what possible states can a socket be in? I'll do a test for each". Since no constraints are mentioned, there's no reason they won't try the case that fails.
Answer: do both. One way to do this is to write a test suite before you see/write the code, and then add more tests to cover any special cases you introduce in your implementation. This applies whether or not the tester is the same person as the programmer, although obviously if the programmer writes the second kind of test, then only one person in the organisation has to understand the code. It's arguable whether it's a good long-term strategy to have code only one person has ever understood, but it's widespread, because it certainly saves time getting something out the door.
[Edit: I decline to say how these bugs came about. Maybe the programmer of the first one was clinically insane, and for the second one there are some restrictions on the port used, in order to workaround some weird network setup known to occur, and the socket is supposed to have been created via some de-weirdifying API whose existence is mentioned in the general sockets docs, but they neglect to require its use. Clearly in both these cases the programmer has been very careless. But that doesn't affect the point: the examples don't need to be realistic, since if you don't catch bugs that only a very careless programmer would make, then you won't catch all the actual bugs in your code unless you never have a bad day, make a crazy typo, etc.]
I guess it depends how good of testing you want. If you just want to sanity check the common scenarios, then by all means, just give the testers / pizza-eaters the application and tell them to go crazy.
However, if you'd like to have a chance at finding edge cases, performance or load issues, or a whole lot of other issues that hide in the depths of your code, you'd probably be better off hiring testers who know how and when to use white box techniques.
Your call.
IMHO, I think the industry view of testers is completely wrong.
Think about it ... you have two plumbers, one is extremely experienced, knows all the rules, the building codes, and can quickly look at something and know if the work is done right or not. The other plumber is good, and get the job done reliably.
Which one would you want to do the final inspection to make sure you don't come home to a flooded house? In fact, in what other industry do they allow someone who knows hardly anything about the system they are inspecting to actually do the inspection?
I have seen the bar for QA go up over the years, and that makes me happy. In time, QA may become something that devs aspire to be.
In short, not only should they be familiar with the code being tested, but they should have an understanding that rivals the architects of the product, as well as be able to effectively interface with the product owner(s) / customers to ensure that what is being created is actually what they want. But now I am going into a whole seperate conversation ...
Will it happen? Probably sooner than you think. I have been able to reduce the number of people needed to do QA, increase the overall effectiveness of the team, and increase the quality of the product simply by hiring very skilled people with dev / architect backgrounds with a strong aptitude for QA. I have lower operating costs, and since the software going out is higher quality, I end up with lower support costs. FWIW ... I have found that while I can backfill the QA guys effectively into a dev role when needed, the opposite is almost always not true.
If there is time, a tester should definitely go through a developers code. This way, you can improve your tests to get better coverage.
So, maybe if you write your black box tests looking at the spec and think you have the time to execute all of those and will still be left with time, going through code cannot be a bad idea.
Basically it all depends on how much time you have.. Another thing you can do to improve coverage is look at the developers design documents. Those should give you a good idea of what the code is going to look like...
Testers have the advantage of being familiar with both the dev code and the test code!
I would say they don't need to know the internal code details at all. However they do need to know the required functionality and system rules in full detail - like an analyst. Otherwise they won't test all the functionality, or won't realise when the system misbehaves.
For user acceptance testing the tester does not need to know the internal code details of the app. They only need to know the expected functionality, the business rules. When a bug is reported
Whoever is fixing the bug should know the inter-dependencies between various features.