Related
I am interested in a general approach, not a particular library or framework.
I am trying to avoid spaghetti-code-ish conditionals when possible. It may be that I'm thinking of these apps in totally the wrong way, but I can't seem to find anything online that's helpful.
The problems arise when there are multiple sorts of states that all affect each other. Here's an example:
Image a Person that has these abilities:
Walk.
Run.
Sit.
I make a command pattern with Walk.execute(), Run.execute() and Sit.execute(). These commands are triggered by user input or whatever. What's great about this, of course, is that if running needs to be debugged or amended, it's easy to know where to look in the code base.
Now I'm asked to add a dog. The dog can...
Walk.
Run.
Sleep.
If the person is walking and the dog is running, the dog (being on a leash) will run slower than he will if he's running on his own. And if the dog is sleeping, the person will sit for longer than he normally does. In other words, the dog and the person are affected by each other.
Then I'm asked to add a refrigerator with food in it. The fridge affects the dog and person, because the amount they eat affects their energy for running, etc.
I am continually asked to tweak this system. "Please make the dog run sleep more, but only if he and the person have been walking and the fridge is almost empty." And then, later: "Please change it so that the amount the person has run has walked has no affect on the dog's sleep."
I want to avoid packing DogSleep with tons of conditionals, and I also want to avoid DogSleepIfDogAndManHaveBeenWalkingAndFridgeIsAlmostEmpty. That's not scalable.
At any time, we might want to rip the fridge out of the system. Or we might want to change it in some fundamental or subtle ways.
A more real-life example is a media player that can be displayed as a compact player, an expanded player (say one that takes up half of your screen) and a full-screen player. So there are those three screen states.
Meanwhile, the player could be displaying an image, video or text.
If it's displaying a video, that video could be playing, paused, completed, etc.
The state of the video affects the way the different screen states render. The type of media also affects the rendering. And the screen state affects things the other objects (e.g. maybe, for whatever reason, you're supposed to be able to pause a video when the player is compact but not when it's expanded or in full screen).
What is the best way to make multiple state systems interact with each other when those interactions are complex and flow in both (or multiple) directions? How can one organize such code so that it's easiest to debug and scale?
The primary way to manage complexity in complex systems is to find analogies and concepts in the real world upon which you can base your design and implementation. So, when you talk about people, dogs and refrigerators interacting in some system you base methods of interaction on the way these entries interact in the real world - which is largely autonomously based on sensed inputs and internal interests/goals/needs. When you talk about media players and images and text then finding the analogies is more difficult but just as important.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
In many online chess lobbies, I've seen instances of 'engining', where a cheater would open a chess program at the same time as the main game window. He would then set it up so that the opponent's moves are relayed to the computer, then which he would copy the computer's moves, until he (almost always) wins.
As a game developer and moderator, what is there to do about this situation?
I can't see that there is anyway to prevent someone to using a chess engine to assist them, unless you can observe the player.
You might have some luck protecting against fully automated bots, though.
Online poker sites use anti-bot measures similar to what you're describing. I recommend the series of articles How I Built a Working Poker Bot for a good overview of how these systems work, and how they are defeated.
I agree with the others who said that there's not much you can do to stop the most dedicated cheaters, but you might be able to prevent casual cheating. (The problem with that, of course, is that then the dedicated cheaters will rule your site.)
Many chess computers work to formulas and end game books, so they will often play the same move in a particular situation. You could run users game history through a variety of chess computers and see if the users chosen moves after the opening moves have correlation with how the various chess computers play. This could be used to highlight users that are using chess computers.
Sites like chesscube monitors you for some time if you comes under the radar of suspicion. They monitor how much time you are taking for hard moves and relative simple moves. If there isn't some serious difference, they may conclude you are cheating. Also I believe they implement some method to check the shifting between windows, however I'm not sure about what they use for it. But I personally know guys who had been banned. So their method is pretty good.
I second what JesperE say, You have to monitor the guy for sometime before arriving on an opinion.
As a holder of a similar site, I would suggest just to let them be. If you are not intended to monetize the bets, the cheaters will move to their level of Chess program that plays for them, and fall off.
Best practice is to keep several player rooms according to level, thus cheaters will even be welcomed, allowing strong players to reach out to higher level, and adding practice to rookies.
Technically, there's nothing I can think of you can do.
Socially, there's a lot. For example, all of the online board game servers I've seen make very public the user's win/loss record, and compute the user's rank from that. Doesn't that just encourage people to want to win? Instead, I'd record all games, but not present a win/loss record anywhere (does anybody at a real chess tournament know how many games they've won/lost ever?). Make rank a user-entered number, used for the purposes of finding an appropriate partner only, so simply showing rating of 5000 is meaningless. If you need to have some kind of 'user rating', then add a commentary system, to let users comment on moves of other people's games, and then let other users rate the comments. Commentary is one thing I haven't seen computers do intelligently yet, so it's something you can probably assume comes from a real person.
I would suggest having them have a webcam behind them but slightly to the left so you could see if they were pulling up another window such as a chess engine, as a chess master (rated 5th in Canada) I was baffled at how I was losing against players so frequently on the internet (the high timed games, ironically whenever I beat an engine user I was immediately accused of cheating) yet I would never lose to anyone except those top players in Canada's country tournaments where the best of the best were there. The difference? Those people couldn't use a chess engine while I was staring them down as they made their move. All you people that cheat, I fail to see the point, you aren't winning, you aren't furthering yourselves in the games, all you are doing is wasting your time mimicing a computer, you aren't even analyzing the board! I only play 5 minute games and blitz because these cheaters can't efficiantly use their engines in such a short time period but this is not how chess is supposed to be played you are supposed to think about you moves.
Nothing effective.
Depending on how much access you have to the computer the user is playing on, you can scan his process list for known chess programs and kick him if you see one... but there is no guarantee that he is actually using it in the manner you describe, and he can always use it on a separate computer if he has duel displays or a KVM.
Cheaters will find a way to cheat.
The good news in this case, is that the computer programs for chess are reasonably beatable unless they are running on some serious hardware.
Good luck.
You can theoretically prevent the automatic relaying of moves (but doing it manually is not much of an obstacle unless you're playing speed chess), perhaps even prevent any chess programs to be run on the same machine. But IMO that's a waste of effort, because you'll never be able to prevent people from running a chess program on a different machine sitting next to them.
Im not very familiar with this enviroment. But maybe CAPTCHA would help stop automated robots. You could also generate statistics for your users (games won/lost/average speed to move, etc). The first movements should be fast, but later on the movements should be slower as complexity increases. so you can highlight cheaters, monitor them and maybe ban their accounts as Wikipedia does with some editors. You could even make a point based system as stackoverflow does, to whitelist known good/clean players.
I cant see any way of stopping this from happening - pretty much whatever you do the cheater will still be able to manually "copy" the move that the other player has made (to another computer if necessary).
How about somehow using social mechanisms to discourage these sorts of players? Cheating in this way is obviously in itself fairly unrewarding in the long term for the cheater - if you can find and eliminate / safeguard any potential gain (for example ranked tournaments with prizes) that the cheater might be able to use this to explit against, then you should at least be able to keep the percentage of cheaters down allowing most other users to enjoy "genuine" chess games.
I don't know the specifics, but I'm sure you could get statistics about the behavior of players that cheat this way--in other words, find things that the cheaters have in common (length of turns, consistency, etc) and have your application automatically find those and put a "red flag" on players that look suspicious. Then you can personally review them (or have someone else do it) and see if indeed they look suspicious. If so, ban them.
Other than that, there is really not much you can do unfortunately. The above suggestion is a lot of work, so unless you're willing to put in the hours to create such a system, I wouldn't even bother with it. Whatever barriers you put up, determined cheaters will get around them.
I have two accounts on chess.com. The first one i use to cheat. I have rybka deep 3 which is the most beastly chess program i know. On this account i have played 70games and lost 8times. 6 of those times are to time running out. The other two was from playing two GMs. I would never enter a tournament with it because thats just crossing the line for me but regular rated game i cheat like crazy. I don't do it because i want to win. I do it because i want to see who can beat this program. The two GMs that beat it. It was one of the greatest chess games ive ever seen. They never won after that and i played them a lot after that. I have another account which is my legit account that balances out my conscience. Im more in between beginner and intermediate. Anyways great players can tell when someone is using a comp program. Ive been accused like a dozen times for cheating because some of the moves rybka pulls are just straight godly. I have gotten banned once before on chess.com for cheating. It sucked cuz I had some epic games saved on there but on my new account that i have for about 3months now has not been banned. Maybe because the people i play see it as a challenge than them getting duped. IDK but ill soon start losing on purpose to fall under the radar xD. So if you want to catch a cheater ill say look for people with ridiculous stats like 80games, 9losses, 3draws with ratings of 2200+(If youre using the regular chess rating system).
There isn't much you can do to prevent cheating other than using correlative methods and a banhammer. You can make it very difficult for them to get a new account once they are banned or better yet just match them up against other cheaters transparently. Eventually they'll get bored. While it does depend on how much access you have, I've seen some java applets that will effectively create a hash of hardware profiles similar to apples UDID, and then theres cookies. Matching banned account email hashes to strings in other login cookies wouldn't be too difficult either. Taking it to the next level, if you had an app running locally you could also peek at the process list. Looks like this might be a bit dated by now though.
The only solution is to show that cheating does not do anything beneficial for humanity and to show to cheaters that humanity is more important then some ego based reward they get from cheating. You cannot detect a computer vs human in all cases without putting them in a isolated room with no way to cheat. There are very good players, are they cheaters?
The reason there are cheaters is because society has allowed cheaters to exist and perpetuate and benefit.
Ask yourself, why does a person cheat? Self-worth? money?
When a society doesn't fulfill the needs that every human being needs the human beings find ways to do so.
e.g., if you are poor and can cheat at something(e.g., politics), become "wealthy", get a hot wife, famous, etc... then you are likely to do that.
In a perfect society everyone is equally wealthy and everyone works to help each other. There are no famous people because that creates an imbalance and everyone realizes that. There are no ugly or hot people because everyone is what they are. There are no fat people either because everyone works out to be healthy and not to be a drain on society(which makes them look better).
America and Christianity(and just about every other religion) have created a huge inversion of morality. By focusing on materialism, imaginary scenarios of life after death(Santa Clause and Tooth Fairy stuff just wrapped up in a guy named Jesus and a book instead of a Tooth).
The problems are fundamental and much more than chess and to solve the chess problem one must solve the fundamental problem. Society and humanity must change. Since this won't happen any time soon and is too complex a problem for 99.9% people to grasp(most are too busy reading their bible for answers that never work) the best thing to do for any individual that understands the consequences of cheating is to simply not cheat and life with the cheaters the best they can.
Don't go to the dark side and things will eventually get better. Cheaters cheat... they cheat each everyone, themselves, and life and there are consequences.
e.g., suppose you could divide the world up in to two groups. Cheaters and non-cheaters. You could isolate both. The cheaters would eventually die out because they don't do anything worth while. They are effectively parasites and need a host to life. Remove the host and they die(it doesn't happen in the real world because they can thrive and do quite well, because the non-cheaters are generally ignorant of the fact and unwilling to make hard choices. (e.g., if a husband is a cheater and the wife isn't, she usually stays with him rather than leaving him... or better yet, never marrying a cheater in the first place(which is hard but easier if Christianity didn't make it so hard).)
I'll give you an example: I used to play counter strike(a first person shooter) when it came out. There was obvious cheating going on and I knew it(it's impossible for humans to do certain things, just physics). I did research and found some bots and started cheating myself. When I suspected cheating I would use the bot. Guess what?!?! Sometimes I would lose! Bots are nearly perfect and instantaneous... yet I would lose! That means there are other cheaters! I would call them out and tell them I am cheating and that they are too. Most would say they weren't but I had a few admit they were.
What did I do? I quit playing the game when I realized it was more than 25% cheaters. I knew that I got nothing out of it. Cheaters don't get better at what they do. They cheat because they suck and can't handle the fact that there are better people in the world. Which goes back to society, because society creates these feels(sports, for example... just turn on TV and you'll see someone talking about how X is so great because he did Y... and it subconsciously makes you feel like you are not great, even though you are because your in the top 0.00000000001% of intelligence on this planet and probably galaxy).
Games are suppose to be about learning. When you cheat you don't learn. Society has forgot that and instead make fun of people that are not as good at something as someone else... but this is almost solely due to time. People good at X put in a ton of time in it. Cheaters don't get that. They first think the other person must be cheating because "How could I lose??!?! I am a genius! They must be cheating! I will cheat too! I'm saved, I will go to heaven! So it's ok!".
Think about it!
If people were completely honest and cared about everyone else(because it matters), we wouldn't have: Murders, cheaters, scammers, captchas, passwords, rich/poor, iOS(ok, joking ;), etc...
It can be done but only through enlightenment. When people learn the world isn't flat, that it takes two to tango, etc then we will get beyond all this mess. Until then, we can only work on trying to make it better:
Don't cheat or you become like them and go to the dark side... which is only fun temporarily cause that's how it gets its teeth in you.
If you play any type of game with others expect that they will be cheating. If they are not cheaters, then they won't get mad. When someone accuses me of cheating I take it as a compliment because it means I am better than they are(well, that's ego but it's ok because I don't take it personally ;)
Playing with cheaters only makes you better. Imagine 99% of people on chess.com cheating... and you don't. You will get better because you will learn strategy. They can't beat you in the real world then. The real deal is always better than the fake synthetic crap that America and Christianity pushes down our throats.
Ultimately it doesn't matter. You are going on die. There is only one purpose in life and that's to transcend. Know that if you are against cheating you are already further along than cheaters. But it's not a game, help them transcend and the world and after life will be better.
Or you can move to a large deserted island like me and just allow the bad world to cheat itself to death! I'd invite you to stay but only if you can prove to me that you aren't a cheater! ;)
It is mentioned on the Systems Development Life Cycle page on Wikipedia:
To manage this, a number of system development life cycle (SDLC) models have been created: waterfall, fountain, spiral, build and fix, rapid prototyping, incremental, and synchronize and stabilize.
I found a few things on Google, but I felt that they were vague and they just didn't click for me. Perhaps an explanation from someone here might be more clear.
Fountain: Stand in a circle and throw some patterns and key words in the air to see where they land. Pick up only the ones that land inside the circle. Repeat until cancelled.
Waterfall: Wrangle everyone into a boat, then yell "Geronimo!" while going over Niagra Falls. Pick up the shattered pieces then rinse and repeat. Make sure it's well documented what part of the boat each individual should be sitting in, what they should be holding on to, how loud to yell, and exactly where they should land. See form 3684-B for additional instructions.
Spiral: Pick one team member and have everyone else spin them around in circles until dizy.
Build and Fix: Just throw it against the wall to see what sticks. If something falls off add some duct tape. Used gum may also work. Any part that won't stay stuck, just throw away.
Rapid Prototyping: Do exactly what the client asked for. Repeat until they figure out what they want.
Incremental: Only build the parts you want to, and only when you want to do it. An alternate version is to only build the parts they scream loudest for, and only when they are actually standing at your desk waiting for it.
Synchronize and Stabilize: Like Spiral except only one person at a time spins the unlucky team member. When their turn is over, stop the spinning for a moment.
Waterfall is a model that enforces control and avoids parallelism; every requirement for a task has to be fulfilled before starting the task. Fountain says that a new task can be started before all requirements are met, because not all requirements are necessary at the start of the task.
Think of this: Super Mario Game,
Waterfall: first, design everything, then get hardware done (Hardware Team), then create some test sprites, then code the engine, then create artwork, then music and finish.
Fountain: while the hardware team is doing its job, artwork starts conceptual work, and coding starts some prototyping on preexisting hw. When artists and hw finishes, coders integrate these onto their code and continue 'til finishing the game...
As I understand it, they essentially contain the same steps but a fountain approach is much more iterative, with less focus on initial design and more on analysis.
You basically bodge your way through things. See what needs to happen, and improve it. See what needs to happen. Improve it.
It's more agile but at the cost of project stability. Waterfall is a lot better for large projects.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
We're neck deep in a project right now, schedules are tight (but reasonable). Our general strategy is to get a strong beta done, release it for testing, and get feedback from our testers.
Quite frequently, we're being hit by small things that spiral into long, time-costing discussions. They all boil down to one thing: While we know what features we need, we are having trouble with the little details, things like 'where should this message go' and 'do they need this feedback immediately, or will it break their flow, so we should hold off'?
These are all things that our testers SHOULD catch, but
a) Each 'low priority' bug like this drains time from critical issues
b) We want to have as strong a product as possible
and
c) Even the best testing group will miss things from time to time.
We use our product, and we know how our users use the old version...but we're all at a loss as to how to think like a user when we try to use the new version (which has significant graphical as well as underlying changes).
edit - a bit more background:
We're writing a web app used by a widely-distributed base of users. Our app is a big part of their jobs, but not the biggest (and, of course, we only matter to them when it doesn't work). Getting actual users in to use our product is difficult, as we're geographically distant from the nearest location that serves as an end user (We're in Ohio, and I think the nearest location we serve is 3+ hours away).
The closest we can get is our Customer Service team (who have been a big help, really) but they don't really think like the users either. They also serve as our testers (it really motivates them to find bugs when they know that any they DON'T find may mean a big upswing in number of calls). We've had three (of about 12 total) customer service reps back here most of the week doing some preliminary testing...they've gotten involved in the discussions as well.
Watching someone using the app is a huge benefit to me. Possibly someone who is not entirely familiar with it.
Seeing how they try to navigate, how they try to enter information or size windows. Things we take for granted after creating/running the app hour after hour, day after day.
Users will always try and do things you never expected and watching them in action might bring to light how you can change something that might have seemed minor, but really makes a big impact on them.
Read Don't make me think.
Speaking generally, you can't. There's not any way you can turn off the "programmer" part of your brain and think like a user.
And you're right about (c), testing groups don't necessarily catch all the bugs. But the best thing you can do is get a testing group comprised of real, honest-to-goodness end users, and value their feedback. Draw further conclusions from their general comments.
If you want to know how your users will see your system, the closest you can get is usability testing with real users. Everything else is just heuristics and experience, and is also subject to error. There's no such thing as a bug-free product, but you should be able to get a "strong" product with usability testing.
Buy a cheap, easy to use video camera and record your testers using the app. Even better, get some people unfamiliar with the app. to use it and video them. It's relatively cheap, and you'd be surprised what it will highlight.
I like policy of "eating your own dog food"("http://en.wikipedia.org/wiki/Eat_one's_own_dog_food). It brings you one step closer, because you become a user, although you might think like one.
Try to use your app when you are very hurry (e.g. you have someone who waits for a dinner).
You will see all this little things because you have to wait, you have to go back to the mouse of the keyboard, etc.
And also, make your wife use it. Or your mother.
Another useful test : help someone to use it, by phone. If he can't find the button with your directions, that's probably a bug.
The important thing is to get enough information that you yourself can become a "user". Once you do that you can answer most questions yourself.
The way I always do this is to go talk with them about what they need to do, what they typically do, and how they use their current tools to do it. Then (very important) sit with them while they do it. Make sure you get on with them well enough that you can come back to them with questions about how they handle edge cases you think of later (often the answer will be the appalling "we go around the system manually for that").
I will almost always notice something they are doing that is a royal PITA that they didn't bring up because they are used to having to do that and don't know any better. I will always notice that their %90 typical workflow isn't the easiest workflow the tools provide.
You can't really rely on plain old-fashioned requirements gathering by itself, because that is asking them to think like a developer. They generally don't know what is possible to do with your software, what is easy, and what is hard. Also they typically have no clue on GUI design principles. If you ask them for design input they will just tell you to put any new control on their favorite page, until the thing looks like a 747 control panel.
The problem is often that even the users don't know what they want until they are actually working with the software. Sometimes, a small oversight can be a big usability problem, sometimes a well thought out function that was requested by many users sees only little use.
My suggestions to decrease the risk of not implementing the right usability features:
Take a look at users actually doing their day to day work. Even if they use another software or no software at all. You will be able to determine the artifacts they often need to get their job done. You will see what data they frequently need. Concentrate on the artifacts, data and workflows most used. They should be the most usable. Exotic workflows may be a bit more time consuming for the users than often used workflows.
Use working prototypes of the GUI to let users work through a realistic workflow. Watch them and note what hinders them and what works well. Adjust your prototypes accordingly.
If an issue arises in an often-used part of your software, it is time to discuss it now and in details. If the issue concerns a seldom used part, make it a low priority issue and discuss it if you have the time. If issues or suggestions are low priority, they should stay low priority. If you can't determine if solution A or solution B is the best, don't run in circles with the same arguments over and over. Just implement one of the solutions and see if the beta testers like it. The worst thing you could do is waste time over tiny issues, while big issues need to be fixed.
A software will never be perfect, because the viewpoints of users differ. Some users will think that a minor problem breaks the whole application. Others will live with even severe usability issues. People tend to lend their ear to those who argue the loudest. Get to know your users to separate the "loud" issues from the important ones. It takes experience to do this, and sometimes you will make wrong decisions, but there is no perfect way, only one of steady improvement.
If you can, set aside a certain amount of usability development resources for the rollout phase of your software. Usability issues will arise when people start working with it in a real production environment. Sometimes it is not important to present the perfect software, but to solve issues quickly as they arise.
The flippant (yet somewhat accurate) answer to how to think like a user is put a knitting needle in your ear and push really hard.
The longer response is that we as programmers are not normal and I mean that in a good way. I scratch my head at the number of people who still run executables they receive from strangers in emails and then wonder how their computer got infected.
Any group of people will in time develop their own jargon, conventions, practices and expectations. As a programmer you will expect different things from an operating system than Joe User will. This is natural, to be expected yet hard to work around.
It's also why BAs (business analysts) exist. They typically come from a business or testing background and don't think like programmers. They are your link to the users.
Really though, you should be talking to your users. There's no poitn debating what users do. Just drag a few in and see what they do.
A usability test group will help.. tests not focused on discovering bugs, but on the learning curve of the new design, made by a group of users, not programmers.
I treat all users like malicious idiots.
Malicious because I assume all users are going to try and break my code, do stuff that is not allowed, avoid typing in valid data, and will do anything in their power to make my life hell.
Idiots because again I can't assume they will understand simple stuff like phone formats, will run away screaming if presented to many choices, and will not make any leap of faith on complicated instructions. The goal is to hold their hand the entire way.
At the same time, its important to make sure the user doesn't realize you think they're an idiot.
To think like a user, be one. But are these actually bugs that your testers are reporting? Or are they "enhancement requests"? If the software behaves as designed per requirements and they just don't like the way it operates, that's not a bug. That's a failure of requirements and design. Make it work, make it rock solid, make it easy to change and you'll be able to make it what your users want.
I see some good suggestions here, especially observing people trying to use you app. One thing I would suggest is to look at the order in which things are presented to the user on paper forms (if they use these to do data entry from) and make the final data entry page mimic that order as closely as possible. So many data entry errors (and loss of data entry speed) are from them having to jump around on the page and losing their place. I did some work for a political campaign this year and in every case, entering data was made much more difficult because the computer screen did things in a differnt order than the paper inputs. This is particularly important if the form is one that can't be changed (like a voter registration form, a campaign has to use what the state provides) to match the computer screen. ALso be consistent from screen to screen if possible. If it is first Name last name on one form, making it last name first name on the next will confuse people and guanteee data entry errors.
If you are truly interested in understanding users though I strongly suggest taking a course in Human factors engineering. It is an enlightening experience.
The 'right' way to do this is to prototype (or mock up) your new interface features, and watch your users try to use them. Nothing is as enlightening as seeing a real user try to use a new feature.
Unfortunately, given most projects time and resources, this is not possible. If that is the position you are in I would recommend you discuss in the team who has the best grasp of usability, and then make them responsible for usability decisions - but that person will need to regularly consult real users to make sure his/her ideas are consistent with what the users want.
I'd suggest doing some form of usability testing; I've participated in such in the past, and found them quite useful.
If you were writing a ticketing system, for example, bring up tasks, and ask questions like "how would you update this ticket" or "what do you expect to happen if this button is clicked".
You don't necessarily need a full application, either, in some places screen shots can be used.
You could take the TDD/BDD approach and get the users involved before beta, having them work with you on refining requirements as you write your unit tests. We're beginning to incorporate some of those trends into our current project, and we're seeing fewer bugs in the areas where we have involved the users earlier.
There is no "think like a user" technique, get your hands on someone who knows nothing of the project and throw what you have done at them.
It's the only way to see how the look + feel + functionality present themselves to the end user.
Once you shocked that person who knew nothing of the product, listen to all of their idiotic (or so you think they are) complaints, fix them, arrange every silly cosmetic thing they point out (either by fixing the UI or by improving whichever documentation you had)..
and after you have satisfied the person you chose to look at your app from zero knowledge on the subject first round, pick another ...and another... until they stop being shocked when they see it, and they don't get stuck on.. "ok.. what does this do?" kind of phases.
You (as a member of the project, be it the project manager, developer, etc) will never think like a user is my answer to that question.
Old saying: You can make something "fool proof" but you can't make it "Damn-fool proof".
Additionally: When you make something "idiot proof" the world invents a better idiot.
Other than that, I agree with what everyone else said.
Ask someone with absolutely no knowledge, insight or programming experience to use the program and try to figure out every function of the program.
People who would NEVER use such a program are most likely to find bugs.
See it as a new Safari user (or FF) who tries to put the URL inside the search field...
As a programmer you guess no-one would be that stupid (or, well.. unknowing), but people actually sometimes find themselves in these situations. As a programmer, we miss these things.
What are your best usability testing tips?
I need quick & cheap.
While aimed at web design, Steve Krug's excellent "Don't Make Me Think: A Common Sense Approach To Web Usability" features (in the second edition, at least), a great chapter entitled "Usability Testing On 10 Cents A Day", which I think is applicable to a much wider range of platforms.
The chapter specifically deals with usability testing done quick and dirty, in a low-budget (no money and/or no time) environment, and illustrates some of the most important considerations for getting an initial "feel" of the thing.
Some of the points I like in particular are:
You don't need to test with a huge number of people (a sentiment also echoed by Jakob Nielsen)
A live reaction is worth a lot; if possible, make sure the developers can see the reaction (perhaps using a video camera and a TV; it doesn't need to be an expensive one)
Testing a few people early is better than a lot later
Joel Spolsky is known for advocating "hallway usability testing", where you grab a few passing users and ask them to complete some simple task. Partly inspired by the "a few users yield the bulk of the results" philosophy, it's also relatively convenient and inexpensive, and can be done every so often.
Ask someone non-techy and unfamiliar with it to use it.
The archetypal non-technical user, one's elderly and scatterbrained maiden aunt. Invoked in discussions of usability for people who are not hackers and geeks; one sees references to the “Aunt Tillie test”.
The Aunt Tilly Test (Probably needs a better name in today's day and age, but that's what it's referred to)
You have to watch people use your application. If you work in a reasonable sized company, do some 'hallway testing'. Pull someone who is walking past your door into the room and say something like, 'Could you please run the payroll on this system for the next month? It should only take two minutes'.
Hopefully they won't have any problems and it shouldn't be too much of an imposition on the people walking past. Fix up any hiccups or smooth over any processes that are unnecessarily complex and repeat. A lot.
Also, make sure you know what usability is and how to achieve it. If you haven't already, check out The Design of Everyday Things.
Some good tips here.
One mistake I made earlier on in my career was turning the usability test into a teaching exercise. I'd spend a fair amount of time explaining how to use the app rather than letting the user figure that out. It taught me a lot about whether my applications were easy or hard to use by how puzzled they got trying to use the app.
One thing I did was put together a very simple scenario of what I wanted the user to do and then let them go do it. It didn't have step-by-step instruction ("click the A button, then click the B button") but instead it said things like "create a new account" and "make a deposit". From that, the user got to 'explore' my application and I got to see how easy it was to use.
Anyhow, that was pretty cheap and quite enlightening to me.
Quick and cheap won't cut it. You have to invest in a user experience framework, starting with defining clear goals for your app or website. I know it's not what people want to hear, but after supervising and watching a lot of user testing over the years, using Nielsen's discount usability methods is just not enough in most cases. Sure, if your design really sucks and have made huge usability errors, quick and dirty will get 80% of the crud out of the system. But, if you want long-term, quality usability and user experience, you must start with a good design team. And I don't mean good graphic designers, but good Information Architects, interaction designers, XHTML/CSS coders, and even Web Analytics specialists who will make sure your site/app is measurable with clear goals and metrics. I know, it's a lot of $$$, but if you are serious with your business (as I am sure most of us are), we need to get real and invest upfront instead of trying to figure out what went wrong once the whole thing is online.
Another topic to research is Heuristics for usability. This can give you general tips to follow. Here's another use of heuristics
If you don't know where to begin, start small. Sit a friend down at your computer. Explain that you want them to accomplish a task using software, and watch everything they do.
It helps to remain silent while they are actually working. Write everything down. "John spent 15 seconds looking at the screen before acting. He moused over the top nav to see if it contained popup menus. He first clicked "About Us" even though it wasn't central to his task." Etc.
Then use the knowledge you gain from this to help you design more elaborate tests. Tests with different users from different knowledge realms. More elaborate tasks and more of them.
Film them. A web-cam mounted on the monitor is a good way to capture where their eyes are moving. A video recorder coming over their shoulder at 45 degrees is a good way to capture an overview. Bonus points if you can time-sync the two. Don't worry if you can't do it all. Do what you can do.
Don't plan your test as if it's the last one you'll ever need and you want to get it perfect. There is no perfect. The only thing approaching perfection is many iteration and much repetition. You can only approach 100% confidence as the number of tests approaches the number of actual users of your software. Usually nobody even gets close to this number, but everybody should be trying to.
And don't forget to re-test people after you incorporated the improvement you saw were needed. Same people, different people, either is ok.
Do what you can do. Don't lament what you can't do. Only lament what you could have tested but didn't.
I am answering very late but I was thinking about asking a similar questions about some ideas. Maybe it is better to keep everything in this question.
I would say that:
Do not teach people about your app. Let them have fresh eyes.
Ask them to make some tasks and record their actions with a tool like camstudio http://camstudio.org/
After the test, ask them to answer so simple questions. Here is my list:
What was your first feeling when you accessed the app?
Can you define the key concepts that are used by the app?
What are the top-3 positive things about the application?
What are the top-3 negative things about the application?
What do you think about these ideas?