— 8 February 2005 —

Are You a Good Tester?

Study by Terence Andre, et al. of the US Air Force Academy on the effectiveness of the usability problem inspector.

Andre’s thesis is to come up with a more reliable method for expert reviews other than heuristics and cognitive walkthroughs. The Usability Problem Inspector is a combination of heuristics and cognitive walkthrough.

Problems with heuristics: tends to identify one-time, low-priority problems, and evaluators identify false alarms. Problems with cognitive walkthrough: requires knowledge of cognitive science, tedious to do, excessive time involved.

Aparently taking the best of both worlds give better results, at least according to their study of 20 people using the different methods. I am not sure if their numbers are reliable, but supposedly they got better reuslts in the hit and miss category than the other two methods.

Of course, a big problem with any expert review is that you have to decide what constitutes a problem.

Going back to the previous post on paper vs. computer prototype testing one would think that you could just spend your time doing a paper eval and probably getting better results. Study after study seems to show that you just shouldn’t rely on expert evaluations as your sole source for indentifying problems with your product.

We are very good at what we do, but don’t hire anyone who tells you they can tell you what’s wrong by just looking. They may be right, but they may be wrong. You will just never know for sure until you implement. And then you are paying a lot more to fix problems than during the design phase.

Respond Eloquently Below

April 2008 March 2008 February 2008 January 2008 December 2007 November 2007 October 2007 September 2007 August 2007 July 2007 June 2007 April 2007 March 2007 February 2007 January 2007 December 2006 November 2006 October 2006 September 2006 August 2006 July 2006 June 2006 May 2006 April 2006 March 2006 February 2006 January 2006 December 2005 November 2005 October 2005 September 2005 August 2005 July 2005 June 2005 May 2005 April 2005 March 2005 February 2005 January 2005 December 2004 November 2004 October 2004 September 2004 August 2004 July 2004 June 2004 May 2004 April 2004 March 2004 February 2004 January 2004 December 2003 October 2003 September 2003 August 2003 July 2003 June 2003 May 2003 April 2003 March 2003 February 2003 January 2003 December 2002 November 2002 October 2002 September 2002
Snook Approved!