Heuristics vs. Usability: Evaluating the Vendor Experience

I won’t go in depth on what heuristics actually are and are not except to say that there are two reasons to do and not to do an heuristic evaluation:

Do, if:

  • You have no time/money/opportunity to do usability.
  • Your feedback will not have any impact on the implemented design, but you want to influence the next iteration of the product.

Do not do, if:

  • There are no standards or guidelines on which to build an heuristic checklist.
  • It’s only your opinion (however educated it might be) on the successes and failures of the product or service in question.

I am not a big fan of using an heuristic evaluation to “test” a product, but there have been times of late where I used the “rules of thumb” above on a few projects where I was brought in well into the development cycle. I’d much rather do usability (even if the feedback isn’t going to affect the release of the product) even just once with only 5 users than sit down and run a checklist test. The beauty an heuristics evaluation is that you can be sure the product meets or does not meet the needs of the checklist.

Recently, I found myself thrown into a project where we are only a couple of months from implementation. The product is a vendor app, and we will have very limited ability to change anything about it, for this release or perhaps ever. So, lacking time, the first thought was to do an heuristic evaluation. I am still not sure if there will ever be an opportunity to optimize the interface in future releases, but it is worth a shot.

Luckily though, one of the success factors of the project is buy in from the impacted areas; those receiving the new product. Even though the users will be using the product, the general feeling of those responsible for this effort is that the users should feel pretty good about making the change to the new product. The downside is there is no user access; all I get are SMEs.

Problem: I have no time to do usability, and I don’t want to do an heuristic evaluation. Solution: Do some heuristability. Yup, it’s hybrid time. The plan is to take our “rule of thumb” checklist (because thankfully there are standards and guidelines available), create a few scenarios based on the top two or three primary tasks for each user group, and give the checklist (with guidance) to the SMEs and have them run the scenarios and give the feedback.

Not at all an ideal situation, but I think the technical term is “rolling with it.” We’ll see how it goes. I’ve never done it this way before, but this way I get some interaction feedback, some interface feedback, and it will be done by people who are closer to representing the user groups than I am.

Any time you get someone telling you there is only one way to do find out if a product is useful, usable, and satisfying you are either talking to the VP of User Experience who actually “gets it” (see next paragraph), or you are talking to someone who is perceived as a roadblock to development. I don’t want to start an “is usability a science or an art” debate because the reality is, of course, it depends. You just hope that what you do doesn’t qualify you as a Mad Scientist (unless that’s your career goal), or that your work wouldn’t qualify for an NEA grant.

There is an ideal world out there that is attained by some companies (please hire me if this describes your company;) where usability and contextual inquiry drive changes to existing products and opportunities for new products. Where once change is initiated, iterative design and frequent testing drive what will be developed. Where visual design and interaction specifications are coded to. Where pilot (beta) testing includes usability validation testing. Where the process begins anew post-implementation. Where you are involved every step of the way.

But this harmonic convergence rarely occurs, and usually you have to do the best you can with what you have, especially when your company is implementing a vendor product.

So this is how you get to the point of doing heuristability. One of the upsides of knowing the rules is know when it’s okay to break them. My good friends Ron Zeno and Mike Rundle will agree, but will also remind us that when you start talking about usability, you just cannot be sure anyone even knows the rules.