Heuristic evaluation
Summary
Heuristic evaluation is a form of usability inspection where usability specialists judge whether each element of a user interface follows a list of established usability heuristics. Expert evaluation is similar, but does not use specific heuristics.
Usually two to three analysts evaluate the system with reference to established guidelines or principles, noting down their observations and often ranking them in order of severity. The analysts are usually experts in human factors or HCI, but others, less experienced have also been shown to report valid problems.
A heuristic or expert evaluation can be conducted at various stages of the development lifecycle, although it is preferable to have already performed some form of context analysis to help the experts focus on the circumstances of actual or intended product usage.
Benefits
The method provides quick and relatively cheap feedback to designers. The results generate good ideas for improving the user interface. The development team will also receive a good estimate of how much the user interface can be improved.
There is a general acceptance that the design feedback provided by the method is valid and useful. It can also be obtained early on in the design process, whilst checking conformity to established guidelines helps to promote compatibility with similar systems.
It is beneficial to carry out a heuristic evaluation on early prototypes before actual users are brought in to help with further testing.
Usability problems found are normally restricted to aspects of the interface that are reasonably easy to demonstrate: use of colours, lay-out and information structuring, consistency of the terminology, consistency of the interaction mechanisms. It is generally agreed that problems found by inspection methods and by performance measures overlap to some degree, although both approaches will find problems not found by the other.
The method can seem overly critical as designers may only get feedback on the problematic aspects of the interface as the method is normally not used for the identification of the 'good' aspects.
Method
This method is to identify usability problems based on established human factors principles. The method will provide recommendations for design improvements. However, as the method relies on experts, the output will naturally emphasise interface functionality and design rather than the properties of the interaction between an actual user and the product.
Planning
The panel of experts must be established in good time for the evaluation. The material and the equipment for the demonstration should also be in place. All analysts need to have sufficient time to become familiar with the product in question along with intended task scenarios. They should operate by an agreed set of evaluative criteria.
Running
The experts should be aware of any relevant contextual information relating to the intended user group, tasks and usage of the product. A heuristics briefing can be held to ensure agreement on a relevant set of criteria for the evaluation although this might be omitted if the experts are familiar with the method and operate by a known set of criteria.
The experts then work with the system preferably using mock tasks and record their observations as a list of problems. If two or more experts are assessing the system, they should not communicate with one another until the assessment is complete. After the assessment period, the analysts can collate the problem lists and the individual items can be rated for severity and/or safety criticality.
Reporting
A list of identified problems, which may be prioritised with regard to severity and/or safety criticality is produced.
In terms of summative output the number of found problems, the estimated proportion of found problems compared to the theoretical total, and the estimated number of new problems expected to be found by including a specified number of new experts in the evaluation can also be provided.
A report detailing the identified problems is written and fed back to the development team. The report should clearly define the ranking scheme used if the problem lists have been prioritised.
More Information
Nielsen, Jakob. How to Conduct a Heuristic Evaluation
Variations
Three to five experts are recommended for a thorough evaluation. A quick review by one expert (often without reference to specific heuristics) is usual before a user-based evaluation to identify potential problems.
If usability experts are not available, other project members can be trained to use the method, which is useful in sensitising project members to usability issues.
Background Reading
Bias, R.G. and Mayhew, D.J. (Eds.). Cost justifying usability. Academic Press, 1994, pp.251-254.
Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), pp. 373-380.
Nielsen, J. & Landauer, T. K. (1993). A Mathematical Model of Finding of Usability Problems. Proc. INTERCHI '93 (Amsterdam NL 24-29 April),