Usability Inspection Methods

Inspection-based evaluation methods predicting usability problems can be applied for evaluating products without involving users. A new method (named SEEM), inspired by Norman's theory-of-action model [18] and Malone's concepts of fun , is described for predicting usability and fun problems in children's computer games. This paper describes a study to assess SEEM's quality. The results show that the experts in the study predicted about 76% of the problems found in a user test. The validity of SEEM is quite promising. Furthermore, the participating experts were able to apply the inspection-questions in an appropriate manner. Based on this first study ideas for improving the method are presented.

The Usability Pattern Inspection (UPI) is a new usability inspection method designed for the added downstream utility of producing concrete design recommendations. This paper provides first empirical evidence that UPI measures up to the established inspection method Heuristic Evaluation (HE) regarding defect identification. It is shown that there is also some potential for synergy between UPI and HE. The further research plan of measuring UPI is presented.

Le manque d'utilisabilité des Environnements Virtuels (EVs) est un obstacle majeur à leur utilisation à grande échelle dans de nombreux domaines d'activités professionnels. Ces environnements informatisés présentent plusieurs spécificités (i.e., recours à la 3D, à la multimodalité, etc.) non prises en compte par les méthodes classiques d'évaluation de l'utilisabilité. Il est pourtant nécessaire de disposer de méthodes permettant d'apprécier leur qualité ergonomique. Cette thèse développe les différentes étapes d'élaboration et de validation des Critères Ergonomiques (C.E.) adaptés aux Interactions Homme-Environnements Virtuels. Une première étape a consisté à définir les C.E. à partir de l'élaboration et de la classification de recommandations ergonomiques, dédiées aux EVs, issues de la littérature du domaine. Sur la base de 170 recommandations, 20 critères élémentaires ont été identifiés, définis, justifiés et illustrés. Cette étape est le fondement de l...

Usability is one of the most relevant quality aspects in Web applications. A Web application is usable if it provides a friendly, direct and easy to understand interface. Many Usability Inspection Methods (UIMs) have been proposed as a cost effective way to enhance usability. However, many companies are not aware of these UIMs and consequently, are not using them. A secondary study can identify, evaluate and interpret all data that is relevant to the current knowledge available regarding UIMs that have been used to evaluate Web applications in the past few decades. Therefore, we have extended a systematic mapping study about Usability Evaluation Methods by analyzing 26 of its research papers from which we extracted and categorized UIMs. We provide practitioners and researches with the rationale to understand both the strengths and weaknesses of the emerging UIMs for the Web. Furthermore, we have summarized the relevant information of the UIMs, which suggested new ideas or theoretica...

Motivation Usability problems that the heuristic tries to avoid. 1. Visibility of System Status Conformance Question Are users kept informed about system progress with appropriate feedback within reasonable time? Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks. Motivation Feedback allows the user to monitor progress towards solution of their task, allows the closure of tasks and reduces user anxiety. Description of Heuristic Evaluation 2 2. Match between system and the real world Conformance Question Does the system use concepts and language familiar to the user rather than system-oriented terms. Does the system use real-world conventions and display information in a natural and logical order? Evidence of Conformance Necessary evidence must be identified through user studies (or through assumptions about users!), and through the analysis of individual tasks. Motivation A good match minimises the extra knowledge required to use the system, simplifying all task action mappings (re-expression of users intuitions into system concepts). 3. User control and freedom Conformance Question Can users do what they want when they want? Evidence of Conformance Necessary evidence takes the form of a diverse set of design features, for example "undo and redo", clearly marked exits etc. Motivation Quite simply, users often choose actions by mistake. 4. Consistency and Standards Conformance Question Do design elements such as objects and actions have the same meaning or effect in different situations? Evidence of Conformance Necessary evidence must be identified through several analyses (consistency within system, conformance to style guides, consistency across task methods). Motivation Consistency minimises user knowledge required to use system by letting users generalise from existing experience of the system or other systems. Description of Heuristic Evaluation 3 5. Error prevention Conformance Question Can users make errors which good designs would prevent? Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks and of system details (e.g. adjacency of function keys and menu options, discriminability of icons and labels). Motivation Errors are the main source of frustration, inefficiency and ineffectiveness during system usage. 6. Recognition rather than recall Conformance Question Are design elements such as objects, actions and options visible? Is the user forced to remember information from one part of a system to another. Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks. Motivation Forcing users to remember details such as command and file names is a major source of error. Recognition minimises user knowledge required to use the system. Summarising available commands or options may allow the user to guess their meaning or purpose. 7. Flexibility and efficiency of use Conformance Question Are task methods efficient and can users customise frequent actions or use short cuts? Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks, and the presence of design features such as keyboard accelerators etc. Motivation Inefficient task methods can reduce user effectiveness and cause frustration. 8. Aesthetic and minimalist design Conformance Question Do dialogues contain irrelevant or rarely needed information? Description of Heuristic Evaluation 4 Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks. Motivation Cluttered displays have the effect of reducing search times for commands or users missing features on the screen. Users unfamiliar with a system often have to find an action to meet a particular need-reducing the number of actions available could make the choice easier. 9. Help users recognize, diagnose and recover from errors Conformance Question Are error messages expressed in plain language (no codes), do they accurately describe the problem and suggest a solution? Evidence of Conformance Necessary evidence must be identified through analysis of error messages. Motivation Errors are the main source of frustration, inefficiency and ineffectiveness during system usage. The first ten heuristics were originally suggested by Jakob Nielsen. These materials have been developed for use on UK EPSRC project no. GR/K82727 (Extending HCI Design Principles and Task Analysis for Software and Data Visualisation) by Darryn Lavery, Gilbert Cockton and Malcolm Atkinson. A structure has been applied to the heuristics, and to the best of our intentions we have kept the original meanings of the individual heuristics. The structure and any unintended changes to the meanings remain the responsibility of the authors and not Jakob Nielsen.

We need more reliable usability inspection methods (UIMs), but assessment of UIMs has been unreliable [5]. We can only reliably improve UIMs if we have more reliable assessment. When assessing UIMs, we need to code analysts’ predictions as true or false positives or negatives, or as genuinely missed problems. Defenders of UIMs often claim that false positives cannot be accurately coded, i.e., that a prediction is true but has never shown up through user testing or other validation approaches. We show this and similar claims to be mistaken by briefly reviewing methods for reliable coding of each of five types of prediction outcome. We focus on falsification testing, which allows confident coding of false positives.

We describe the impact on analyst performance of an extended problem report format. Previous studies have shown that Heuristic Evaluation can only find a high proportion of actual problems (thoroughness) if multiple analysts are used. However, adding analysts can result in a high proportion of false positives (low validity). We report surprising interim results from a large study that is exploring the DARe model for evaluation method effectiveness. The DARe model relates the effectiveness of an evaluation method to evaluators' command of discovery and analysis resources. Previous work has shown that Heuristic Evaluation poorly supports problem discovery and analysis: heuristics tend to be inappropriately applied to problem predictions. We developed an extended problem report format to let us study analyst decision making during usability inspection. Our focus was on the quality of insights into analyst behaviour delivered by this extended report format. However, our first use of this format revealed unexpected improvements in validity (false positive reduction) and appropriate heuristic application. We argue that the format has unexpectedly led to more care and caution in problem discovery and elimination, and in heuristic application. Evaluation performance can thus be improved by indirectly 'fixing the analyst' via generic fixes to inspection methods. In addition, we provide the first direct evidence of how evaluators use separate discovery and analysis resources during usability inspection.

Motivation Usability problems that the heuristic tries to avoid. 1. Visibility of System Status Conformance Question Are users kept informed about system progress with appropriate feedback within reasonable time? Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks. Motivation Feedback allows the user to monitor progress towards solution of their task, allows the closure of tasks and reduces user anxiety. Description of Heuristic Evaluation 2 2. Match between system and the real world Conformance Question Does the system use concepts and language familiar to the user rather than system-oriented terms. Does the system use real-world conventions and display information in a natural and logical order? Evidence of Conformance Necessary evidence must be identified through user studies (or through assumptions about users!), and through the analysis of individual tasks. Motivation A good match minimises the extra knowledge required to use the system, simplifying all task action mappings (re-expression of users intuitions into system concepts). 3. User control and freedom Conformance Question Can users do what they want when they want? Evidence of Conformance Necessary evidence takes the form of a diverse set of design features, for example "undo and redo", clearly marked exits etc. Motivation Quite simply, users often choose actions by mistake. 4. Consistency and Standards Conformance Question Do design elements such as objects and actions have the same meaning or effect in different situations? Evidence of Conformance Necessary evidence must be identified through several analyses (consistency within system, conformance to style guides, consistency across task methods). Motivation Consistency minimises user knowledge required to use system by letting users generalise from existing experience of the system or other systems. Description of Heuristic Evaluation 3 5. Error prevention Conformance Question Can users make errors which good designs would prevent? Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks and of system details (e.g. adjacency of function keys and menu options, discriminability of icons and labels). Motivation Errors are the main source of frustration, inefficiency and ineffectiveness during system usage. 6. Recognition rather than recall Conformance Question Are design elements such as objects, actions and options visible? Is the user forced to remember information from one part of a system to another. Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks. Motivation Forcing users to remember details such as command and file names is a major source of error. Recognition minimises user knowledge required to use the system. Summarising available commands or options may allow the user to guess their meaning or purpose. 7. Flexibility and efficiency of use Conformance Question Are task methods efficient and can users customise frequent actions or use short cuts? Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks, and the presence of design features such as keyboard accelerators etc. Motivation Inefficient task methods can reduce user effectiveness and cause frustration. 8. Aesthetic and minimalist design Conformance Question Do dialogues contain irrelevant or rarely needed information? Description of Heuristic Evaluation 4 Evidence of Conformance Necessary evidence must be identified through analysis of individual tasks. Motivation Cluttered displays have the effect of reducing search times for commands or users missing features on the screen. Users unfamiliar with a system often have to find an action to meet a particular need-reducing the number of actions available could make the choice easier. 9. Help users recognize, diagnose and recover from errors Conformance Question Are error messages expressed in plain language (no codes), do they accurately describe the problem and suggest a solution? Evidence of Conformance Necessary evidence must be identified through analysis of error messages. Motivation Errors are the main source of frustration, inefficiency and ineffectiveness during system usage. The first ten heuristics were originally suggested by Jakob Nielsen. These materials have been developed for use on UK EPSRC project no. GR/K82727 (Extending HCI Design Principles and Task Analysis for Software and Data Visualisation) by Darryn Lavery, Gilbert Cockton and Malcolm Atkinson. A structure has been applied to the heuristics, and to the best of our intentions we have kept the original meanings of the individual heuristics. The structure and any unintended changes to the meanings remain the responsibility of the authors and not Jakob Nielsen.

This study compared the results of a usability inspection conducted under two separate conditions: An explicit concurrent think-aloud that required explanations and silent working. 12 student analysts inspected two travel websites thinking-aloud and working in silence to produce a set of problem predictions. Overall, the silent working condition produced more initial predictions, but the think-aloud condition yielded a greater proportion of accurate predictions as revealed by falsification testing. The analysts used a range of problem discovery methods with system searching being favoured by the silent working condition and the more active, goal playing discovery method in the think-aloud condition. Thinking-aloud was also associated with a broader spread of knowledge resources.

Usability is one of the most relevant quality aspects in web applications. A web application is usable if it provides a friendly, direct and easy to understand interface. Many Usability Inspection Methods (UIMs) have been proposed as a cost effective way to enhance usability. However, many companies are not aware of these UIMs and consequently, are not using them. A secondary study can identify, evaluate and interpret all data that is relevant to the current knowledge available regarding UIMs that have been used to evaluate web applications in the past few decades. Therefore, we have extended a systematic mapping study about Usability Evaluation Methods by analyzing 26 of its research papers from which we extracted and categorized UIMs. The results analysis provides practitioners and researches with the rationale to understand both the strengths and weaknesses of the emerging UIMs.

In this digital age, a university's website serves as a portal to its knowledge and services. These websites represent the actual university, where prospective students plan for admission, current students connect regularly, and other stakeholders can get the details they need. These users expect university websites to be professionally developed, well-organized, and have a user-friendly interface that allows them to easily search, navigate, and collect information. Users unwittingly introduce cultural norms into their lives, leading them to assume the same preferences in their daily interactions. These preferences reflect the user's cultural context as well as the domain in which the application is used. The heuristic evaluation is a widely applied method for usability evaluation. It is an inspection-based, fast, and low-cost method for detecting usability issues in a system. These heuristics are successful at identifying general usability design defects, but they fall short of detecting possible domain and culture-specific usability issues. In this regard, this research uses the domain and culture-specific heuristic evaluation to evaluate the usability of the websites of Pakistan's top ten universities. The evaluation's findings are collectively discussed, and possible usability problems in the website design of Pakistani universities are identified.

Usability is one of the most relevant quality aspects in Web applications. A Web application is usable if it provides a friendly, direct and easy to understand interface. Many Usability Inspection Methods (UIMs) have been proposed as a cost effective way to enhance usability. However, many companies are not aware of these UIMs and consequently, are not using them. A secondary study can identify, evaluate and interpret all data that is relevant to the current knowledge available regarding UIMs that have been used to evaluate Web applications in the past few decades. Therefore, we have extended a systematic mapping study about Usability Evaluation Methods by analyzing 26 of its research papers from which we extracted and categorized UIMs. We provide practitioners and researches with the rationale to understand both the strengths and weaknesses of the emerging UIMs for the Web. Furthermore, we have summarized the relevant information of the UIMs, which suggested new ideas or theoretica...

Usability is one of the most relevant quality aspects in web applications. A web application is usable if it provides a friendly, direct and easy to understand interface. Many Usability Inspection Methods (UIMs) have been proposed as a cost effective way to enhance usability. However, many companies are not aware of these UIMs and consequently, are not using them. A secondary study can identify, evaluate and interpret all data that is relevant to the current knowledge available regarding UIMs that have been used to evaluate web applications in the past few decades. Therefore, we have extended a systematic mapping study about Usability Evaluation Methods by analyzing 26 of its research papers from which we extracted and categorized UIMs. The results analysis provides practitioners and researches with the rationale to understand both the strengths and weaknesses of the emerging UIMs.

This paper presents and adapts the Cognitive Dimensions of Notations framework (Green and Petre, 1996) for use in designing and analysing notations (and user interfaces) in both digital and traditional music practice and study. Originally developed to research the psychology of programming languages, the framework has since found wider use in both general HCI and music. The paper provides an overview of the framework, its application, and a detailed account of the core cognitive dimensions, each discussed in the context of three music scenarios: the score, Max/MSP, and sequencer/DAW software. Qualitative and quantitative methodologies for applying the framework are presented in closing, highlighting directions for further development of the framework.

Usability is one of the most relevant quality aspects in Web applications. A Web application is usable if it provides a friendly, direct and easy to understand interface. Many Usability Inspection Methods (UIMs) have been proposed as a cost effective way to enhance usability. However, many companies are not aware of these UIMs and consequently, are not using them. A secondary study can identify, evaluate and interpret all data that is relevant to the current knowledge available regarding UIMs that have been used to evaluate Web applications in the past few decades. Therefore, we have extended a systematic mapping study about Usability Evaluation Methods by analyzing 26 of its research papers from which we extracted and categorized UIMs. We provide practitioners and researches with the rationale to understand both the strengths and weaknesses of the emerging UIMs for the Web. Furthermore, we have summarized the relevant information of the UIMs, which suggested new ideas or theoretica...

Usability is one of the most relevant quality aspects in web applications. A web application is usable if it provides a friendly, direct and easy to understand interface. Many Usability Inspection Methods (UIMs) have been proposed as a cost effective way to enhance usability. However, many companies are not aware of these UIMs and consequently, are not using them. A secondary study can identify, evaluate and interpret all data that is relevant to the current knowledge available regarding UIMs that have been used to evaluate web applications in the past few decades. Therefore, we have extended a systematic mapping study about Usability Evaluation Methods by analyzing 26 of its research papers from which we extracted and categorized UIMs. The results analysis provides practitioners and researches with the rationale to understand both the strengths and weaknesses of the emerging UIMs.

Structured Problem Report Formats have been key to improving the assessment of usability methods. Once extended to record analysts' rationales, they not only reveal analyst behaviour but also change it. We report on two versions of an Extended Structured Report Format for usability problems, briefly noting their impact on analyst behaviour, but more extensively presenting insights into decision making during usability inspection, thus validating and refining a model of evaluation performance.

We need more reliable usability inspection methods (UIMs), but assessment of UIMs has been unreliable [5]. We can only reliably improve UIMs if we have more reliable assessment. When assessing UIMs, we need to code analysts' predictions as true or false ...

This research takes an analyst-centred approach to improving Usability Inspection Methods. The research approach adopts novel instruments and methods, especially manipulation and monitoring of analyst's knowledge resources during usability inspection. The research aims to develop and validate a predictive model of analyst performance that supports improvement of usability inspection, resulting in a positive impact on design quality.

Structured Problem Report Formats have been key to improving the assessment of usability methods. Once extended to record analysts' rationales, they not only reveal analyst behaviour but also change it. We report on two versions of an Extended Structured Report Format for usability problems, briefly noting their impact on analyst behaviour, but more extensively presenting insights into decision making during usability inspection, thus validating and refining a model of evaluation performance.

Heuristic Evaluation (Nielsen and Molich, 1990; Nielsen, 1994) is a method of usability evaluation where an analyst finds usability problems by checking the user interface against a set of supplied heuristics or principles.

Three new heuristics for software visualisations have been added to the set described in (Nielsen, 1994). A set of evaluation materials for these heuristics are presented, in particular:

• A structured description of Heuristic Evaluation to ease learning and use;
• A self assessed test to allow analysts to assess their understanding of Heuristic Evaluation;
• A checklist of heuristics for use in the analysis;
• Problem record sheets to record usability problems during analysis. Currently, these only record the usability problem predicted. In future, these could be modified to include recommended solutions to these problems (Jeffries, 1994);
• A questionnaire to fill in after the analysis

Heuristic Evaluation (Nielsen and Molich, 1990; Nielsen, 1994) is a method of usability evaluation where an analyst finds usability problems by checking the user interface against a set of supplied heuristics or principles.

A set of evaluation materials for the heuristics described in (Nielsen, 1994) are presented, in particular:

• A structured description of Heuristic Evaluation to ease learning and use;
• A self assessed test to allow analysts to assess their understanding of Heuristic Evaluation;
• A checklist of heuristics for use in the analysis;
• Problem record sheets to record usability problems during analysis. Currently, these only record the usability problem predicted. In future, these could be modified to include recommended solutions to these problems (Jeffries, 1994);
• A questionnaire to fill in after the analysis

Cognitive Dimensions (Green and Petre, In Press) is a broad-brushed evaluation technique for a restricted set of systems called notations which are used to design information structures, for example programming languages, spreadsheets and visualisations. Instead of examining surface features the method examines the structure of the system.

A set of evaluation materials for Cognitive Dimensions are presented, in particular:

• A structured description of Cognitive Dimensions to ease learning and use;
• A self assessed test to allow analysts to assess their understanding of Cognitive Dimensions;
• A checklist of Cognitive Dimensions for use in the analysis;
• Problem record sheets to record usability problems during analysis. Currently, these only record the usability problem predicted. In future, these could be modified to include recommended solutions to these problems (Jeffries, 1994);
• A questionnaire to fill in after the analysis

A Cognitive Walkthrough is a method for predicting usability problems with an interactive system. Usability problems are aspects of the system which could reduce the usability of the system for the user, for example to confuse them, to slow them down to stop them completing their task. In particular, Cognitive Walkthough focuses on problems related to the learnability of a system.

The following materials are provided:

• A description of Cognitive Walkthrough;
• A self assessed test to allow analysts to assess their understanding of Cognitive Walkthrough;
• A checklist of Cognitive Walkthrough for use in the analysis;
• Problem record sheets to record usability problems during analysis. Currently, these only record the usability problem predicted. In future, these could be modified to include recommended solutions to these problems (Jeffries, 1994);
• A questionnaire to fill in after the analysis

We describe the impact on analyst performance of an extended problem report format. Previous studies have shown that Heuristic Evaluation can only find a high proportion of actual problems (thoroughness) if multiple analysts are used. However, adding analysts can result in a high proportion of false positives (low validity). We report surprising interim results from a large study that is exploring the DARe model for evaluation method effectiveness. The DARe model relates the effectiveness of an evaluation method to evaluators' command of discovery and analysis resources. Previous work has shown that Heuristic Evaluation poorly supports problem discovery and analysis: heuristics tend to be inappropriately applied to problem predictions. We developed an extended problem report format to let us study analyst decision making during usability inspection. Our focus was on the quality of insights into analyst behaviour delivered by this extended report format. However, our first use of this format revealed unexpected improvements in validity (false positive reduction) and appropriate heuristic application. We argue that the format has unexpectedly led to more care and caution in problem discovery and elimination, and in heuristic application. Evaluation performance can thus be improved by indirectly 'fixing the analyst' via generic fixes to inspection methods. In addition, we provide the first direct evidence of how evaluators use separate discovery and analysis resources during usability inspection.

Structured Problem Report Formats have been key to improving the assessment of usability methods. Once extended to record analysts' rationales, they not only reveal analyst behaviour but also change it. We report on two versions of an Extended Structured Report Format for usability problems, briefly noting their impact on analyst behaviour, but more extensively presenting insights into decision making during usability inspection, thus validating and refining a model of evaluation performance.

We describe how an assessment of the Heuristic Evaluation Usability Inspection Method led to the derivation of a simple analyst-centred theory that assesses and scopes methods by their discovery resources (DR) and analysis resources (AR).  Our DR-AR Model improves method assessment by supporting the generation of practically relevant conjectures and the means to explore them.  One such conjecture rejects established recommendations for the practice of usability inspections, on the use of multiple analysts, which we show increases false alarms.  The ability to generate and test such fresh insights into usability approaches confirms that the DR-AR model is a promising resource for research and practice.

99 analysts worked in-groups to apply the Heuristic Evaluation method to an office application’s drawing editor. The use of structured problem report formats eased merging of analysts’ predictions and subsequent matching to actual problems revealed by user testing with 16 participants. Actual problems were discovered via user tests, which were based on tasks designed to ensure that all predicted problems would be thoroughly addressed. Analysis of accurate and inaccurate predictions has supported the derivation of the DR-AR model for usability inspection method effectiveness. The model distinguishes between the discovery of candidate (possible) problems and their subsequent confirmation (or elimination) as probable problems. We confirm previous findings that heuristics do not support the discovery of possible usability problems. Our results also show that heuristics were only used appropriately to confirm possible problems that turned out to have low impact or frequency. Otherwise, heuristics are used inappropriately and, if acted on, would result in poor problem remediation. Our results also show that heuristics are very poor at eliminating improbable problems, and thus mostly incorrectly confirm predictions that could compromise the efficiency and effectiveness of usability activities. Overall, heuristics provide a poor analyst resource for the successful elimination/confirmation of im/probable problem predictions. Analysis of false predictions reveals that more effective analyst resources are knowledge of users, interaction, application domains, the application itself and design knowledge from HCI. Using the DR-AR model, we derive a strategy for UIM improvement.

If you ask someone outside the Human-Computer Interaction (HCI) field about usability, many will mention the "classic" discount methods popularized by Jakob Nielsen and others. Discount methods have the appeal of seeming easy to do, and, more importantly for business, being inexpensive. This is especially attractive to smaller startup companies with low budgets. But are discount methods really too risky to justify the "low" cost? This month's business column authors think so, based on their research and experience. Indeed, they believe that these discount methods may actually backfire and end up discrediting the field. Following a lively discussion on the CHI-WEB listserv, we asked them to explain what they see the risks to be, and what they believe we, as a profession, can and should do about it, David Siegel and Susan Dray, Column Editors

We present a new framework for Structured Usability Problem EXtraction (SUPEX), which structures decision making during problem extraction. It avoids the unreliability and inconsistency associated with direct extraction of problems from raw test data. It does so via flexible phases that ground user difficulties in test data, and then systematically group and generalise difficulties before causal analysis and solution recommendation. The SUPEX framework can be adapted to different evaluation contexts.