Human-moderated remote user testing: An empirical evaluation

The current literature suggests that the performance of in-lab and remote usability testing methods results in similar findings. However, developers still view usability testing as an expensive process that requires streamlining and improvement. This paper attempts to discover if remote testing is a viable alternative to laboratory based testing and to evaluate the costs of laboratory and remote usability testing to confirm the cost benefits of remote asynchronous testing. Therefore, a remote asynchronous usability evaluation was conducted on a selected website while a laboratory-based usability evaluation was also conducted as a control group. The results reported that , remote testing can be considered of comparable or higher quality than laboratory testing when analyzing the similarities and differences of both testing methods. Remote testing was also reported to use up to 15 days less direct evaluator/consultant involvement, which resulted in cheaper costs per problem.

Developments in Human Factors in …, 2006

Whilst testing the usability of a product in a usability laboratory, usability researchers generally try to extract usability information from a representation of ‗context in use'. However, there might be cases where it is difficult to simulate the real-world usage due to lack of contextual information (e.g. scarcity of information on low-socio-economic status users' living conditions, etc.), or physical limitations (e.g. difficulty in representing the actual physical environment, etc.). This set of facts requires testing in natural settings. Discussion is focused on the relative advantages and disadvantages of each setting, along with the technical infrastructure.

Lecture Notes in Computer Science, 2007

This paper describes the planning and implementation of a crossborder usability test that was to be executed in five European countries. The usability evaluation was designed by the Usability Group at Helsinki University of Technology who also performed the testing for the Finnish partner. In the other countries the usability tests were to be implemented by teams of subject matter specialists with very heterogeneous disciplines ranging from software engineering to social sciences, gender equality and vocational counselling. This paper describes the level of materials and training prepared for the remote usability testing and discusses its adequacy both via test personnel satisfaction and comments, and by comparing the found usability problems and observed phenomena in the test sessions between the test executed by the usability experts and the subject matter specialists.

This study proposes a new methodology for conducting synchronous remote usability studies using a threedimensional virtual usability testing laboratory built using the Open Wonderland toolkit. This virtual laboratory method is then compared with two other commonly used synchronous usability test methods: the traditional lab approach and WebEx, a web-based conferencing and screen sharing approach. A study was conducted with 48 participants in total, 36 test participants and 12 test facilitators. The test participants completed 5 tasks on a simulated e-commerce website. The three methodologies were compared with respect to the following dependent variables: the time taken to complete the tasks; the usability defects identified; the severity of these usability defects; and the subjective ratings from NASA-TLX, presence and post-test subjective questionnaires. The three methodologies agreed closely in terms of the total number defects identified, number of high severity defects identified and the time taken to complete the tasks. However, there was a significant difference in the workload experienced by the test participants and facilitators, with the traditional lab condition imposing the least and the virtual lab and the WebEx conditions imposing similar levels. It was also found that the test participants experienced greater involvement and a more immersive experience in the virtual world condition than the WebEx condition. These ratings were not significantly different from those in the traditional lab condition. The results of this study suggest that participants were productive and enjoyed the virtual lab condition, indicating the potential of a virtual world based approach as an alternative to the conventional approaches for synchronous usability testing.

International Journal of Advanced Computer Science and Applications

Online Nurse Test for Indonesian Nurse Competency (ONT UKNI) is a mobile application that was developed to help increase the success rate of nurse competency test participants. By using this application, users can learn more about the materials tested and conduct try out as a competency test simulation. However, ONT UKNI has not yet passed adequate testing stages, especially in terms of User Interface/User Experience (UI/UX). The Covid-19 pandemic situation presents challenges in the UI/UX testing process. Testing process which is ideally carried out face-to-face with respondents to get further insight, have to be carried out using another approach following the new normal protocol. This study aims to test the usability of UI/UX with an unmoderated remote testing approach on ONT UKNI application using a USE questionnaire. The test was performed using 26 respondents and all were nursing profession students of Universitas Muhammadiyah Yogyakarta. Respondents performed 8 tasks on ONT UKNI and answered set of questionnaire that will be tabulated and analyzed. The results indicate that usefulness, ease of learning, and satisfaction variables get the Very Good category while the ease of use variable gets the Good category. Overall, usability testing using an unmoderated remote testing approach can be carried out and able to provide information about areas where users are satisfied with ONT UKNI application. However, some areas still have room for improvement such as better UI design and implementation of gamification.

2007

Based on our belief that critical incident data, observed during usage and associated closely with specific task performance are the most useful kind of formative evaluation data for finding and fixing usability problems, we developed a Remote Usability Evaluation Method (RUEM) that involves real users self-reporting critical incidents encountered in real tasks performed in their normal working environments without the intervention of evaluators. In our exploratory study we observed that users were able to identify, report, and rate the severity level of their own critical incidents with only brief training.

1997

To my family, whose unconditional love and support inspire me and keep me reaching for higher stars. "¡Los quiero mucho!" ACKNOWLEDGMENTS I would like to thank my advisor and role model, Dr. H. Rex Hartson for his guidance throughout my graduate work, and Rieky Keeris for her caring and effort in ensuring I always had enough "research fuel" to keep on going with my work. Their sincerity, love, and friendship are invaluable and I feel proud to be considered their "adopted son". Dr. Deborah Hix, leading member of the "thesis demolition team", kept my work focused and I thank her for her guidance and devotion. I would also like to thank the other members of my committee, Dr. Mary Beth Rosson and Dr. Robert C. Williges, who have been helpful and supportive, and contributed greatly to this document. Many other people have contributed to the completion of this work, and I am happy to have an opportunity to thank them: John Kelso, who has worked in the remote evaluation project since we started, for being supportive and for being patient when things went wrong in the usability lab; Pawan Vora, for his help in revising the thesis and contributing to this document;

2014

Several emerging countries experience increasing software development activities. With the purpose of provide useful feedback on possible courses of action for increasing application of usability evaluation in such countries, this paper explores the status of usability evaluation in a digitally emerging country. Our aim is to identifying common characteristics or behavioral patterns that could be compared with digitally advanced countries. We used an online survey answered by 26 software development organizations, which gave a snapshot of the application of usability evaluation in these organizations. We found many similarities with advanced countries, several completely new obstacles more connected with software development matters and a relatively positive improvement in the lack of “usability culture”. These findings suggest good conditions to improve conduction of usability evaluations in digitally emerging countries.

Applied Ergonomics, 2017

Collaborative virtual reality-based systems have integrated high fidelity voice-based communication, immersive audio and screen-sharing tools into virtual environments. Such three-dimensional collaborative virtual environments can mirror the collaboration among usability test participants and facilitators when they are physically collocated, potentially enabling moderated usability tests to be conducted effectively when the facilitator and participant are located in different places. We developed a virtual collaborative three-dimensional remote moderated usability testing laboratory and employed it in a controlled study to evaluate the effectiveness of moderated usability testing in a collaborative virtual reality-based environment with two other moderated usability testing methods: the traditional lab approach and Cisco WebEx, a web-based conferencing and screen sharing approach. Using a mixed methods experimental design, 36 test participants and 12 test facilitators were asked to complete representative tasks on a simulated online shopping website. The dependent variables included the time taken to complete the tasks; the usability defects identified and their severity; and the subjective ratings on the workload index, presence and satisfaction questionnaires. Remote moderated usability testing methodology using a collaborative virtual reality system performed similarly in terms of the total number of defects identified, the number of high severity defects identified and the time taken to complete the tasks with the other two methodologies. The overall workload experienced by the test participants and facilitators was the least with the traditional lab condition. No significant differences were identified for the workload experienced with the virtual reality and the WebEx conditions. However, test participants experienced greater involvement and a more immersive experience in the virtual environment than in the WebEx condition. The ratings for the virtual environment condition were not significantly different from those for the traditional lab condition. The results of this study suggest that participants were productive and enjoyed the virtual lab condition, indicating the potential of a virtual world based approach as an alternative to conventional approaches for synchronous usability testing.

International Journal of Testing, 2006