Human-moderated remote user testing: Protocols and applications

Peter Hastings

Outline

Human-moderated remote user testing: Protocols and applications

Peter Hastings

2004

Abstract

Local user testing accrues high costs when solely relied on during the iterative cycle of usercentered design. A number of automatic evaluation methods have emerged with the aim of reducing costs incurred by repeated local tests, international user testing and post-deployment assessments. When considering costs, it is important not to overlook the benefits provided in human-moderated user testing. This paper focuses on the benefits of human moderation, provides test administration protocols for human-moderated remote testing and examines those protocols in real life settings.

Figures (6)

Thinking aloud was ‘induced’ by two methods, user training in the beginning of the test and
continuous prompting by the moderator. Taking into account the distance factor between
team and user, we were led to believe that remote users would strictly adhere to the think-
aloud protocol as a means of eliminating that distance. We measured this assumption by
timing users of both tests as they thought aloud. Timings were collected by extracting the
think-aloud instances from the total test time. 2-second separator delays were added between
each instance, simulating natural speech. — Thinking aloud was ‘induced’ by two methods, user training in the beginning of the test and continuous prompting by the moderator. Taking into account the distance factor between team and user, we were led to believe that remote users would strictly adhere to the think- aloud protocol as a means of eliminating that distance. We measured this assumption by timing users of both tests as they thought aloud. Timings were collected by extracting the think-aloud instances from the total test time. 2-second separator delays were added between each instance, simulating natural speech.

ee ee eee Ee SOOO eS

Due to the communicative design of the human-moderated remote test, we saw no reason for
the qualitative data yielded by both tests to be dissimilar. We recognize that qualitative data
depends on the breadth of the information received. However, interpretation is a subjective
matter and as a result the quality of subjective data remains a challenge to measure. Taking
that into account, in addition to using the data for error interpretation, we used timing as a
way to objectively evaluate the amount of qualitative data we received from each user. Each
user was timed throughout the test, and later the qualitative data provided was extracted from
the total time. As with the think-aloud, delays were added in between the sentences. — ee ee eee Ee SOOO eS Due to the communicative design of the human-moderated remote test, we saw no reason for the qualitative data yielded by both tests to be dissimilar. We recognize that qualitative data depends on the breadth of the information received. However, interpretation is a subjective matter and as a result the quality of subjective data remains a challenge to measure. Taking that into account, in addition to using the data for error interpretation, we used timing as a way to objectively evaluate the amount of qualitative data we received from each user. Each user was timed throughout the test, and later the qualitative data provided was extracted from the total time. As with the think-aloud, delays were added in between the sentences.

Our expectations were confirmed.
Participants of both tests were able to
complete approximately the same number
of tasks (P=0.734). — Our expectations were confirmed. Participants of both tests were able to complete approximately the same number of tasks (P=0.734).

In the human-moderated remote test, communication between the team and user was
conducted by application sharing using Netmeeting, through a phone connection and by a
web cam, which allowed both sides to see each other. Although there were several seconds of
delay in the video transfer, we believed that users would feel increased comfort by putting a
face to the team members. We tested the success of our communicative approach by
administering a post-test questionnaire to all remote users. Our questions were designed to
determine whether the remote test setup was easy to use, whether the environment was
maintained friendly and whether there was sufficient assistance given when problems arose.
Users were asked to state how comfortable they were at the beginning and the end of the test
session. — In the human-moderated remote test, communication between the team and user was conducted by application sharing using Netmeeting, through a phone connection and by a web cam, which allowed both sides to see each other. Although there were several seconds of delay in the video transfer, we believed that users would feel increased comfort by putting a face to the team members. We tested the success of our communicative approach by administering a post-test questionnaire to all remote users. Our questions were designed to determine whether the remote test setup was easy to use, whether the environment was maintained friendly and whether there was sufficient assistance given when problems arose. Users were asked to state how comfortable they were at the beginning and the end of the test session.

References (17)

Bartek & Cheatham, 2003] Bartek, V. & Cheatham, D. Experience remote usability testing, Part 1. 2003. [On-line]. Available: http://www- 106.ibm.com/developerworks/web/library/wa-rmusts1.
Boren & Ramey, 2000] Boren, M.T. & Ramey, J. Thinking Aloud: Reconciling Theory and Practice. IEEE Trans. Prof. Comm., 43, 2000, pp 261-278.
Burmeister, 2001] Burmeister, O. Usability testing: revisiting informed consent procedures for testing Internet sites. In J. Weckert (Ed), Volume 1 -Computer Ethics 2000. Canberra, AU, 2001, pp. 3-10.
Cleary, 2000] Cleary, Y. An Examination of the Impact of Subjective Cultural Issues on the Usability of a Localized We Site -The Louvre Museum Web Site. Archives & Museums Informatics, 2000. Available: http://www.archimuse.com/mw2000/abstracts /prg_80000197.html
Ericsson & Simon, 1984] Ericsson, K. A. & Simon, H.A. Protocol Analysis: Verbal Reports as Data. MIT Press, Cambridge, MA, 1984.
Hammontree et al., 1994] Hammontree, M., Weiler, P. & Nandini, N. Remote Usability Testing. Interactions, 1(3), 1994, pp. 21-25.
Hartson et al., 1996] Hartson, H. R., Castillo, J., Kelso, J. & Neale, W. Remote Evaluation: The Network as an Extension of the Usability Laboratory. Proceedings CHI'96
Conference, Vancouver, Canada, 1996.
Hartson & Castillo, 1998] Hartson, H. R. & Castillo, J. Remote evaluation for post- deployment usability improvement. Proceedings of AVI '98 Conference, L'Aquila, Italy, 1998. [Kawaguchi, 2000] Kawaguchi, M. The X-Factor. Newsweek, 2000.
Nielsen, 1993] Nielsen, J. Usability Engineering. Academic Press, Boston, 1993. [Nielsen, 2002] Nielsen, J. Offshore Usability. [On-line]. Available: http://www.useit.com/ alertbox/20020916.html
Ramey & Boren, 2001] Ramey, J. & Boren, R. Keep Them Thinking Aloud: Two Ways to Conduct a Verbal Protocol and Why It Matters. Proceedings 2001 Conference Usability - A Winning Experience, Las Vegas, NV, 2001.
Rapanotti et al., 2002] Rapanotti, L., Dunckley, L. & Hall, J. G. Extending Low Cost Remote Evaluation with Synchronous Communication. Proceedings 16 th British HCI Group Annual Conference, London, UK, 2002.
Rodríguez & Gutierrez, 2000] Rodríguez, M. & Gutierrez, D. Data Gathering Agents for Remote Navigability Testing. Proceedings SCI'2000 Conference (Systemics, Cybernetics and Informatics), Orlando, FL, 2000.
Rubin, 1994] Rubin, J. Handbook of Usability Testing, John Wiley & Sons, Canada, 1994, pp.30-46.
Scholtz, 2001] Scholtz, J. Adaptation of Traditional Usability Testing Methods for Remote Testing. Proceedings of the 34th Hawaii International Conference on System Sciences, Hawaii, 2001.
Trillo, 1999] Trillo, N.G. The Cultural Component of Designing and Evaluating International User Interfaces. Proceedings 32 nd Hawaii International Conference on System Sciences, Hawaii, 1999.
Westat, 2002] Westat, J. F. The 2002 User-Friendly Handbook for Project Evaluation. NSF, Arlington, VA, 2002.

Human-moderated remote user testing: Protocols and applications

Sign up for access to the world's latest research

Abstract

Related papers

References (17)

Related papers

Related topics