6. User Testing
Developing a Test Protocol
Thanks to some helpful referrals from her client, Lulu has reached out and contacted a number of people to assist with user testing. Now she is ready to plan her testing. To conduct your user testing, in most cases you will want to develop a “test protocol” that details the activities and the steps taken to test a particular set of features. Most testing sessions with users should not exceed one hour, so a test protocol should provide a number of relatively short scenarios that can be completed in 5-10 minutes, allowing time between each scenario for a few questions to probe the user’s experience.
Talk Aloud Protocol
One of the best ways to gather information about a user’s experience using your web content is to have them talk aloud as they complete tasks. Record verbal responses as well as any emotional responses or facial expression in a log. You can also ask simple questions like “what were you thinking when…?” to determine their thoughts. It may be preferable to video or audio record sessions and analyze them afterwards so you as the observer can focus your attention on guiding and probing the tester’s thoughts.
Note: If you plan to video or audio record testing sessions, always be sure to inform user testers of this fact.
Test Protocol Example – The Lake Devo Website
As an example of a potential user testing protocol, we will look at the plans put together to user test Lake Devo, a web-based role playing game used by students at Toronto Metropolitan University to design and play out various scenarios related to their subject of study. The Lake Devo website has recently undergone extensive improvements for accessibility.
The User Testers
A couple of screen reader users who had not used Lake Devo previously were recruited to complete a series of tasks that would test the accessibility and usability of key features in the software. Both were skilled JAWS users and knowledgeable about web technology. They were both also students in online courses at the university.
The testers were told they would receive compensation in the form of a $100 CAD prepaid credit card. Regardless of whether the testers completed all the scenarios, they received their compensation.
Pre-Test Tasks
Before starting the testing session, the Observer (in this case the web developer) provided a description of Lake Devo, its purpose, its main features, and an overview of how it might be used to develop a role playing movie. Each user tester was informed at the start of the session that if at anytime they needed a break, felt uncomfortable or wished to discontinue testing, they were free to do so.
Then the Observer introduced the scenarios the tester would be asked to complete and how the session would unfold. For each of the scenarios, the Observer essentially “trained” the user testers by doing the following:
- Reading out the scenario in full
- Walking through the scenario him/herself describing aloud the steps taken to complete