February 12, 2020

What is Usability Testing Anyway?

 

Some people refer to usability testing as “user testing,” but there is a good reason to refer to it as usability testing instead. We’re testing the usability of the site (how usable is it for people), we’re not testing the users. If the user is unable to successfully pass a task, we mark that as a failure for the website, not for the user. This is important to make clear to your test participants, as we don’t want them to feel that they’re being tested and that there is a correct answer. It’s ok to tell them that at the start by saying “there are no right or wrong answers here, so please answer freely.”

​

The Nitty-Gritty of Usability Testing

 

Let’s do a quick overview of what usability testing is by using an example. Say we’re working with a nonprofit organization whose mission it is to improve the quality of youth education programming in the communities they work with with the intended outcome of kids improving their chances of getting a good job. A key part of this mission is to disseminate best practices, case studies, and research to local groups who work directly with youth. The organization’s website is crucial for organizing and disseminating this content, and one of the markers of success for this organization is that the resources get high page traffic and downloads. However, they’re finding that these indicators aren’t as high as they want them to be, and they don’t know why. 

 

Enter Usability testing. 

​

Usability testing is a research method to get at how users navigate and use a digital product, in this case, a website. During the testing, we ask users to complete tasks, and then watch their behavior as they try to complete that task. This helps us get insights into how well the website is achieving its goals.

 

For our example, we could ask a test participant “From the homepage, find a case study about a youth education program in the U.S. and show me how you would download it.” 

 

A number of things could happen in this scenario, such as: 

​

1. The participant clicks on “Resources” from the main navigation, filters by country in the listing, clicks on a resource, and points out the download button. This would be considered a success. 

2. The participant scrolls around the homepage, finds a featured Resource on the page, clicks on it, and points out the download button. This would also be considered a success.  

3. The user clicks on some other item in the main navigation and goes down a rabbit hole in that section, ultimately unable to find a resource. This would be considered a failure.

 

If we see that a lot of users are going down path 3 or taking an unreasonably long time trying to find the resource, then we know that there is something that needs to be improved on the website. For example, the terminology isn’t clear in the main navigation, or people’s mental models don’t match the way the content is organized, or the download button isn’t in a clear location on the resource page. 

 

Part of the test would be asking them to talk through their thought process as they are navigating the website or making decisions about what to do next. We would also ask quite a few follow-up questions at the end about what they liked or what could be improved. This gives us rich data to determine why they experienced an issue and how it could potentially be improved.

​

Why Usability Tests are Useful

 

As designers, we create experiences that follow best practices, but we ourselves do not always represent our end-users. While we do our best to put ourselves into the shoes of our users and imagine how they would use our sites, we are not (usually) our users. 

​

As an industry, we follow best practices and standards, which have largely shaped how users interact with digital platforms. However, digital trends are constantly changing as technological capabilities evolve. As designers who contribute to these trends, we must always remember the value of testing and iterating. Who better to tell us the effectiveness of our designs than our users themselves?

 

If a usability test shows that users are having trouble finding and downloading a particular resource that could really improve their programming, then we know that we can improve on it. And if that means more people are downloading and using this resource, then they are having more impact in their work. Ultimately, this is what it’s all about—making intentional strategic choices on the website that have a positive impact on people in the world.

​

Setting Up a Usability Test

1. Approach

 

Example: The test will be conducted in in-person moderated tests on tester-provided laptops. We will recruit by using the organization’s newsletter to send out a call for participants.

​

First, determine how you will be conducting the test. Will you be testing in-person or remotely? Will the tests be moderated or unmoderated, using an online recruiting and testing tool? Will you be testing on desktop, mobile, or tablet (or all 3)? How will you be recruiting test participants?

​

2. Timeline

 

Example: 2 weeks to prepare the test; 2 weeks to recruit the participants; 2 days to conduct the test in person; and 3 weeks to analyze the results.

​

What is your timeline for preparing the test, recruiting participants, conducting the test, and analyzing the results?

​

3. Research Questions

 

Example: Why are the resources having low page views and downloads? 

​

What are the key things that you want to learn from conducting the test? Do you have goals for what you want to achieve? Have you built out assumptions and hypotheses about how your audience might behave? For example, is the content on a certain page helpful for users? Are the main navigation labels clear? Is it easy to find out how to download a resource? Is it clear what this particular content is?

​

4. Screener

 

Example: Age: 18 to 25; Gender: Any

 

As indicated earlier, it’s crucial that you test with our target end-users, so that we can ensure that they are able to find what they need. To do this, you should determine exactly who the end-users are and prioritize them. It can be helpful to create personas of these end-users. Once you’ve identified your priority audiences, write out questions that will narrow your pool of possible test participants to people that fit into those audiences. You could do this by asking for age, education, country, and the industry they work in. 

​

5. Tasks

 

Example: From the homepage, find a case study about a youth education program in the U.S. and show me how you would download it.

​

The next step is building the test! Write out the exact tasks you want your test participants to attempt. It’s usually helpful to start with a general scenario for who they are and how they ended up at the site. When you write out your tasks, you should also write down what qualifies as success or failure. This will make the process of analyzing the test results more efficient. Remember when you’re going through the tasks with the participant to ask them probing questions to get at why they clicked on something or what they were expecting when they clicked on something.

​

6. Follow-Up Questions

 

Example: Did you use the main menu to navigate the site? Why or why not? If you did, was it easy to understand and use?

​

This is your chance to ask direct questions to the participant about their experience, such as: Which content was the easiest to find? What was your favorite thing you saw today? What was your least favorite thing? Why did you choose to look at the content you did? 

​

Usability Testing in Action: How to Use What You Learned

 

Usability testing typically provides the following common insights:​

• Validation of user pathways

• Insight into areas where pathways have missing steps / information

• First-hand reactions to content, messaging, and design

• Uncovering of technical bugs or broken links

​

After you have your findings, including a general sense of what you are looking to achieve, you will want to: ​

• Prioritize research findings – what is most crucial to solve for?

• Ideate around how to solve for the more critical issues

• Put those ideas to the test!

​

Making Data-Informed Decisions on a Regular Basis 

 

If you think about the vast diversity of people in our world—our personalities, our backgrounds, our experiences with technology— that diversity is also reflected in how people use websites and digital products. However, we as humans are also constantly creating new experiences in the spirit of innovation and creativity.

​

In other words, usability testing is not a “one-and-done” solution. We need to be continually testing, iterating, and improving our digital properties to ensure that users can find what they need and organizations can achieve their missions.