User testing and user research can help us to make informed decisions for both iteration and product direction. User experience (UX) consists primarily of user testing and user research, which are most effective when used in-tandem with QA and analytics tracking.

UX is by no means meant to exist as the sole form of evaluation, and should work alongside and complement data collection and analytics. UX, by nature is primarily qualitative, which is in fact one of its main strengths. Where analytics can help us understand how something is performing — and even show us the exact pitfalls in a specific Ul — it can be much harder to interpret a clear next step without thinking like one of our users or hearing from a user directly.

By balancing our evaluations and experimentations between analytics and UX methodologies we can get a better understanding of both our overall user-base and a subset of our individual users. The goal of this balance is to be able to make informed decisions, keep track of our experiments, evaluate and define next steps all while keeping our users in the forefront.

Ordered by where in the design process testing should occur

1. Discount Usability Testing

When : Pre-user testing, when shipping out a new feature

What : Heuristic evaluations or task analysis performed internally

Time : Half of a day

Tools : Heuristic Evaluations, Cognitive Walkthroughs, Participatory Walkthroughs, Storyboarding

“Discount Usability Testing,” although ‘free’ can be very powerful and help us to quickly and inexpensively ensure that a base level of usability is achieved throughout the product. Discount Usability Testing requires an understanding of both UX principles and of our users.

Discount Usability Testing can take a number of forms, ranging from Internal testing and evaluation performed internally to Guerrilla testing. For the purpose of this doc and our protocol however, we will be focusing on internal testing only, specifically usability inspection methods such as Heuristic Evaluations and Cognitive Walkthroughs.

These methods rely on having evaluators inspect a user interface with the goal of finding usability problems and assessing the severity of these problems. Evaluators can use a shared list of heuristics, mental model research or a task to analyze the interface.

Read More:

2. Accessibility Testing (Automated)

When : While evaluating existing pages, Before shipping out a new feature, re-design

What : Online health check or keyboard test

Time : Half of a day to test and propose changes

Tools : Keyboard testing, WAVE (web accessibility tool)

In order to best serve all of our users and to ensure equal access for all, it is important that we test the accessibility of our websites for users with disabilities. There are a number of online tools that we can use quickly uncover where our work falls below accessibility web standards. Additionally keyboard tests can help to illuminate the level of difficulty that a user using a screen reader might have performing tasks on a specific webpage.

Read More:

3. Guerrilla Testing

When:

  • Post-internal analysis, when shipping out a new feature
  • As exploratory research with mocks, storyboards or prototypes
  • Can be used to complement analytics or A/B testing

What : 5–10 minute interviews or surveys with 25 external users

Time : Typically 1–3 days to set-up test, run test and evaluate test

Tools : Surveys (Qualtrics is a powerful survey software that can even be used for A/B testing and heat mapping ), paper prototypes, inVision (quick walkthrough prototypes)

The main principles behind Guerrilla Testing is that it is quick and inexpensive, but also powerful because it generates a good amount of qualitative and potentially quantitative data from current or potential users. Guerrilla Tests can focus on branding, usability or user research and can occur in person or online. Utilizing surveys to perform guerrilla testing can make these tests easy to disseminate and analyze.

Read More:

4. User Interviews (Formal Usability Testing)

Can be performed throughout the lifecycle of a project

When :

  • Can be used to complement analytics or A/B testing
  • Before starting work on a new re-design or feature
  • To help us evaluate what might need to be changed in an interface
  • The ultimate tiebreaker, why be stuck in a debate when we can talk to users?

What : 1 5 minute to hour long interviews with 5–1 0 users

Time : Typically 2–3 days to set-up test, run test and evaluate test + recruiting time

Tools : Silverback (video and screen capture for iOS), Morae (video and screen capture for PC), Ovo Studios (for multiple webcams and screen capture, PC only), UserTesting.com (for remote testing with new users)

Formal Usability Testing or User Interviews are used to gain user insight on a flow or an entire product. User interviews can either be guided or unguided and can take place in person or remotely through a user testing service. Guided User Interviews will generally focus on a task or a specific flow of tasks. While navigating through the interface and addressing each task, users are generally asked to talk aloud. Additionally, testers can decide to stop users between tasks to further evaluate the interface. Unguided testing generally also requires users to talk aloud, however the session can focus on user exploration rather than being task driven. Generally users fill out a demographic survey before beginning a test and an exit survey to assess the user’s overall reaction and emotional state after the test or interview.

All Formal User Testing (both remote and in person) should be recorded with screen capturing software, therefore allowing deep analysis. These recordings can also be used to invite the larger team to see how users are reacting to the product or a specific interface.

Read More:

5. Extended Studies

When : Before shipping a major change or after for evaluation purposes

What : Diary studies or other extended field studies

Time : 1 week to run, half of a week for evaluation

Extended studies, such as Diary Studies or User Shadowing are used to track a user interacting with the product for an extended period of time (generally one week) in a user’s normal environment. This might mean For a Diary Study, users are recruited and asked to use the product everyday for a minimum amount of time and answer a short survey about using the product either directly after each use or simply at the end of their day.

The benefits of a Diary Study are the ability to see how the product stands up in a variety of different scenarios and locations, especially those where users may be distracted. Additionally, this Diary Studies can be used to focus in on a user’s first week experiences from onboarding forward.