The best way to validate design and requirement decisions with your users is via usability testing. The general idea is to gather feedback on early concepts, understand workflows & user pain points and allow designers and product managers to document findings and create/maintain personas, roles and system workflows.
Agile methodology lends itself extremely well for iterative usability testing. In this article I’ll review my own team’s process for how we plan to integrate usability testing into our product development cycle.
As a general rule, the best test subjects are real product users, but it is better to test anyone rather than not to test at all. We’ve created 3 tiers of participants:
- Tier 1: Customers/Partners – Ideally, we only test real users from our customers and partners in the field. We are establishing a charter customer program to make real users readily available for such testing.
- Tier 2: Recruited Participants – When needed, we may supplement our testing with recruited participants – either paid, gifted or volunteer.
- Tier 3: Internal Employees – As a last resort if no other participants are available, we’ll test our ideas on Sales, Marketing, Business Development, and/or any other customer-facing groups. This approach still provides valuable feedback and validation, but we need to keep in mind that results may be slightly skewed.
Planning, Preparation and Facilitation
To prepare for the test, our design team collaborates closely with product management to define the usability tasks to test and the interview questions concerning value. We try to concentrate on the key tasks that users will do most of the time.
How we test users is determined by multiple factors, including what type of test environments are needed (i.e., Web v. Mobile), as well as when and where we are testing. Over the years I’ve tested in-house, remotely (using a video conferencing tool like Adobe Connect, JoinMe or Skype) and on-site at a customer’s office. The latter is usually the best case since you can get a feel for the user’s actual workspace limitations, workflows and peer interactions – but it is not always possible, so take what you can get.
We also consult with our product team to decide the most efficient approach, based on the available time and resources needed to:
- Create a wireframe or build an interactive prototype
- Present working code in a test environment
- Use card sorting, paper prototypes and/or taxonomy studies.
When conducting the test, it’s important to make sure to provide the user with a proper orientation to the environment and ask for permission to record. It also helps to mention that that they won’t be hurting your feelings by giving an honest opinion, and to continually remind them to “think aloud” to provide feedback and context. Watch what they do, and observe both verbal and non-verbal clues about why they fail or become confused.
The testing environment and equipment (laptop, phone, tablet, software) may vary, depending on the product type (desktop v. mobile), but should be a quiet, closed space that can minimally accommodate a tester, a facilitator and a note taker. Recording audio and video is highly recommended; using cameras to capture both screen and the user’s face provides emotion and context to any issues that arise. A laptop equipped with a webcam and screen recording software is ideal — I typically use Silverback on a MacBook Pro for this. Remember that face-to-face testing is ideal, but any testing is better than no testing.
When to Test
Our recommendation is to test users early and often, during all phases of the product lifecycle. This includes:
Conceptual/Prototyping Stages – This happens earlier in the project, during the requirements/planning stage (iteration -1, no code or design specifications). Try to recruit 8-10 testers to participate in 45-60 minute individual sessions covering multiple features or a new application. It’s critical to get input from existing customers for validation and understanding real workflows via wireframes and/or an interactive prototype. This type of testing usually requires a detailed script and covers a new product or multiple user stories. In the past, I’ve successfully used Flash, HTML, Axure and PowerPoint to create mid-to-high fidelity interactive prototypes. I’ve found that the choice of tool is less important than its ability to simulate an experience that mimics the desired end product, or that it can be delivered within the chosen test platform – and you absolutely must be able to work efficiently with the tool.
Development Stages – During product development, testing is done during each sprint. Note that the duration of the sprint isn’t important, as long as at least one round of testing is conducted per sprint. Usability testing is much lighter and informal during this stage, so try to recruit about 5 participants for 15-30 minute individual sessions that focus on a specific feature. You’ll be able to quickly spot trends and patterns in usage, which will allow you to iterate on your design during the sprint, if needed. Tests can either use live code (from a QA environment) or a quick wireframe/mockup if testing items for a future sprint. Allocate a set amount of hours for usability testing activity in the sprint backlog – to be burned down – making sure that all UT activities (planning, testing and analysis) fit within your time estimate.
After testing is completed, we generate a Usability Observation List that will be shared with the team (via verbal review at scrum, and entered into our wiki within JIRA). Product Management will prioritize these results into one of two categories: Bugs or Enhancement Requests. Bugs are entered into the tracking system (we use JIRA, but TFS, or even something as simple as Excel works as well) and should be fixed before the end of the sprint. See figure 1 for a visual representation of our process.
Post Launch – After your product is launched, be sure to have an internal retrospect with the product/feature team to discuss what was successful versus what was a problem and consider how to streamline and improve your UT process. In the past, I’ve used surveys, email discussions, message boards and field-testing to gather feedback from end-users. It’s also helpful to have the product team speak directly with external product champions about adoption rates and to ensure reference-ability for marketing and future UT sessions. Lastly, but most importantly, reach out to your charter customers to follow up on the end results now that they are using features in their daily workflows.
Our deliverables vary depending on what type of testing has been done. For conceptual testing, deliverables typically include a Summary & Recommendations document that may include a deep dive to identify what the core root causes were for failures based on actual observations, conversations and concrete recommendations to improve the user experience. Recommendations are categorized into Urgent, Important or Nice-to-Have to help the product team accurately prioritize, and the overall scores, statistics and notes are presented. This document is uploaded into the wiki, along with any audio/video recordings to be archived for reference. If you have the resources to convert the recordings into a transcript, it can be quite helpful for easily searching and quickly scanning.
Deliverables during sprint testing usually include the Usability Observation List that quickly identifies the points of failure and provides recommendations to improve the user experience. These findings are communicated in the daily scrum and uploaded to the wiki for future reference (along with any audio/video recordings).
Finally, the post-launch deliverables can include a retrospect meeting, updating the feature enhancement list based on customer feedback, and identifying reference-able customers. This last deliverable can lead to highlighting a customer in a case study, or inviting interested customers to participate in future UT sessions.
As you can see, user testing is an invaluable part of the agile methodology that can help you better understand your customer’s needs and make your products and apps more useful to users. No matter how or when you test, you’ll reap many benefits such as early problem detection, increased user satisfaction, reduced support costs and increased efficiency for users. And you’ll constantly be amazed by the innovative ideas for enhancements and new features that come directly from your customers.
The bottom line is that if you are developing in agile, then you should be incorporating some form of usability testing into your iterative process. It does not need to be a formal or expensive process – it just needs to capture user feedback in some way so that you can ensure a usable product. As Admiral Grace Hopper once said, “One accurate measurement is worth more than a thousand expert opinions.”