Validation testing: A way to challenge your assumptions

Putting people at the centre of our work means trusting that they’re the experts of their own contexts and needs.

Once we identify a problem that we want to solve, we come up with a solution. But at the early stages of developing a government service, we must keep in mind that our assumptions are probably wrong - that’s because we’re not the people who will use the service. And if we can’t rely on our narrow perspectives alone to fully understand the service experience, we have to turn to the experts: the people who may eventually use the service in question.

Starting last year, we partnered with the Canada Revenue Agency (CRA) on a project that aims to break down barriers for individuals in low-income and simple tax situations, when filing taxes and accessing their benefits. We’ve completed a Discovery phase, where the goal was to gain an understanding of the barriers people encounter when filing a return, and what they need to overcome them, through research. Moving into an Alpha phase, our focus was on testing and refining potential solutions to those problems to determine how to best meet the needs of tax filers.

We reached a point where we had a clear idea of a solution, but were still a few weeks away from turning that concept into a functioning prototype. Non-functional design prototypes are lightweight and don’t take long to create, so rather than waiting until we had “the real thing,” we decided to test what we had: the core concept. We did this by conducting validation testing, an approach that’s allowed us to challenge our assumptions about what people need and expect when filing a tax return.

But wait - what’s validation testing?

Validation testing is one of the earliest ways we can test a design or prototype. The goal is to make sure that a core concept, ie. the way we decide to solve a problem, matches how real people think about the problem, and actually solves it given the contexts of their lives.

Developers like Paul use this information to refine and iterate on the technical implementation of the service:

“As the technical lead for the team, I’m in charge of informing the direction of functional prototypes and ultimately, building the live service. In developing any service, building the high-fidelity working solution takes the most energy and time. Validation testing - putting the concept in front of the people you’d like to help and getting their reaction to it - is the first step in confirming that it’s actually worth building.

From the perspective of a developer, the earlier the better. Once we have a rough idea of a way to solve the problem, we should be testing it early to make sure we’re headed in the right direction. The reason to work on a multidisciplinary team is to work through the lifecycle of a problem: to understand the various pain points and then propose a solution that meets the needs of people.” —Paul Craig, Developer

Validation testing is not a replacement for usability testing, which is an indispensable research tool for uncovering issues with a digital interface in addition to other considerations that impact usability. Validation testing is useful when you don’t have a functioning interface, as it allows the team to zero in on the core concept: what we’re asking someone to do, and whether or not that makes sense in the context of their lives.

How validation testing has helped us so far…

Our individual views of the world are limited, so part of designing services for others involves testing them repeatedly with the people who will eventually use them. We do this by asking people to guide us through the service from their perspective, and teach us about what they need to achieve their goals.

You don’t need a fully functional website to conduct validation testing – you just need something that communicates the concept well enough that participants can picture themselves using it. Early last month, we put together a clickable prototype that communicated our service concept in broad strokes, and tested it with participants at community food banks to identify problems with our initial assumptions and validate whether or not the service would meet their needs. It was a humbling and helpful experience to course-correct some of our assumptions.

One of our initial assumptions was that tax filers file their own returns by themselves. Through validation testing, we learned that many people delegate tax filing to another person in their family - this taught us that our assumption that the person who received the invitation letter to use the service would also be the person using the online service wasn’t entirely correct.

Allowing our team to challenge fundamental assumptions about the core concept of a service helps us understand whether or not we’re moving in the right direction early on, when it’s easy to change course. Ultimately, this limits risk and saves us time and money in the long run.

What’s next?

Even after we come up with a solution based on research with the people who would use our service, we recognize that our solution is just a hypothesis.

As we move forward with our project into Beta, we expect to test that hypothesis over and over. We plan to iterate further with more research, using methods such as usability testing.

Want to learn more about validation testing and how to incorporate it into your work? Email us or reach out on Twitter - we’d love to hear from you.