How Many customers Should you observe
March 10, 2011 Leave a comment
Full Article here
Asking Customers What they Want?
It’s not as simple as asking customers what they want and building the product or adding features. Steve Jobs famously said in 1998:It’s really hard to design products by focus groups. A lot of times, people don’t know what they want until you show it to them.
I’d especially agree with the focus group part.
This does not mean it’s never appropriate to ask customers what they want. Techniques like conjoint analysis work well when prioritizing features that are well understood and have a certain perceived value (e.g. features in a car). It’s less effective for solving problems customers don’t know they have.
We went to a customer’s house or workplace and watched them do what they do and recorded their behavior and problems they encountered and how they solved them. Data from follow-me-homes were used for new product ideas and improving existing products.
We tried not to interfere and instead took copious notes about how customers used the product–and we certainly didn’t ask customer what features they wanted in the product. The notes from these sessions took many forms, but would contain information like this:
- Customer used a calculator to figure out the sales-tax instead of using the product feature.
- Customer exports data from their Point-of-Sale register into QuickBooks at the end of each day.
Designers, researchers and even managers took turns following customers home. The notes were categorized and grouped and sorted. After just a few customer-visits, some observations and behaviors were listed many times. For every new customer there would be at least one or two new observations.
While we were seeing redundancy, we wanted to know how many customers we needed to follow-home before we had a reasonable picture of the problems so we could prioritize them, come up with solutions and build a better product. At best there were rules of thumb of between 5 and 20 of for each customer type.
While the idea of observing customers to generate innovation[pdf] is popular, there is little guidance on the number of customers you should plan on observing.
Sample Size Calculations for Discovering Product Opportunites
To answer this question on how many customers we need to observe to find opportunities for innovation, we can use the same approach that is used in finding the number of users you’d need to detect problems in an interface. It’s an approach Jim Lewis will explain in our forthcoming book and is also subject to vague and controversial sample size conventions.
To use it, we need to define two things.
- Issue frequency: Pick the minimum percentage of customers that will have the behavior or problem that you want to have a high likelihood of observing. For example, you might decide that you want to reliably detect (at least once) critical events that will happen to one out of five (20%) or more of your customers.
- Chance of observing the behavior: Specify how certain you need to be of seeing (at least once) the issues you identified in Step 1. For example 90%, 85% or 80%.
For example, let’s say you want to know how many customers you need to follow-home to be 90% sure you’ve seen, at least once, problems that 20% or more of users will exhibit (like not knowing a certain feature exists).
To find out you use a modification to the binomial probability formula.
Sample Size = log(1-.90)/log (1-.20) = 10.3 and to be safe we can round up to 11
So after following 11 customers home, you’ll have at least a 90% chance of observing (at least once) behaviors and problems that 20% or more of customers will have.