Last week,
—Head of Strategy & Analytics at Faire and author of the excellent Dan Hock’s Essays—posted this sketch showing what many of us in product development take for granted:I think this sketch says two important things about research.
The first is that we should never forget the value of user segmentation when interpreting the perceptions of people who use our products. Averaging together global user perception is useful for some statistical inferences, but it masks nuances in what different groups of people are doing and why. Those nuances are often essential for good product strategy.
My second takeaway from the sketch is illustrated by my message below:
People’s relationships with a product evolve over time as they continue to use it. Specific needs, frictions, and reactions vary between people and also change within people as users and products mature. When I first used Facebook, I used it for connecting and socializing with friends. A year later, I primarily used it to promote a business effort. These days, I almost exclusively use it to share photos of my growing son with family who live abroad.
A particularly dramatic change in product perception happens if you transition from user to employee. When you go from utilizing a product a couple of times a day for a specific user need to launching it constantly from 9 to 5 in an effort to solve problems, your thoughts and feelings about that product inevitably shift.
I’d describe one of those shifts as “tunnel vision”. You no longer enter the product with the open mind of a user, but rather the mind of a fixer with responsibility on your shoulders. You might start to miss certain aspects of the experience that are noticeable to a user because you’ve repeated them so many times. You might start to detect small anomalies in the experience tied to your work responsibilities that most users don’t care about. You might start to feel bored by particular features that most users love because your experience of them is oversaturated by excessive test usage.
As Dan puts it in his note above, these issues can lead to “bad product decisions”. The people making the decisions are a very small group of odd users who mostly use the product in ways that nobody else uses them. The good news is there’s at least one resource in the research toolkit that helps to remedy this clash.
Time to shine for qual interviews
I’ve previously written about the distinctions between qual and quant research at length, but the most relevant variable here is that qual interviews allow you to escape your own bubble as a product developer and dig into the perspective of real users. Although quantitative methods like surveys provide user perspective at a more reliable scale, they require you to write a rigid list of questions based on current team priorities and expectations. Those expectations are vulnerable to the same developer biases that we’re trying to escape here.
User interviews provide a more flexible conversational space for discovering surprises in how people think or feel when they encounter a product experience. Questions and prompts guide the discussion, but an interviewee can say whatever comes to mind, and a well-informed interviewer is free to follow up on anything that seems interesting or unexpected for the product team.
For that reason, as a team develops new concepts, it’s helpful to interview a small group of well-targeted users to get their reactions sooner rather than later. Early concept testing should be more than just a sanity check. Although qual research is often done to find support for a product team’s plans, I’d argue it’s better to adopt a more scientific approach and look for compelling evidence that might disprove the user value of those plans.
When a product concept holds up well under heavier scrutiny, you can be more confident that it’s a productive investment. Think of it like pressure testing—if prospective users throw all of their negative energy at a product concept, yet it escapes with a couple of fixable cracks and leaks, you can work on iterating the design. On the other hand, if the hinges fly off in a giant explosion, you probably need to return to square one.
Researchers shouldn’t be pessimistic naysayers but product development is expensive and it comes with giant opportunity costs. Recommending a different product direction when user feedback is unfavorable is highly worthwhile; this should in fact be seen as a primary research responsibility. If research in industry should be good at anything, it’s predicting a bad product decision before it goes too far.
“…I have already implicitly given my four basic rules for research. Let me now state them explicitly…:
1. Listen to the Gentiles
2. Question the question
3. Dare to be silly
4. Simplify, simplify”
If you enjoyed this newsletter, please spread the word by sharing it with others who might find value in it. If you’re new here, you can subscribe below.
Well said Erman! I find that one of the biggest problems is that new user onboarding is the most important part of the user experience, and it is very hard to remember what it was like to be a new user of a product that you think about every day.