post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Are you really understanding your users or going through the motions?

Why can’t we get accessibility right? Claire Bauden (SVP Global Product Design, Global Payments Inc.)

Ditch the PRD template and embrace the PRD checklist

How to price your subscription product: insights and examples

Can product operations and innovation be friends?

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Sunday Rewind: The good, bad, and ugly of OKRs by Jenny Herald

Product Management

Emily Tate

<p>At #mtpcon San Francisco, <a href="https://twitter.com/leisa" target="_blank" rel="noopener">Leisa Reichelt</a>, head of research and insights at Atlassian, took on our current approach to user research and how we take an evidence-based approach to doing completely the wrong thing.</p>
<p>Five years ago, the industry was focused on just getting people to do <em>any</em> user research. Product managers weren’t regularly talking to customers at all. These days, the drive for companies to be “customer obsessed” has brought about a new era where <em>everyone</em> is talking to customers. Designers, product managers, and researchers all think that doing using research is a primary function of their job. And while research is best when done as a team, how much time are we spending to grow our expertise in research so that we do it well?</p>
<p>While we should be out there doing research, we&#8217;re letting bad research pass for evidence, and we make decisions on data we shouldn’t be using for this purpose. We mess up research by trying to quantify things and make them seem more scientific than they really are.</p>
<h2>Perspective and Truth</h2>
<p><img loading="lazy" decoding="async" class="aligncenter size-full wp-image-13686" src="//wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow.jpg" alt="Leisa Reichelt shows how different a duck looks from its shadow" width="2048" height="1365" srcset="/wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow.jpg 2048w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow-300x200.jpg 300w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow-768x512.jpg 768w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow-1024x683.jpg 1024w" sizes="(max-width: 2048px) 100vw, 2048px" /><br />
Looking at only parts of a picture can give a warped perspective, and we can see things that appear meaningful but which aren’t. We have to make sure we are getting a big enough picture in research to make a good decision. If you see the shadow of a duck, it can look very different from the duck itself. This happens in research as well. We are getting a narrow view of data and are making decisions based on the shadow. The shadow is real, but the duck is more real, and we have to broaden our context to get the full truth.</p>
<h2>Common Research Mistakes</h2>
<h3>Customer Interviews</h3>
<blockquote><p>People can make up an opinion about anything, and they’ll do so if asked. <cite>Jakob Nielsen</cite></p></blockquote>
<p>Customer interviews are a common method of user research that teams mess up. The answers you get in interviews depend on the questions you ask, and by asking the wrong questions, you can get really excited about data that doesn’t matter at all. We ask feature-based questions such as “What is the thing you do most often in Jira?” and get narrow answers. We should be asking questions like, “What are you doing most often when you come to use Jira?” With broad context, you learn about the “why”, hear gaps and opportunities that the product may be missing, and can eventually dig deeper to probe into the features themselves.<br />
<img loading="lazy" decoding="async" class="aligncenter size-full wp-image-13684" src="//wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon.jpg" alt="Leisa Reichelt speaks at mtpcon" width="2048" height="1365" srcset="/wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon.jpg 2048w, /wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon-300x200.jpg 300w, /wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon-768x512.jpg 768w, /wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon-1024x683.jpg 1024w" sizes="(max-width: 2048px) 100vw, 2048px" /><br />
To do better customer interviews:</p>
<ul>
<li><strong>Start with wide context:</strong> don’t start with questions about your product</li>
<li><strong>Be user-centered, not product-centered:</strong> you should be able to use the same script for a competitor’s product</li>
<li><strong>Invest in analysis:</strong> get transcripts of the interviews, take the time to do affinity diagrams, and really get the full value from the research</li>
</ul>
<h3>Usability Testing</h3>
<p>We make decisions based on usability testing all the time. A popular method that has emerged is “Unmoderated Remote Usability Testing.” With this type of testing, you use software that allows you to build tests, post them, and let people complete usability tests from anywhere on their own. It allows you to get a lot of testing done without ever talking to a user directly, but it comes with a lot of pitfalls.<br />
<img loading="lazy" decoding="async" class="aligncenter size-full wp-image-13685" src="//wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing.jpg" alt="Leisa Reichelt talks about unmoderated usability testing at mtpcon" width="2048" height="1365" srcset="/wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing.jpg 2048w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing-300x200.jpg 300w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing-768x512.jpg 768w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing-1024x683.jpg 1024w" sizes="(max-width: 2048px) 100vw, 2048px" /><br />
With in-person usability tests, an experienced researcher is able to discern the difference in what a user <em>says</em> and what they <em>do.</em> They’re able to see body language that indicates when a user is frustrated, or see when a user only completed a task by randomly clicking around a screen. But with unmoderated usability testing, teams often simply download data rather than actually watch the videos of the tests; these things get turned into a simple metric of “task completed”, and quotes and ratings are pulled without the broader context.</p>
<p>To do better usability testing:</p>
<ul>
<li><strong>Make tests task not feature-based:</strong> test what people would really be doing, not the feature you wish they’d use</li>
<li><strong>Watch the videos in real time:</strong> don’t just download the data from unmoderated usability testing</li>
<li><strong>Look beyond task completion:</strong> it’s not OK that people eventually get to the right outcome &#8211; the quality of the experience along the way matters</li>
</ul>
<h3>Surveys</h3>
<blockquote><p>Faith in data grows in relation to your to your distance from the collection of it. <cite>Scott Berkun</cite></p></blockquote>
<p>Product managers love surveys. The further away you are from the point of collection of data, the more you trust it. But misleading questions and a lack of context can provide data that tells a very different story from the truth. Leisa gave an example from Atlassian Summit, where a user said in a dot-voting survey that they had great work-life balance. When she asked them about their answer, they said “I can work from home, so even though I work too much, I have flexibility.” The context gives a different perspective from the data point. Just because you can put it in a graph, doesn’t make it science. We need to be way more critical about what data we trust.<br />
<img loading="lazy" decoding="async" class="aligncenter size-full wp-image-13688" src="//wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey.jpg" alt="Leisa Reichelt shows a bad user survey at mtpcon" width="2048" height="1365" srcset="/wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey.jpg 2048w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey-300x200.jpg 300w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey-768x512.jpg 768w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey-1024x683.jpg 1024w" sizes="(max-width: 2048px) 100vw, 2048px" /><br />
To do better surveys:</p>
<ul>
<li><strong>Use qualitative techniques before quantitative:</strong> do interviews or contextual enquiry before you design a survey</li>
<li><strong>Triple test your survey:</strong> make sure people understand questions consistently and are able to answer accurately</li>
<li><strong>Use humans to analyse words:</strong> don’t rely on automated text analysis. Read the feedback.</li>
</ul>
<h2>Stop Quantifying Qualitative Research</h2>
<p>Why do we take these research methods and use them incorrectly? Leisa thinks the main reason is that we don’t have time. But if we claim to be “customer obsessed”, why are we trying to box research and empathy into the smallest amount of time possible? Take the time to do it right.</p>
<p>Leisa leaves us with the metric we should actually be aiming for. Rather than completing bad user research, just ask: <em>“In the last <strong>six weeks,</strong> have you personally observed a customer, using your product?”</em> If we’re going to make better products, the truth of our customers&#8217; experience is what we need. And truth can’t be automated or graphed. It takes time, and it takes humans.</p>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

core/freeform

Product Management Conference

Product teams in flux – Pendomonium+#mtpcon roadshow: Berlin

AI in practice at Berlin’s leading companies – Pendomonium+#mtpcon roadshow: Berlin

AI in practice at Amsterdam’s leading companies – Pendomonium+#mtpcon roadshow: Amsterdam

Product teams in flux – Pendomonium+#mtpcon roadshow: London

AI in practice at London’s leading companies – Pendomonium+#mtpcon roadshow: London

What’s next for product management in 2024? Nichole Mace (SVP Product and User Experience, Pendo)

“You’re not as competent as you think” 5 lessons from Ken Chin on navigating a product career (CPO, Spark Networks)

Perspectives from an unconventional CPO – Yuelin Li (CPO, Onfido)

The future of product management – Trisha Price (CPO, Pendo)

The future of product management in 2024 and beyond (Brian Walsh, SVP Product, Pendo)

Quantifying Qualitative Research by Leisa Reichelt

Emily Tate