post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Are you really understanding your users or going through the motions?

Why can’t we get accessibility right? Claire Bauden (SVP Global Product Design, Global Payments Inc.)

Ditch the PRD template and embrace the PRD checklist

How to price your subscription product: insights and examples

Can product operations and innovation be friends?

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Sunday Rewind: The good, bad, and ugly of OKRs by Jenny Herald

Product Management

Louron Pratt

<p>In this spotlight session at #mtpcon London 2023, <span style="text-decoration: underline;"><a href="https://www.linkedin.com/in/dmcarlile/">David Carlile</a></span>, Sr Director, Product Strategy at Optimizely shared insights on how we can reduce risk and fail faster by building experimentation into the product development lifecycle. Watch the video in full, or read on for his key points.</p>
<h2>Product development and experimentation</h2>
<p>David walks us through the product development process to fail effectively and consequently build better features. Here are some examples of questions to ask at each stage:</p>
<ul>
<li><span style="font-weight: bold;">Discover: </span>Which features should be added to our roadmap and why?</li>
<li><span style="font-weight: bold;">Design:</span> What should these features look like, and how should they be built?</li>
<li><span style="font-weight: bold;">Build:</span> What is the most efficient and cost-effective way to build this feature?</li>
<li><span style="font-weight: bold;">Validate:</span> How can we validate that this feature works?</li>
<li><span style="font-weight: bold;">Roll-out: </span>How can we roll out this feature in the most efficient way, whilst reducing risk?</li>
<li><span style="font-weight: bold;">Iterate:</span> How can we optimise this existing feature?</li>
</ul>
<p>This is a very similar process to the experimentation side of product management, he explains:</p>
<ul>
<li><span style="font-weight: bold;">Discover:</span> Test to learn: Painted door tests to validate demand</li>
<li><span style="font-weight: bold;">Design: </span>Test to decide: User studies and rapid prototyping to validate feature design</li>
<li><span style="font-weight: bold;">Build: </span>Use feature flags to limit blast radius</li>
<li><span style="font-weight: bold;">Validate: </span>Test to measure: roll out test to prove behavioural change and business impact</li>
<li><span style="font-weight: bold;">Roll-out: </span>Staged rollouts to monitor the quality and performance of the experience</li>
<li><span style="font-weight: bold;">Iterate: </span>Test to learn: Experiments on iterations of features to optimise user experience and business impact</li>
</ul>
<h2>Common challenges</h2>
<p>When going through this process for both the product development process and experimentation, David shares a few common challenges we may face.</p>
<p><span style="font-weight: bold;">Ship-first mentality:</span> This would mean not asking enough questions about whether we’re shipping the right thing. David explains that we can use testing to validate our assumptions before we invest in building products.</p>
<p><span style="font-weight: bold;">Productivity threat: </span>Often, when we’re looking to build a rigid case for a product or feature, it can be seen as a blocker to productivity. He encourages us to push back on the productivity threat to ensure the validation of new products and features.</p>
<p><span style="font-weight: bold;">Org-wide buy-in:</span> It can be difficult to influence the whole business to invest in this new process, especially if a business has a built-in product, feature, or way of working.</p>
<p><span style="font-weight: bold;">Validation only:</span> David adds how this development cycle can be perceived as the final stage to simply validate the product or feature that we were already going to proceed with. It’s important to integrate this validation stage earlier into the product development lifecycle, he explains.</p>
<h2>Build the right ‘thing’</h2>
<p>To ensure that we’re building the right products, David breaks down three product test types. Also, he includes examples to showcase what successful companies have been doing to experiment and validate well, such as media platform, Medium.</p>
<ul>
<li>Test to learn: Learn about user demand and interest in new features. Learn how users respond to your feature and why they respond the way they do.</li>
<li>Test to decide: Decide on feature design, build and roll-out strategy.</li>
<li>Test to measure: Validate the impact of the feature on user experience and/or key business metrics.</li>
</ul>
<p><img decoding="async" src="//wp-content/uploads/2023/12/b589532b-f793-4d77-87fe-c6ab13d2880a_Screenshot202023-11-1420at2010.34.15.jpg" /></p>
<p>In terms of collaboration, David breaks down the key responsibilities of each team.</p>
<ul>
<li>Product teams &#8211; Testing to learn: Exploratory testing, validates the demand.</li>
<li>Engineering &#8211; Testing to decide: Feature flagging, easy rollouts, decouples code release from deployment.</li>
<li>Marketing &#8211; Testing to measure: Message validation, personalisation, quantifying the impact.</li>
</ul>
<p>This showcases how businesses can optimise experiences, and consistently test across various teams and channels from end-to-end.</p>
<p><img decoding="async" src="//wp-content/uploads/2023/12/7f2f2e09-75ac-4856-a6e4-da2b8227e8a3_Screenshot202023-11-1520at2009.50.22.jpg" /></p>
<h2>Measuring continuous optimisation</h2>
<p>David says that constant iteration and repetition are key. He lists some metrics we can consider to validate program performance:</p>
<h3>Program metrics</h3>
<ul>
<li>Test velocity</li>
<li>Conclusive rate</li>
<li>Win rate / Lose rate</li>
<li>Roll-out and roll-back rate</li>
<li>Test set-up time</li>
<li>Tests per initiative</li>
<li>% product discovery tests</li>
<li>Test duration</li>
</ul>
<p>Although these metrics are important, David mentions that value metrics are equally critical to provide an estimation of what effect a new product or feature is having.</p>
<h3>Value metrics</h3>
<ul>
<li>Positive impact</li>
<li>Negative impact</li>
<li>Annual estimated impact</li>
<li>ROI</li>
</ul>
<h3>Key takeaways</h3>
<p>Wrapping up his talk, David shared some key takeaways from using metrics to drive value and validate business ideas</p>
<ul>
<li>Program metrics should evolve over time: they should reflect cultural challenges and evolve with your experimentation maturity</li>
<li>Program metrics can be harder to collect: Extra work may be required to capture and catalogue project data and operational metrics</li>
<li>Program metrics serve as a directional compass for culture change: Establish baselines, implement cultural practices, and give time to remeasure</li>
<li>Experimentation value includes achieved gains and avoided losses: Experimentation should lead to improvement and catch potential mistakes.</li>
</ul>
<hr />
<p><span style="font-style: italic;">Try Optimizely Feature Experimentation for free and start shipping confidently with</span></p>
<p><span style="font-style: italic;">Rollouts &#8211; Optimizely’s free feature flagging and experimentation solution. </span><span style="font-style: italic; text-decoration: underline;"><a href="https://www.optimizely.com/free-feature-flagging?utm_source=mtp&amp;utm_medium=referral&amp;utm_campaign=free-rollouts" target="_blank" rel="noopener">Create a free account here</a></span></p>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

core/freeform

Reduce risk, fail faster: Experimenting across the product development lifecycle