post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Are you really understanding your users or going through the motions?

Why can’t we get accessibility right? Claire Bauden (SVP Global Product Design, Global Payments Inc.)

Ditch the PRD template and embrace the PRD checklist

How to price your subscription product: insights and examples

Can product operations and innovation be friends?

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Sunday Rewind: The good, bad, and ugly of OKRs by Jenny Herald

Product Management

James Gunaca

<p><span style="font-weight: 400;">Machine learning is a practice often associated with behind-the-scenes algorithms and processes which automate repetitive problem-solving where human effort would be too costly or impractical. When I hear people talking about machine learning (ML) in product management, the typical line of thought is about building ML into a <a href="https://www.mindtheproduct.com/economics-of-software-part-1-why-traditional-economics-concepts-dont-apply-to-software/" target="_blank" rel="noopener">software</a> product to create some new feature. This can lead to great product experiences that wouldn&#8217;t otherwise be possible—think photo applications like Photos on iOS which recently launched the ability to recognise pets in your photo library. This ML process runs on your phone when you&#8217;re not using it and enhances your photo library without any human effort. While fascinating, I think limiting discussions around ML to these types of product experiences shortchanges the practical applications Machine Learning can deliver for better product experiences </span><i><span style="font-weight: 400;">as well as</span></i><span style="font-weight: 400;"> business outcomes.</span></p>
<h2>Seeing an opportunity for ML</h2>
<p><span style="font-weight: 400;">A common problem product managers face is customer churn. Product managers spend weeks, months, and years crafting a product experience they believe their customers will love. The customers come, they use the product, they love it, but over time their interest may fade, their needs could change, or something else pulls them away and they churn. How much a PM knows about why their customers churn can be highly correlated with just how big of a problem churn is. For subscription products, monitoring, addressing, preventing, and mitigating churn could be the difference between product success and the abysmal failure of an entire business.</span></p>
<p><span style="font-weight: 400;">I had the fortune of working with a product org that saw the benefits of machine learning in both the end-product experience, as well as in data analytics. With a team of dedicated ML engineers, we were able to build a model which could analyse customer behaviours commonly associated with customers who were likely to churn and apply a score for customers likeliness to churn. The problem we aimed to solve was knowing which customers were likely to churn so steps could be taken to mitigate that churn. We managed to solve the first part of the problem by establishing a churn prediction score for customers and did so with a high degree of accuracy. The next problem we had to solve was determining what steps to take for those customers with high churn scores—and decide if that problem could be (or should be) solved with more ML development, or something else. </span></p>
<p><span style="font-weight: 400;">With a list of customers with high churn predictability scores, we pursued an optimal combination of CRM tactics (e.g., email, in-app messaging, push notifications) and marketing messages. When crafting messaging to mitigate churn, we considered (1) reminding customers of benefits they have (or may not be using), (2) notifying customers of new features, and (3) offering customers an incentive to renew or keep their subscription (e.g., a discount). These efforts are better constructed, delivered, and evaluated by product marketing experts rather than ML-engineers, though. The bigger challenge, then, was to bring all of <a href="https://www.mindtheproduct.com/storytelling-and-driving-accountability-with-data/" target="_blank" rel="noopener">data</a> and findings to substantiate the overall investment.</span></p>
<h2>Understanding the value ML has provided</h2>
<p><span style="font-weight: 400;">In a quarterly business review where results on ML investment were being shared, a question was posed by the leadership team—&#8221;how many customers have we prevented from churning as a result of this ML work?&#8221; This was an absolutely valid question but one the initial customer problem and eventual solution didn&#8217;t set out to solve. Was the ML work successful in identifying customers who were likely to churn? Yes. Did we figure out what messages we could send customers to prevent them from churning? Yes. So, why couldn&#8217;t we succinctly answer that question with a number? We lacked experimentation results that showed a reduction in customer churn we could attribute specifically to the churn model, separate from the marketing messages that were sent—so we set out to do that next. </span></p>
<p><span style="font-weight: 400;">Up until this point, we had also made investment to operationalise more of the churn prediction work, finding ways to integrate it more natively in our CRM tools and setting up automations—and we had built a roadmap to do all of this. We had gone too far down the path of the ML churn prediction work under the assumption that &#8220;</span><i><span style="font-weight: 400;">of course churn prediction scores will help us prevent churn!</span></i><span style="font-weight: 400;">&#8221; but we hadn&#8217;t proven it. We paused the development work and focused on substantiating further investment with data and measurement to determine if it was worth investing more, or moving on to something else. In retrospect, we had gone too far in our work before determining if this ML investment was the best use of the resources or if it was akin to a &#8220;science project&#8221;.</span></p>
<h2>Working with ML engineers</h2>
<p><span style="font-weight: 400;">Outside of the benefits and <a href="https://www.mindtheproduct.com/challenges-of-product-in-enterprise-organisations-daniel-elizalde-on-the-product-experience/" target="_blank" rel="noopener">challenges</a> of leveraging ML to improve your product, PMs should be thoughtful about their approach to working with ML engineers. Authoring and building ML algorithms requires a different skillset than what is used in more traditional software engineering roles for application and website development. But, as with working with any type of engineer, it’s a best practice to include engineers in the early stages of the product development cycle. Building a shared understanding of your customers, the problem you’re trying to solve, and why the work is important will absolutely aide in the development process, regardless of the engineering expertise needed for whatever you will build. When it comes to ML development, though, the watch-outs are to know when to cut losses and work on building something else and avoid the sunk cost fallacy. Being prepared to have this conversation with your ML engineering team or manager will help you overcome obstacles or changes in strategy.</span></p>
<h2>To sum up</h2>
<p><span style="font-weight: 400;">Machine Learning can bring a lot of benefits to a product, as well as to product marketing and engagement measures that keep your customers active. With ML engineers in place, spend time thinking about how their function can best unlock value for your customers </span><i><span style="font-weight: 400;">as well as</span></i><span style="font-weight: 400;"> your business. At the same time, re-evaluate your ML investments and decide if they&#8217;re worth pursuing further, or if your ML resources could be deployed to solve another problem with a better outcome that can be proven. Lastly, anticipate the questions you&#8217;ll get from business leaders to really prove the value or results that ML efforts yield.</span></p>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

core/freeform

The future of product management: Insights from ProductTank San Francisco

Product lessons learned making early moves with AI in media: Lindsey Jayne (CPO, Financial Times)

A year with ChatGPT and product innovation: Navigating the AI landscape

How to keep your head about generative AI (when everyone is losing theirs) by Claire Woodcock

What we get wrong about technology by Tim Harford

Product management in the age of ChatGPT by Yana Welinder

How Canva uses AI-powered features to drive PLG

The role of an AI product manager by Hiroki Nakamura

A gentle introduction to AI in product by Rand Hindi

Metrics don’t have all the answers: a human-centered approach to business strategy by Samihah Azim

Benefits and pitfalls of using machine learning to mitigate customer churn

James Gunaca

Comments

Read next