post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Are you really understanding your users or going through the motions?

Why can’t we get accessibility right? Claire Bauden (SVP Global Product Design, Global Payments Inc.)

Ditch the PRD template and embrace the PRD checklist

How to price your subscription product: insights and examples

Can product operations and innovation be friends?

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Sunday Rewind: The good, bad, and ugly of OKRs by Jenny Herald

Product Management

Jing Hu

<p><em>This article is part of our AI Knowledge Hub, created with <a href="https://www.pendo.io/?&amp;utm_source=mtp&amp;utm_medium=website&amp;utm_campaign=ai-hub-post" target="_blank" rel="noopener">Pendo</a>. For similar articles and even more free AI resources, <a class="c-link" href="https://www.mindtheproduct.com/ai-knowledge-hub/" target="_blank" rel="noopener noreferrer" data-stringify-link="https://www.mindtheproduct.com/ai-knowledge-hub/" data-sk="tooltip_parent">visit the AI Knowledge Hub now.</a></em></p>
<p>This deep dive guest post written by <a href="https://www.linkedin.com/in/jhuu/" target="_blank" rel="noopener">Senior Global Product Manager, Jin Hu</a>, delves into the opportunities that lie ahead for product managers with the arrival of ChatGPT and other advances in AI.</p>
<p>Hi there! As a product person in back office operations, I always try to find ways to improve efficiency, even if it means eventually eliminating some jobs (please don’t hate me, this is often a consequence of doing my job well). Lately, I’ve been really interested in exploring how <a href="https://www.mindtheproduct.com/the-chatgpt-hype-separating-reality-from-fiction-in-product/" target="_blank" rel="noopener">GPT</a> can help streamline my work process and improve the productivity of the products I’m responsible for.</p>
<p>A few days ago I came across a paper titled “GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models” by authors from OpenAI, OpenResearch, and the University of Pennsylvania. It’s been getting a lot of buzz, with several YouTubers discussing its findings. I highly recommend reading the paper or checking out some of the videos, I left some links at the end of this article.</p>
<p>I initiated a chat with GPT out of pure curiosity regarding a table mentioned in the paper (Screenshot below). This table describes the regression of occupation-level, human-annotated exposure to GPTs on skill importance for each skill in the O*NET Basic skills category, plus the programming skill.</p>
<p>To explain this to a 5-year-old, the higher the number in the table the more the skill is likely to be augmented/exposed by different levels of LLM involvement (𝛼, 𝛽, 𝜁. search the paper for more detail). Exposed here means “<span style="font-style: italic;">a measure of whether access to an LLM or LLM-powered system would reduce the time required for a human to perform a specific DWA or complete a task by at least 50 percent.</span>”</p>
<p><img decoding="async" src="//wp-content/uploads/2023/05/840f0a6f-870f-4fc2-a957-abd9d7094d0a_0d8qMYq78_lV7_guv.jpg" alt="" /></p>
<p>The conclusion made in the paper on this table is the same as you can see here.</p>
<p>The higher the number, the augmentation is stronger, and the negative number means the LLM could not be of assistance to augment the specific skill.</p>
<p>Now we have set the stage. Let’s continue to follow my journey.</p>
<p>I wanted to know how product managers’ day-to-day tasks mapped with the O*NET Basic Skills and find out how many of them required “science and critical thinking”. Here’s a table that GPT generated based on my requests:</p>
<p><img decoding="async" src="//wp-content/uploads/2023/05/e02c77ab-0068-4434-bcf5-3ee244596e42_0Fm1Mo6Uky1aqZIaq.jpg" alt="" /></p>
<p>Table 1. A list of generic <a href="https://www.mindtheproduct.com/the-role-of-an-ai-product-manager/" target="_blank" rel="noopener">product manager</a> tasks mapped with O*NET Basic Skills also other skills needed identified by GPT. Each row is the skills required to complete the respective task successfully.</p>
<p>Please note, this is a list of tasks for an individual contributing product manager, not for people managers. Also, I am well aware this isn’t a comprehensive list of tasks nor the O*NET skills matched perfectly. I still decided to publish it as-is, given this isn’t the main point to make here and it’s good enough.</p>
<p>There were a few “errors” from GPT that I had to correct during the chat; otherwise, the result was very impressive. Here are three types of errors made:</p>
<ul>
<li>No clear explanation on why a task can/can’t be assisted by GPT, e.g., GPT initially said it couldn’t help with user research. I argued otherwise. I broke down the research process into more specific tasks and pointed out which tasks could use GPT for productivity.</li>
<li>GPT picked up some more niche requirements for product managers and not necessarily applicable to all product managers, e.g., GPT initially suggested product managers are responsible for pricing.</li>
<li>GPT failed to pick up some generic product manager&#8217;s day-to-day tasks, e.g., <a href="https://www.mindtheproduct.com/tips-for-successful-stakeholder-management/" target="_blank" rel="noopener">stakeholder management</a>.</li>
</ul>
<p>From time to time, there are similar errors in one answer from GPT. When I asked it to correct the error, it normally also picked up the other ones and provided a better version of output.</p>
<p>Based on the statement from the paper and combined the table from GPT, I conclude more than half of the PM tasks won’t be exposed to GPT4 (current algorithm). Saying this, I wanted to know in more detail, exactly which are the tasks the GPT can or can not help with. After a while, our chat grew closer to what I was looking for. Here’s the result, a table listing generic product managers&#8217; day-to-day tasks, whether GPT can be of assistance, and how.</p>
<p><img decoding="async" src="//wp-content/uploads/2023/05/372daba9-79a7-4a83-9727-13e8c5ba6c76_09LWx6OVbBkHMW0KP.jpg" alt="" /></p>
<p>Table 2. Continue using the same set of the generic product management tasks, based on the description of each task to see if GPT recognise this is something GPT can be of assistance, and in what way.</p>
<p>As you can see here, there is no single definitive “Yes” in the table, only some “Maybes.” You might also find that the “Maybe” in this table isn’t exactly the same tasks as the tasks that require <span style="font-weight: bold;">Critical thinking and science as skills</span>. I think two reasons, first is that even a big part of a product manager task requires Critical thinking and scientific research skills, but they still can be broken down into smaller parts that could be assisted by GPT.</p>
<p>Second, the context between Table 1 and Table 2 is different. Table 1 is about mapping between the O*NET Basic Skills and the product management tasks based on a statement from the paper. Whereas there is no pre-defined list or standard for GPT to reference on when we generated Table 2, all information is gathered based on what’s available for the current GPT version.</p>
<p>&nbsp;</p>
<h2>Until next time …</h2>
<p>In conclusion, our conversation was a fascinating exploration of the <a href="https://www.mindtheproduct.com/the-most-important-soft-skills-for-product-managers/" target="_blank" rel="noopener">essential skills of product managers</a> and how GPT could assist with them. Here is a list of tasks you can try getting help from GPT. Some of them require GPT to have access to more context, eg. the detail and background of the program you’re working on, etc. Also, some of the work that could be done by GPT should in theory be further improved by apps built on top of AGI with specific instructions.</p>
<ol>
<li>Conduct market research: GPT can assist with this task by analyzing online forums, social media, and other sources of data to identify trends and patterns in consumer behaviour.</li>
<li>Identify user needs and pain points: GPT can assist with this task by analyzing user feedback and identifying common themes and issues that need to be addressed.</li>
<li>Create user stories: GPT can assist with this task by generating user stories based on user feedback and requirements. The premise is that GPT has access to all the context of the company and the program you are part of.</li>
<li>Develop product positioning and messaging: GPT can assist with this task by analyzing market trends and consumer behaviour to develop effective product positioning and messaging.</li>
<li>Create a <a href="https://www.mindtheproduct.com/3-myths-of-product-adoption-martina-lauchengo/" target="_blank" rel="noopener">Go-To-Market</a> strategy: GPT can assist with this task by analyzing market trends and consumer behaviour to develop effective Go-To-Market strategies.</li>
<li>Product marketing: GPT can assist with this task by generating marketing copy and content based on market trends and consumer behaviour.</li>
</ol>
<p>While GPT could provide some assistance in certain areas, human input and decision-making were still crucial in ensuring the success of product management. It was clear that while AI has the potential to revolutionize the field of product management, it could not yet fully replace the skills and expertise of a human product manager.</p>
<figure style="width: 1067px" class="wp-caption alignnone"><img loading="lazy" decoding="async" src="//wp-content/uploads/2023/05/1c4bf712-8f19-4a85-be45-91155dfb762f_06pxRlOw2fYqfDFdA.jpg" alt="" width="1067" height="1600" /><figcaption class="wp-caption-text">Photo by Ray ZHUANG on Unsplash</figcaption></figure>
<h2 id="927e" class="kb kc hw be kd ke kf kg kh ki kj kk kl km kn ko kp kq kr ks kt ku kv kw kx ky bj">Appendix- O*NET Basic Skills</h2>
<h3>Content</h3>
<p>These are background structures needed to work with and acquire more specific skills in a variety of different domains.</p>
<ul>
<li><span style="font-weight: bold;">Reading Comprehension</span> — understanding written sentences and paragraphs in work-related documents.</li>
<li><span style="font-weight: bold;">Active Listening</span> — giving full attention to what other people are saying, taking time to understand the points being made, asking questions as appropriate, and not interrupting at inappropriate times.</li>
<li><span style="font-weight: bold;">Writing</span> — communicating effectively in writing as appropriate for the needs of the audience.</li>
<li><span style="font-weight: bold;">Speaking</span> — talking to others to convey information effectively.</li>
<li><span style="font-weight: bold;">Mathematics</span> — using mathematics to solve problems.</li>
<li><span style="font-weight: bold;">Science</span> — using scientific rules and methods to solve problems.</li>
</ul>
<h3>Process</h3>
<p>These are procedures that contribute to the more rapid acquisition of knowledge and skill across a variety of domains.</p>
<ul>
<li><span style="font-weight: bold;">Critical Thinking</span> — using logic and reasoning to identify the strengths and weaknesses of alternative solutions, conclusions, or approaches to problems.</li>
<li><span style="font-weight: bold;">Active Learning</span> — understanding the implications of new information for both current and future problem-solving and decision-making.</li>
<li><span style="font-weight: bold;">Learning Strategies</span> — selecting and using training/instructional methods and procedures appropriate for the situation when learning or teaching new things.</li>
<li><span style="font-weight: bold;">Monitoring</span> — monitoring/assessing performance of yourself, other individuals, or organizations to make improvements or take corrective action.</li>
</ul>
<h3>Cross-Functional Skills</h3>
<p>The paper also selected <span style="font-weight: bold;">Programming</span> from the list of cross-functional skills because of our prior knowledge about the models’ ability to code.</p>
<ul>
<li><span style="font-weight: bold;">Programming</span> — writing computer programs for various purposes.</li>
</ul>
<h3>Serious stuff- LLM Assistance And Research Statement</h3>
<p>GPT-4 and ChatGPT were used for writing, coding, and formatting assistance in this project.</p>
<p>Additional research is required to investigate the wider implications of LLM progress, including their potential to enhance or replace human labour, the effect on job quality, inequality impacts, skill development, and other outcomes. By attempting to comprehend the abilities and potential consequences of LLMs on the workforce, policymakers and stakeholders can make better decisions to navigate the complex AI landscape and its role in shaping the future of work.</p>
<h1 id="b9a7" class="kb kc hw be kd ke kf kg kh ki kj kk kl km kn ko kp kq kr ks kt ku kv kw kx ky bj" data-selectable-paragraph="">Reference and useful links</h1>
<ul class="">
<li id="4944" class="jq jr hw ih b ii kz im la iq lx iu ly iy lz jc jv jw jx jy bj" data-selectable-paragraph=""><a class="af lf" href="https://arxiv.org/abs/2303.10130" target="_blank" rel="noopener ugc nofollow">GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models</a></li>
<li id="cce8" class="jq jr hw ih b ii jz im ka iq ib iu ic iy id jc jv jw jx jy bj" data-selectable-paragraph=""><a class="af lf" href="https://www.onetonline.org/find/descriptor/browse/2.A" target="_blank" rel="noopener ugc nofollow">O*NET Basic skills category</a></li>
<li id="ed0d" class="jq jr hw ih b ii jz im ka iq ib iu ic iy id jc jv jw jx jy bj" data-selectable-paragraph=""><a class="af lf" href="https://www.youtube.com/watch?v=5SgJKZLBrmg&amp;t=669s" target="_blank" rel="noopener ugc nofollow">GPT 4 Can Improve Itself — (ft. Reflexion, HuggingGPT, Bard Upgrade and much more)</a></li>
</ul>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

core/freeform

The future of product management: Insights from ProductTank San Francisco

Product lessons learned making early moves with AI in media: Lindsey Jayne (CPO, Financial Times)

A year with ChatGPT and product innovation: Navigating the AI landscape

How to keep your head about generative AI (when everyone is losing theirs) by Claire Woodcock

What we get wrong about technology by Tim Harford

Product management in the age of ChatGPT by Yana Welinder

How Canva uses AI-powered features to drive PLG

The role of an AI product manager by Hiroki Nakamura

A gentle introduction to AI in product by Rand Hindi

Metrics don’t have all the answers: a human-centered approach to business strategy by Samihah Azim

Deep dive: Revolutionising product management with ChatGPT

Jing Hu

Comments

Read next