Revolutionizing Product Management with ChatGPT: A Deep Dive into How Chatbot Can Help Product Managers Succeed
Hi there! As a product person in backoffice operations, I always try to find ways to improve efficiency, even if it means eventually eliminating some jobs (please don’t hate me, this is often a consequence of doing my job well). Lately, I’ve been really interested in exploring how ChatGPT can help streamline my work process and improve the productivity of the products I’m responsible for.
A few days ago I came across a paper titled “GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models” by authors from OpenAI, OpenResearch, and the University of Pennsylvania. It’s been getting a lot of buzz, with several YouTubers discussing its findings. I highly recommend reading the paper or checking out some of the videos, I left some links at the end of this article.
I initiated a chat with ChatGPT out of pure curiosity regarding a table mentioned in the paper (Screenshot below). This table describes the regression of occupation-level, human-annotated exposure to ChatGPT on skill importance for each skill in the O*NET Basic skills category, plus the programming skill.
To explain this to a 5-year-old, the higher the number in the table the more the skill is likely to be augmented/exposed by different levels of LLM involvement (𝛼, 𝛽, 𝜁. search the paper for more detail). Exposed here means “a measure of whether access to an LLM or LLM-powered system would reduce the time required for a human to perform a specific DWA or complete a task by at least 50 percent.”
The conclusion made in the paper on this table is the same as you can see here that “
The higher the number, the augmentation is stronger, and the negative number means the LLM could not be of assistance to augment the specific skill.
Now we have set the stage. Let’s continue to follow my journey.
I wanted to know how product managers’ day-to-day tasks mapped with the O*NET Basic Skills and find out how many of them required “science and critical thinking”. Here’s a table that ChatGPT generated based on my requests:
Table 1. A list of generic product manager tasks mapped with O*NET Basic Skills also other skills needed identified by ChatGPT. Each row is the skills required to complete the respective task successfully.
Please note, this is a list of tasks for an individual contributing PM, not for people managers. Also, I am well aware this isn’t a comprehensive list of tasks nor the O*NET skills matched perfectly. I still decided to publish it as-is, given this isn’t the main point to make here and it’s good enough.
There were a few “errors” from ChatGPT that I had to correct during the chat; otherwise, the result was very impressive. Here are three types of errors made:
- No clear explanation on why a task can/can’t be assisted by ChatGPT, e.g., ChatGPT initially said it can’t help with user research. I aruged otherwise, I broke down the research process into more specific tasks and pointed out which tasks could use GPT for productivity.
- ChatGPT picked up some more niche requirements for product managers and not necessarily applicable to all product managers, e.g., GPT initially suggested product managers are responsible for pricing.
- ChatGPT failed to pick up some generic product managers day-to-day tasks, e.g., stakeholder management.
From time to time, there are similar errors in one answer from ChatGPT. When I asked it to correct the error, it normally also picked up the other ones and provided a better version of output.
Based on the statement from the paper and combined the table from ChatGPT, I conclude more than half of the PM tasks won’t be exposed to ChatGPT 4 (current algorithm). Saying this, I wanted to know in more detail, exactly which are the tasks the ChatGPT can or can not help with? After a while, our chat grew closer to what I was looking for. Here’s the result, a table listed generic product managers day-to-day tasks, whether ChatGPT can be of assistance, and how.
Table 2. Continue using the same set of the generic PM tasks, based on the description of each task to see if ChatGPT recognise this is something ChatGPT can be of assistance, and in what way.
As you can see here, there is no single definitive “Yes” in the table, only some “Maybes.” You might also find that the “Maybe” in this table isn’t exactly the same tasks as the tasks that require Critical thinking and science as skills. I think two reasons, first is that even a big part of a PM task requires Critical thinking and scientific research skills, they still can be broken down into smaller parts that could be assisted by ChatGPT.
Second, the context between Table 1 and Table 2 are different. Table 1 is about mapping between the O*NET Basic Skills and the PM tasks based on a statement from the paper. Where as there is no pre-defined list or standard for ChatGPT to reference on when we generated Table 2, all information is gathered based on what’s avialble for the current ChatGPT version.
Until Next Time …
In conclusion, our conversation was a fascinating exploration of the essential skills of product managers and how ChatGPT could assist with them. Here is a list of tasks you can try getting help from ChatGPT, some of them require ChatGPT to have access to more context, eg. the detail and background of the program you’re working on, etc. Also some of the work that could be done by ChatGPT should in theory be further imporved by apps built on top of AGI with specific instructions.
- Conduct market research: ChatGPT can assist with this task by analyzing online forums, social media, and other sources of data to identify trends and patterns in consumer behavior.
- Identify user needs and pain points: ChatGPT can assist with this task by analyzing user feedback and identifying common themes and issues that need to be addressed.
- Create user stories: ChatGPT can assist with this task by generating user stories based on user feedback and requirements. The premise is that ChatGPT has access to all the context of the company and the program you are part of.
- Develop product positioning and messaging: ChatGPT can assist with this task by analyzing market trends and consumer behavior to develop effective product positioning and messaging.
- Create Go-To-Market strategy: ChatGPT can assist with this task by analyzing market trends and consumer behavior to develop effective Go-To-Market strategies.
- Product marketing: ChatGPT can assist with this task by generating marketing copy and content based on market trends and consumer behavior.
While ChatGPT could provide some assistance in certain areas, human input and decision-making were still crucial in ensuring the success of product management. It was clear that while AI has the potential to revolutionize the field of product management, it could not yet fully replace the skills and expertise of a human product manager.
Photo by Ray ZHUANG on Unsplash
Appendix- O*NET Basic Skills
Content
These are background structures needed to work with and acquire more specific skills in a variety of different domains.
- Reading Comprehension — understanding written sentences and paragraphs in work-related documents.
- Active Listening — giving full attention to what other people are saying, taking time to understand the points being made, asking questions as appropriate, and not interrupting at inappropriate times.
- Writing — communicating effectively in writing as appropriate for the needs of the audience.
- Speaking — talking to others to convey information effectively.
- Mathematics — using mathematics to solve problems.
- Science — using scientific rules and methods to solve problems.
Process
These are procedures that contribute to the more rapid acquisition of knowledge and skill across a variety of domains.
- Critical Thinking — using logic and reasoning to identify the strengths and weaknesses of alternative solutions, conclusions, or approaches to problems.
- Active Learning — understanding the implications of new information for both current and future problem-solving and decision-making.
- Learning Strategies — selecting and using training/instructional methods and procedures appropriate for the situation when learning or teaching new things.
- Monitoring — monitoring/assessing performance of yourself, other individuals, or organizations to make improvements or take corrective action.
Cross-Functional Skills
The paper als have selected Programming from the list of cross-functional skills because of our prior knowledge about the models’ ability to code.
- Programming — writing computer programs for various purposes.
Serious Stuff- LLM Assistance And Research Statement
GPT-4 and ChatGPT were used for writing, coding, and formatting assistance in this project.
Additional research is required to investigate the wider implications of LLM progress, including their potential to enhance or replace human labor, the effect on job quality, inequality impacts, skill development, and other outcomes. By attempting to comprehend the abilities and potential consequences of LLMs on the workforce, policymakers and stakeholders can make better decisions to navigate the complex AI landscape and its role in shaping the future of work.
