Scared of losing your job to ChatGPT? Become a Product Analysis superstar with the help of AI
Posted by Vladimir Kamchatkin
I’m an inDrive product analyst. If you looked at my computer, you’d see over twenty conversations currently open with ChatGPT, where I chat with the AI about work and life. And now, I want share my thoughts on how this new tool can help analysts on the job and improve our everyday lives.
It’s important to consider ChatGPT as yet another information source. And like any resource, the information it provides needs to be approached and evaluated with a critical eye. As a bonus, I’ll share some additional resources about ChatGPT and artificial intelligence at the end of the article.
ChatGPT: A tool on steroids
I sent my first message to ChatGPT-3.5 around the beginning of March 2023. At first, I was just messing around with different queries, largely unrelated to work. But then, I started thinking: how can I use the neural network to solve tasks related to product analytics?
Going in, I knew that ChatGPT wouldn’t be able to answer all of my questions with 100% accuracy. That’s why, ideally, I like to know at least 50% of the answer myself before asking the AI bot to answer it too. Then, I can immediately verify whether the AI is sharing accurate information or just coming up with something on its own. Of course, if I don’t have enough knowledge on a subject, I can always double-check the bot’s answers against another online resource, or consult with an expert.
It’s important to recognize that artificial intelligence isn’t an authority on every subject. ChatGPT is not the single source of truth, and it’s not going to do our work for us. Instead, approach it like you would an assistant: let it do the busy work, while you focus on the creative side.
When allocating our time properly with ChatGPT, we end up with the formula X + Y = Y. Here, X represents the tasks you already know how to do and can easily delegate, while Y represents new, more innovative tasks. The common phrase, “20% of your effort produces 80% of the results” applies here too. In fact, it’s a formula we can use with any new instrument.
To me, ChatGPT is a just new tool — much like Excel. And while Excel speeds up work by calculating spreadsheet data, it hasn’t managed to replace a single profession.
ChatGPT: A partner and pal
While ChatGPT can serve as a sort of virtual assistant, it also makes for a great brainstorming partner when approached on equal footing. Working with the AI can be a collaborative process, where, if you provide the right prompts, the chat becomes a discussion.
Here’s a great lifehack I’ve found for improving my queries with the help of ChatGPT. Imagine that ChatGPT is your bright, energetic assistant who always goes above and beyond. If you take the time to sit with it and give it enough clarifying information, you can help it become better familiar with, and more equipped to solve, the task at hand.
Here’s an example of a self-improving prompt that can expand your understanding of a task through iterations of questions and answers.
“I want you to become my prompt creator. Your goal is to help me craft the best possible prompt for my needs. The prompt will be used by you, ChatGPT. You will follow the process below:
1. Your first response will be to ask me what the prompt should be about. I will provide my answer, but then, we will need to improve it through continual iterations following the steps below.
2. Based on my input, you will generate 3 sections.
a) Revised prompt: Rewrite the prompt in a way that is clear, concise, and easily understood by you.
b) Suggestions: Suggest other details that should be included in the prompt in order to improve it.
c) Questions: Ask any relevant questions pertaining to the additional information that I should provide in order to improve the prompt.
3. We will continue this iterative process. I will provide you with additional information and you will update the prompt in the revised prompt section until it’s complete.”
This is an example of how ChatGPT essentially comes with built-in support. Plus, the support can even teach us how to use it. 🙂
Who will replace us: AI or AI-informed humans?
I don’t think that AI will replace people. Rather, it’s the people who know how to use AI who will replace the ones who don’t. After all, workers who embrace ChatGPT in their work will be able to complete tasks more quickly and efficiently.
Soon, one analyst with advanced knowledge of ChatGPT may be able to replace the work of two. And it’s this very thought that has compelled me to start considering how the neural network can be applied to my professional tasks. It’s also what inspired me to write this article.
A study from OpenAI (the company behind ChatGPT) found that the neural network can complete 23–25% of a programmer’s tasks. So, it’s clear that the tool hasn’t succeeded in replacing people yet — but it has made our lives easier. Of course, things can change quickly. By the time this article is released, AI may have already replaced my job as an analyst. But I doubt it.
Another common prediction is that ChatGPT will actually create more jobs, rather than steal them. Supporting this idea, I recently saw a job posting for a “Python Developer working with ChatGPT”. And, in a sense, I’m already performing the role of a prompt engineer myself. After all, I’m creating chats and improving the AI with my queries (see the lifehack above). Perhaps, in 2–3 years, we’ll see vacancies with “Command of ChatGPT” listed in the job requirements.
What would need to happen for ChatGPT to replace people? First of all, there would need to be a system for validating the AI’s responses. At the moment, we need humans to do this job, and the neural network isn’t replacing them any time soon.
Let’s also consider tasks that gradually increase in difficulty, from intern to senior level: as we move up the hierarchy, the tasks change from “simple web searches” to handling and uncovering new ideas that cannot be found on Google. ChatGPT can’t quite reach this point on the hierarchy yet.
Moreover, behind every new tool is an immense system that requires constant attention. And for that, we need working capacity. So, a few questions arise: Is it better to pay for an AI to do the job or just hire a senior developer? How would the neural network even know when something’s broken?
Now, let’s move on and explore some of the features of ChatGPT.
Working with ChatGPT
Below, you can see the difference between how ChatGPT responds to a simple request versus one with more context. In my second prompt, I specify the product, audience, and goal, along with other pertinent information.
Next, let’s look at a real-life analytical task: Write an SQL query for conversion from website visit to banner click.
Based on ChatGPT’s response, we can immediately identify a few ways to improve the query.
1. Although the AI bot provides us with a general template to use, it would be best to provide more information about our specific database structure.
2. We need to consider the order of events.
3. ChatGPT actually tells us what we should look out for in the data: “Note that, in the real world, not all banner clicks necessarily correspond to website visits.” Reading this response, it’s easy to picture ChatGPT as a work buddy giving you suggestions on the job.
Within a month of working with ChatGPT, I began relying on the Google search engine less and less. Why? I found that ChatGPT provided more detailed answers (as you can clearly see from the SQL example above). In comparison, Google’s answers are quite dry, meaning that if you want more information, you need to be thorough.
One thing to keep in mind about ChatGPT is the relevance of the information it provides. The neural network was trained on data from September 2021. So, if you’re working with a newer library, for example, it might respond with “I am not familiar with this library or framework. It’s possible that it is either new, specific to a certain domain, or not widely known in the general software development community.”
However, if you provide a link to an article about the library, ChatGPT will start to explain it. In fact, the tool can analyze code from GitHub and provide information about a library that it hadn’t previously known about. Basically, the neural network learns it on the spot.
Another important aspect of the ChatGPT interface is that it draws upon the current chat’s conversation history to understand context. The maximum token limit for a chat session is around 3–4 thousand tokens for English. Essentially, it trains itself and gains context from the information you provide it over the course of a single chat!
In fact, you can use ChatGPT as a sort of long-term memory storage, a book of ideas.
ChatGPT-3.5 vs ChatGPT-4
In mid-April, I made the switch from the free version of ChatGPT (3.5) to the paid version (4). As I began creating more and more queries, and increasingly relying on the tool, it started taking longer to answer and I was noticing more mistakes.
At the same time, ChatGPT was getting increasingly popular among all users. Consequently, the developers had to implement some limits on use. Of course, for product analysts looking to speed up their work tasks, it’s best to avoid the trouble and just buy the full version.
Here’s what I noticed after making the switch:
1. ChatGPT-3.5 responds faster than ChatGPT-4.
2. That said, there’s a notable difference in the quality of the responses: ChatGPT-4 provides more detailed answers, and even asks for more context to ensure the best results.
3. The paid version also has plugins that increase its capabilities:
4. When you give GPT-4 additional context, it can create truly competent responses that can be applied to work tasks. For example, the tool once generated a range a great options for an A/B test of different CTA designs for an important button. Moreover, it could immediately predict which one would come out on top, and why. Plus, all of ChatGPT’s suggestions have resulted in an uptick in conversion.
How can I turn ChatGPT into a superpower for product analytics?
Let’s ask ChatGPT itself:
ChatGPT can help generate ideas for hypotheses, research findings, and A/B testing metrics. For example, if I give it enough context about a certain hypothesis and product, then ask, “What’s the key metric for this A/B test?”, it almost always gives me an impressive and applicable answer. And also I can ask ChatGPT “In what cases can a feature be rolled out without an AB test?”
As I mention earlier, for me, ChatGPT has entirely replaced both Google and many modern online courses. However, we cannot draw a clear line between these individual concepts. For example, my colleague uses the browser Bing for its AI feature. In this way, Bing has already managed to combine both standard search engines and AI.
Online courses also make for an interesting case study. As a little experiment, I wanted to know whether ChatGPT could handle typical tasks from one of these online courses. However, since ChatGPT-3.5 uses an outdated Python function for date conversion, I ended up with an answer that was only half of the correct value. Basically, it reduced revenue by half. If I hadn’t been checking the code myself, it would have been easy to miss some of these mistakes (which again illustrates the current need to validate responses and data from the AI bot).
Finishing with a case
We want to examine the product’s development history and its impact on business metrics. ChatGPT offered some great suggestions on how to do that:
“To address the problem of low activity among contracted drivers, I suggest taking the following steps:
1. Conduct a survey: Perform a survey of contracted drivers to assess their experience using the app. This can help you identify any pain points or areas of improvement.
2. Optimize the acquisition process: Make the acquisition process more user friendly and productive. This could be done by simplifying the application processes, providing clear instructions and support, and also making sure the platform is accessible and easy to use.
3. Implement a fraud-prevention strategy: Develop a strategy to prevent fraud. This can include verifying drivers’ background information, documents, and IDs, and monitoring acquisition indicators.
4. Provide adequate training and support: Offer new drivers comprehensive training and support to help them become familiar with the app and learn how to use it effectively. This could involve introductory lessons, regular checks, or the creation of a website for driver support.
5. Encourage activity: Provide the drivers with incentives to complete a given number of rides or maintain a particular rating. This can help motivate the drivers to use the app more often and increase engagement.
6. Monitoring and assessments: Monitor and evaluate the effectiveness of the existing strategy for increasing the activity of contracted drivers. Conduct regular analyses of indicators, such as the number of active drivers and completed rides, and make adjustments where necessary.
By taking these steps, you can improve the process for acquiring drivers for the app, increase user friendliness, and boost driver activity. This will result in improved indicators and increased business success.
Additional resources
ChatGPT Prompt Engineering for Developers: This online course takes 1.5 hours to complete, during which you learn tips for writing queries and Python code. I am currently working through the course myself.
Thank you for taking the time to read my article. How do you use ChatGPT in your analytical tasks? Post a comment with examples of how you’ve employed AI on the job.
