Beyond the hype: the current and future impact of ChatGPT
Intro
ChatGPT had an unbelievable diffusion in the market. It reached 1 million users in just five days after its launch and it now has 180 million monthly users, to put that in perspective, that’s more than half of Google Meet’s monthly users.
In 2023, ChatGPT and Claude had a great impact on my job. I use them daily to improve my writing, to quickly gain a brief understanding of topics I’m not familiar with, and to summarize content. In 2023, I also utilized ChatGPT to create features for products I’m working on.
As a curious product person, I asked myself whether my use of ChatGPT at work, combined with the consumption of too many news articles on AI, could cause a biased perspective on the real impact that ChatGPT and the other LLMs (Large Language Models) are having on productivity and society in general. That’s a very common type of bias. Thanks to ChatGPT, I just learned that this bias is named the Baader-Meinhof phenomenon, or frequency illusion.
My survey on ChatGPT usage
So, I decided to validate whether I’m a victim of the frequency illusion. I approached this in a pseudo-scientific manner: I created a survey and asked people in my professional network to respond to it.
Why do I consider my survey pseudo-scientific? For it to be deemed scientific, I should have defined a clear target population (for example, all people working in tech in Europe), determined an appropriate sample size to achieve statistically significant results, and employed a sampling method that minimizes selection bias.
Nevertheless, since I conducted the survey purely out of personal interest, I regard the 47 responses received from my LinkedIn network as sufficient to inform my conclusions for the time being.
Survey results
All 47 participants work in tech with the following roles:
- Product: 24
- UX / Design: 9
- Engineering: 4
- CEO / Founder: 4
- Marketing: 4
- Data science: 1
- CRM Technical Consultant: 1
As expected, most of the respondents are heavy users of LLMs, since those familiar with these technologies are more likely to be attracted to my request and respond positively. It is still interesting that 9 people took the time to read my message and respond to the survey even though they don’t actively use ChatGPT, Claude, or similar tools.
- 31 use LLM at least few times per week
- 7 people use it few times per month
- 9 tried it once or twice
Not surprisingly, ChatGPT is the most used chatbot; four people prefer Claude, and two use Google Bard.
The most revealing question was: “Could you share the last question or prompt you asked of the Generative AI tool?”
Most of the responses pertained to writing content (11 responses), learning something new (7 responses), and summarizing content (6 responses), but I also discovered some interesting uses I hadn’t thought of!
- Competitors research
- Mental health: Please be my therapist.
- Personal and entertainment: Provide a menu with its grocery list, Provide a checklist to make a party
- Coaching: Prepare for upcoming interviews
Personal reflections
All of the responses provided demonstrate the respondents’ technical ability to formulate questions optimized according to the chatbots’ needs. It is clear they are early adopters who have invested a considerable amount of effort and creativity into mastering the chatbot and employing it in creative ways.
Such behavior is typical of early adopters of any technology; they are willing to apply the necessary effort to new tools in exchange for the benefits they offer and the satisfaction of being among the first to explore innovations not yet accessible to everyone.
What Does the Future Hold?
LLM chatbots offer a great deal of value and are utilized by numerous tech professionals to perform a wide variety of tasks. I believe that in the next few years, we will witness a significantly stronger impact of LLMs on work-related productivity across various fields and for society at large.
For this to happen, there are two major impediments that need to be overcome:
Usability / Affordance
LLMs need to become more user-friendly. Chatbots do not represent a natural mode of communication in most scenarios; the examples from my survey illustrate the ability to provide instructions in a technical manner, an effort that only early-adopters are willing to take.
Chatbots like ChatGPT and Claude lack affordance; they don’t inherently show people how to use them or what they can achieve without first consulting a guide or tinkering to obtain the desired outcomes.
LLMs will only become relevant to the vast majority if they are integrated into the apps that people already use in a way that feels natural, and perhaps supported by dedicated hardware devices.”
Content curation
We, as early users, know and accept that the responses from LLMs can be highly inaccurate and unexpected; however, this is not acceptable for widespread usage. I see a need for specialized tools that provide predictable and accurate answers on specific topics. For example, today, I cannot trust ChatGPT-4 to provide me with the correct list of ingredients and steps to cook traditional Italian dishes. As a Sicilian, how can I trust an AI that advises using melon instead of watermelon to prepare the delicious Gelo di Mellone?
Even more concerning are instances where ChatGPT might suggest harmful recipes, as has occurred with some users of a meal suggestion app.
Conclusions
Based on my personal experience and the responses I received from my LLM usage survey, I have come to the conclusion that all the hype surrounding ChatGPT is well-deserved. It is a powerful tool that is already bringing immense value to its early adopters. However, we are still in the early days of LLMs, and before they can be adopted by the majority, they need to evolve in terms of both user experience and content accuracy.
