ChatGPT and your job
I normally enjoy Slack at work — I get to learn a thing or two and get a cute picture of a dog or something. But today was different.
The inevitable conversation around chatGPT was in my own Slack chat today. I love machine learning, worked in the field briefly and now I help provide data to those that do machine learning — this feels different. What is openai’s chatGPT? And why is it making me write this ominous introduction?
If you’re like me you’ll stop reading and play with chatGPT for a bit then come back. Pretty amazing right? You ask chatGPT anything and you get a pretty unbelievably plausible answer. Feels like magic. What can it not do? Is it writing this article?
WHAT IS chatGPT
It’s important to remember what this is exactly (I used to do very basic natural language modeling i.e. NLP) — GPT stands for generative pretrained transformer. It’s different/exciting from other state of the art NLP methods because of the way nodes are connected in this deep neural net, tl;dr it results in it being able to do computations on large sentences filled with different words concurrently instead of sequentially. These transformers consider entire sentences in context while doing computations which is really efficient (you don’t have to disambiguate the context of words for the model i.e. “salsa” as in dance or food). Machine learning engineers still have to decide which examples to show GPT i.e. outliers? Normal ones? They actively tune knobs to make it better (called hyperparameters), and also help confirm good answers for chatGPT i.e. reinforcement learning. Initial iterations of GPT were good for: fill in the blank, inference, or translation.
MAYBE NOW IT CAN DO MORE?
This newest iteration of GPT, chatGPT aka GPT 3.5, feels like it is having a hockey stick moment — similar to what I am seeing in data streaming. In a sentence, data streaming (Confluent) fuels mission critical applications with an always-up, near real-time feed of information (think about if a brokerage’s trading app for placing orders or a monitoring system for brain pressure in a children’s hospital just stopped receiving data?). So data streaming is really the data backbone for all kinds of modern applications , from Instacart to Northrop Grumman to NLP applications similar to ChatGPT. This collision of technological improvements simultaneously exciting and unnerving. Exactly how much of an improvement are we talking about?
MACHINE LEARNING BEFORE GPT
Let’s go back all the way to Kai Fu Lee’s diagram from ~2018 (a lot happens over a few years in tech). See Lee’s Ted Talk here. Tl;dr Lee goes through jobs that will likely be replaced by AI (his vernacular not mine). He posited that AI will have:
- the strongest influence on redundant jobs (bottom left quadrant)
- become vital tools for creative jobs (bottom right)
- work as a lighter “analytic” tool for high compassion jobs (top left)
- and finally the top right is likely to remain least augmented by AI.
Is ChatGPT truly a new step in augmenting or dare I say, dominating a particular quadrant?
TESTING chatGPT
I’m happy to put some questions I get during my job as an experiment to test just how much chatGPT is capable of.
GPT WAS GOOD AT
To not alienate non-technical readers, the tl;dr is I asked GPT a question that requires some stitching together various, different technologies. It was not half bad. The example below is GPT successfully using these specific technologies in concert:
- infrastructure-as-code tool called Terraform
- A kubernetes distribution provided by a Cloud provider (AWS)
- A linux based operating system (also via AWS)
ChatGPT was also decent at providing templated code when I asked it for a few different flavors of kafka consumers. This would have been a great starting point when I was initially starting my project for Current data streaming conference.
It’s impressive how GPT is providing rich feedback in the field of software engineering at large. For this NLP engineer, it provides much better feedback than if he were to run R on his local machine (he created an R emulation within ChatGPT). https://twitter.com/ben_j_radford/status/1599180658827440128
NOT SO GOOD AT
Here I asked GPT a more common question I get, where it just came up with CLI commands that completely do not exist( Confluent Tunnel is not a command in Confluent's CLI -- I work at Confluent). Here I asked it: "how do i connect my on premise data source to Confluent Cloud securely"
Interestingly, if you were to evaluate chatGPT’s intelligence by our (human) traditional evaluation, its low-average apparently: https://twitter.com/SergeyI49013776/status/1598430479878856737
ASKING IT QUESTIONS UNRELATED TO MY JOB
Ok this one is ironic/not really useful, but still, I asked ChatGPT about itself, and it provided misinformation:
Having ChatGPT pretend it is a specific occupation is perhaps the most eye-brow-raising phenomenon across the internet today. For example, asking it to act as a chef, it gave some good dinner ideas with the ingredient list and instructions:
Other fun “act as” prompts here.
OPTIMISTIC OUTLOOK
I would argue if we look at at Lee’s predictions in 2018, chatGPT is having outsized impact on both quadrants right of our Y axis. It’s feedback is incredibly rich. Creative tasks like coming up with scripts for screenplays, decorating physical spaces — these things are often hard to get started. I feed chatGPT a few prompts and I have walls of text to get my ideas flowing — that is undeniably impressive. Closer to my wheelhouse, chatGPT also catalyzes development tasks for software engineering. As of chatGPT 3.5 (fueling chatGPT), it is not capable of creating truly “prod-ready” code; however, it gives me great starting material that I can then refine — saving me hours of initial research through Stackoverflow etc.
NOT SO OPTIMISTIC OUTLOOK
In the back of everyone’s mind right now is can chatGPT do something destructive. Alone of course it cannot, it is not some autonomous agent, it needs human intervention to do anything. But say if you were not careful in testing and interpreting chatGPT’s outputs, and just ported some code over from chatGPT — I could envision a situation where a developer accidentally crashes a server or something to that effect. This doesn’t strike me as really alarming because most engineers have good processes in place for testing, validation, and finally releasing code to a production system.
What might be alarming is as shown from my example above where I asked chatGPT what it is — chatGPT sometimes just produces misinformation (openai acknowledges this). You could argue the circulation of misinformation on the web is already rampant — what’s another source? However, this is the first time we have this misinformation without a face. No one can trace the path of a request from chatGPT’s interface, through its layers of complex neural nets, and deduce exactly how the response is arrived upon. This is a stark contrast to someone quoting an article from a journalistic source (where you might be able to infer political influence etc.).
CLOSING THOUGHTS, GPT 4 is coming
ChatGPT is going to give you an answer. Use it long enough and it will give you a wrong answer. It will misunderstand you. ChatGPT is only iteration 3.5 of the GPT series. Soon it will be providing better answers. I spend a lot of my workday asking questions like: is that really what you mean? Do you know why that requirement is in place? How did that evaluation go? It’s from open-ended questions like these that I get to really help people, and sometimes uncover things they did not know they needed. We are so much more than question-answering machines. We are extremely nuanced. Get out there and really listen — you’ll ask better questions, and that means better answers.
