An evening with ChatGPT (3)
--
Hey ChatGPT, help me with NLP!
In my previous articles I had a look at some core CxD components, and showed how you can let ChatGPT think it’s a pretty good pizzabot. In this third installment, let’s explore whether it can help me with NLP tasks, like, I don’t know, creating an intent model from scratch. After all, it’s an LLM, so some of this should sound familiar.
So here we go again! Ready for some fun?
Impeccable welcome message as usual: greetings, implicit confirmation, I’m not human, capability statement & suggestions (not more than 3), aaaaand….prompt! Someone has been studying their Cathy Pearl here!
Let’s revisit the pizza bot
When I built the ‘Hello world pizzabot’ in ChatGPT, I provided quite a lot of prompts and logic, with pretty amazing results. Some of you asked whether ChatGPT would be able to be a pizzabot straight out-of-the-bot, with just the prompt ‘Pretend you’re a chatbot for ordering pizzas’. That’s something I didn’t think of (or didn’t even dare dream of), so let’s give it a try…
Oh dear…did I just test myself out of a job? That sounds like a very plausible and usable bot, and this time I didn’t specify anything in advance.
Before I give this bot a try, I’d like to know something about its inner workings. After all, if I want to implement this bot in my own platform, I need something more than just a description of the flow. It would be really handy if I had the intents and entities that I’d need for this bot.
List the intents and entities
And what do you know…ChatGPT enlightens me on how it works and how that differs from a traditional chatbot. Interesting for sure, but does this mean that we’re stuck? Fortunately not, because we can always ask ChatGPT to pretend or imagine (which would be a great trait for a bot persona to have, just like the ability to ‘just think along with me for a second’).
And we’re back on track. An impressive one really, ChatGPT lists a number of very useful intents. Like adding additional toppings, and some change intents as well. These might seem trivial, but in practise, they do often make a difference in usability.
And notice that ChatGPT says that these are some of the intents? Would that mean that there can be more?
That is really nice work. Of course, it might not be mind-blowing to a conversation designer, but considering that there’s been no input from my side, this means that it’s really good at spitting out familiar use cases at the right level of granularity. Either that, or we’ve been putting some really great pizza bots out there for ChatGPT to train on!
And some training phrases while we’re at it…
Well, I guess that if intents are possible, Chat GPT might as well populate them with some training phrases.
Of course.
BuT yOu ShOuLd ReAl UsEr QuEsTiOnS!
I know! But there’s only so many ways you can ask for a pizza, right? We can try if we can make these training phrases less monotonous and a bit more user like though…
Hmm, so much for grammatical variation; it seems to basically disregard my specific instructions, and add variation in a very formulaic and predictable way. Perhaps I wasn’t specific enought and should have ask for intra-intent variation. Or perhaps I’m not good at creating training data? What would ChatGPT suggest?
That’s a relief! But when I try to lure ChatGPT into making some specific variations based on linguistic concepts, it really struggles. Let’s have a look.
Don’t be so literal!
OK, if ChatGPT can’t grasp the idea of ellipsis (leaving out words), perhaps I should be a bit more specific in my instructions. I’d like to see some training phrases without pizza, because we don’t use that word in real life all the time either, right?
Aaaargh, no, yes, I see what you mean, but of course that’s not what I meant! I don’t want a synonym or a hyponym instead of ‘pizza’, I want you to leave out that part altogether!
Next try:
Nope. Once more:
Aaaand no. For the first time, I’m kind of disappointed that it doesn’t immediately get what I mean. And it makes me curious about what kind of linguistics concepts it understands. So let’s drill down even more, and take one example sentence.
So what do you know about language then?
After some initial hickups, I get ChatGPT to start some basic text processing, like counting the number of words. An interesting detail: you can clearly see how ChatGPT processes the input: linear, from beginning to end, and starts crafting its answer at the same time. As a result, it doesn’t realise that the first sentence is, in fact, a sentence that I input, rather than a bit of conversation that it should respond to the letter it thought I found. But I can solve that by putting the sentence in quotation marks. And there we go.
I was kind of like ‘if you know quotation marks aren’t words, then why count them as such?”, and I know it’s a bit of NLP pipeline shining through here (probably a lost space that got tokenized), but I do plan to have a good side discussion on ChatGPT’s tone-of-voice one of these days! :-)
And from here, let’s check some basic grammar knowledge. This shouldn’t be a problem for a LLM, I think. And indeed, ChatGPT does really well:
2 prepositional phrases
Phrase level analysis: no problem either. Two prepositional phrases. Easypeasy. Bear this in mind. Two prepositional phrases. Two. Not three. Not one. Two.
So let’s try out some rewrites, like I tried with the training phrases earlier on. Here, ChatGPT really does quite well, accompanying the result with its usualy, rather long-winded explanations.
Let’s try a rewrite without prepositions:
(Insert Kane Simms’ voice here) Interesting!
Now here, something interesting happens. We can clearly see that ChatGPT is capable of doing a rewrite without prepositions, but it only applies it to the first prepositional phrase. Might this again be a result of linear parsing? Does ChatGPT see this as a kind of for loop and just stops after it found the first value that satisfies the end condition? In that case, I might need to change my instruction, not on the grammatical instruction side, but on how to process the sentence. Let’s give it a try.
So I tried this one, and about a dozen other variations, but to no avail: ChatGPT refused to rewrite the ‘to my father’ prepositional phrase. So could it be a grammar thing after all? It correctly identified 2 prepositional phrases earlier on, so I’d assume that it identified two prepositions, right?
Wrong.
Aaargh ChatGPT! What are you doing? How come you can interpret my vaguest ideas to a T, but stumble on some fundamental logic, yet again!
(Spoiler, that’s the entire problem with ChatGPT. It can appropriate stuff really nicely, but it can’t nail down logic, language or facts very well).
OK, let’s negotiate our way out of this.
I don’t really know what’s happening here. Could it be that to might be syntactically ambiguous, and therefore doesn’t get recognised as a preposition straight away?
Well…I guess that you’re probably better off cleaning and optimising your own training data for the time being.
Recap
OK, so what did I learn on this thorougly enjoyable evening:
- ChatGPT can generate a list of obvious and less obvious intents and entities for a non-existing, but familiar chatbot quite easily.
- It won’t give you a lot of real-life training data, nor solve some of your real-life recognition problems straight out of the bot, but it looks promising enough to keep exploring further.
- When it comes to grammatically parsing sentences, I can’t get my finger behind why ChatGPT does the things it does yet. More to explore here too!
- Writing articles like these is just so much nicer with a new mechanical keyboard. Like this one.
