Prompt Engineering | LLaMA-2 Chat | COT | ReAct | Few Shot | Self-Critique
We have multiple resources on internet that discuss about variety of methods to create prompts i.e. prompt engineering. Here, my intention is to explain the major Prompt Engineering methods by following easy going examples.
Happy Reading…..
Specifications of the experiments done are mentioned below.
Model: LLaMA-2 Chat (13B)
Inference: together.ai playground
Default Parameters: Output Length 512, Temperature 0.7, Top-P 0.7, Top-K 50, Repetition Penalty 1
Together.ai provides an inferencing service and also provides some credit balance for trial purposes. I have utilized the same for below examples.
Let’s begin with the methods and examples…
Zero Shot Prompt— In this method, we rely on model to provide us answer to the question without us providing additional support.
As you can see in the example below, we are asking the LLM to find out logical flaw in a sample text. The flaw is related to the usage of pronouns — we are using “his” for peter in first sentence and “She” for peter in second sentence. That’s what is confusing in the text.
Result — Model isn’t able to catch hold of the issue w.r.t bad usage of pronouns
Few Shot Prompt — Model is supplied with one or more example question and answer. The examples help the model build reasoning capability in expected direction. Let’s see how model does with support of an example.
Result — Model is able to catch hold of this issue w.r.t bad usage of pronouns by following the reasoning provided in example.
Few Shot Prompt (Complicated Question)— This looks to be the same as previous example. The only difference is that the question is made complex. It is a bit tough to identify the misuse of pronouns now.
Result — As expected, model miss the key issue w.r.t pronoun usage when we complicated the situation.
Self-Critique — The complicated question which was left unanswered in previous example is used here as well. The difference here is that we force the model to think of multiple outputs and then come up with the best possible output. We do this by adding a line to the prompt i.e. “Think of all possible logical flaws and critique to return the most obvious logical flaw”.
Result — The model is able to pick up the incorrect usage of pronouns after being forced to evaluate its responses using self critique.
Chain Of Thought (COT) — In this technique, you are pushing LLM to think step by step. This happens by adding a text “Let’s think step by step” as shown in below picture.
Please note that there are multiple variations of COT. Once such variation can be seen in the blog available on prompting guide. There is also an excellent video tutorial on prompt engineering by DeepLearning.AI here that gives excellent COT methods.
Result — The model is able to pick up the incorrect usage of pronouns after being forced to think step by step.
Chain Of Thought (Complicated Question) — Here, we complicated the situation to such an extent that COT does not work as expected. Let’s see what happens.
Result — As we see in the output, LLM is unable to figure out the most obvious flaw in the situation.
Chain Of Thought & Few Shot — To overcome the issue we faced in the previous example w.r.t a complicated situation, we provided an example in the prompt to help COT find out the flaw successfully.
Result — Model is able to find out the flaw in situation w.r.t bad usage of pronouns.
Reasoning and Action (ReAct) — Here, we are deviating from the gender inconsistency example that we utilized in prior methods. The idea, here, is to see if a complicated question that needs multiple steps can be answered correctly by giving the model some sample questions and the steps involved in arriving at the answers to these questions.
For example, in the first sample question as seen below, we are supposed to find age of Leo DiCaprio’s girlfriend and then do a mathematical computation on top of her age. This requires multiple steps. We need to find Leo’s identity, then his girlfriend’s identity & age and in the last do the mathematics asked in the question. We provide LLM with all these steps.
Below is the second example that we are providing LLM in the prompt. This is a similar example that requires multiple custom steps to reach to an answer.
Also, notice that we are also asking a question in below picture to the LLM. The question is “Who was the first PM of India and what would be his age if he was alive today”.
Response — Below is the response we received from LLM. As we see, LLM is able to do all the steps based on the samples we supplied to it in prompt. This is what I was talking about.
Result — The answer is still incorrect. However, model is able to understand that it has to perform some steps related to reasoning and provide a response. I was able to get correct answer for the exact same prompt by upgrading the model from LLaMA-2 Chat (13B) to LLaMA-2 Chat (70B).
The example that we did above for ReAct can also be done without giving any sample reasoning and action samples. This can be achieved using LangChain library and will require OpenAI subscription.
That’s all for this article. In case you come across other popular techniques, please feel free to add them in comments. I will try to add examples of those in this article as well.
References:
https://python.langchain.com/docs/modules/agents/agent_types/react
https://learn.deeplearning.ai/chatgpt-prompt-eng/lesson/1/introduction
https://www.promptingguide.ai/
