E6 : Least-to-most Prompting
Published in
3 min readJul 16, 2023
Breaking a complex problem into simpler subproblems and solving them so as to solve the complex problem
Paper Name : Least-to-most prompting enables complex reasoning in large language models
Paper URL : https://arxiv.org/abs/2205.10625
Authors : Google Research, Brain Team - Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, Ed Chi
Conference : ICLR 2023
Please find the annotated paper here
Problem Statement :
- CoT tends to perform poorly on problems that are harder than the exemplars in the prompt
- Prompting has always been unidirectional - a tool to instruct LLM rather than being bidirectional which allows taking feedbacks from LLM as well
Solution :
- Solution consists of two stages namely decomposition and subproblem solving
- Decomposition - Breaking down a complex problem into multiple subproblems using few shot prompting.
- Subproblem solving - Solving the subproblems using few shot prompting to generate the result. Passing the solved subproblem and its result as part of prompt to solve the next subproblem.
Experimentation :
- The approach was tested on three major tasks - symbolic manipulation (last letter concatenation), compositional generalisation (SCAN), maths reasoning (GSM8K)
- As part of the experimentation, least-to-most prompting was used along with text-davinci and code-davinci models to enable reasoning in LLMs.
- Few shot prompting, CoT and least-to-most prompting techniques were compared against each other on the three tasks mentioned.
Observations :
- As part of symbolic manipulation, last letter concatenation task was considered. A list with different number of words (L=4,6,8,10,12) were considered.
- Least-to-most prompting outperformed both CoT and few-shot learning under all scenarios. It outperformed CoT by a huge margin especially for long length lists (L=12).
- Least-to-most prompting achieved an accuracy of 99.7% on compositional generalisation task with code-davinci-002 which was way ahead of the result achieved by CoT (16.7%).
- The accuracy achieved in compositional generalisation task was ahead of all the existing prior works without any neural network training or fine-tuning.
- In maths reasoning, least-to-most prompting outperforms CoT by a narrow margin of 2% only.
- But analysis show that there was a significant improvement of 6% over CoT when solving complex problems that requires decomposing into 5 or more steps. This emphasises the need for solving a complex problem into smaller subproblems.
Limitations :
- Decomposition prompts fail to generalise to a different domain and sometime within domain problems.
- It was observed that decomposition generalisation was difficult even for different problems within the GSM8K dataset itself thus reducing the performance gains of least-to-most prompting.
Conclusion :
- Least-to-most prompting opens way to mimicing human approaches to solve complex problems.
- Least-to-most prompting can be integrated along with CoT to make it even more powerful for reasoning in LLMs.
- Least-to-most prompting can be that step towards making prompts being bidirectional which involves instructing a LLM and as well getting its feedback.