Photo by Markus Winkler on Unsplash

E6 : Least-to-most Prompting

Breaking a complex problem into simpler subproblems and solving them so as to solve the complex problem

Paper Name : Least-to-most prompting enables complex reasoning in large language models

Paper URL : https://arxiv.org/abs/2205.10625

Authors : Google Research, Brain Team - Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, Ed Chi

Conference : ICLR 2023

Please find the annotated paper here

Problem Statement :

1. CoT tends to perform poorly on problems that are harder than the exemplars in the prompt
2. Prompting has always been unidirectional - a tool to instruct LLM rather than being bidirectional which allows taking feedbacks from LLM as well

Solution :

• Solution consists of two stages namely decomposition and subproblem solving
• Decomposition - Breaking down a complex problem into multiple subproblems using few shot prompting.
• Subproblem solving - Solving the subproblems using few shot prompting to generate the result. Passing the solved subproblem and its result as part of prompt to solve the next subproblem.

Experimentation :

• The approach was tested on three major tasks - symbolic manipulation (last letter concatenation), compositional generalisation (SCAN), maths reasoning (GSM8K)
• As part of the experimentation, least-to-most prompting was used along with text-davinci and code-davinci models to enable reasoning in LLMs.
• Few shot prompting, CoT and least-to-most prompting techniques were compared against each other on the three tasks mentioned.

Observations :

• As part of symbolic manipulation, last letter concatenation task was considered. A list with different number of words (L=4,6,8,10,12) were considered.
• Least-to-most prompting outperformed both CoT and few-shot learning under all scenarios. It outperformed CoT by a huge margin especially for long length lists (L=12).
• Least-to-most prompting achieved an accuracy of 99.7% on compositional generalisation task with code-davinci-002 which was way ahead of the result achieved by CoT (16.7%).
• The accuracy achieved in compositional generalisation task was ahead of all the existing prior works without any neural network training or fine-tuning.
• In maths reasoning, least-to-most prompting outperforms CoT by a narrow margin of 2% only.
• But analysis show that there was a significant improvement of 6% over CoT when solving complex problems that requires decomposing into 5 or more steps. This emphasises the need for solving a complex problem into smaller subproblems.

Limitations :

• Decomposition prompts fail to generalise to a different domain and sometime within domain problems.
• It was observed that decomposition generalisation was difficult even for different problems within the GSM8K dataset itself thus reducing the performance gains of least-to-most prompting.

Conclusion :

• Least-to-most prompting opens way to mimicing human approaches to solve complex problems.
• Least-to-most prompting can be integrated along with CoT to make it even more powerful for reasoning in LLMs.
• Least-to-most prompting can be that step towards making prompts being bidirectional which involves instructing a LLM and as well getting its feedback.