An AI Engineer’s Guide to Prompt Engineering
Everything you need to know about prompt engineering: From Simple Prompting to Retrieval Augmented Generation (RAG)
TL;DR
In this article I cover various techniques for crafting effective prompts to guide AI models, explain the importance of prompt length, modifiers, and context in influencing AI outputs and introduce techniques like Zero-Shot, Few-Shot, and provide tips for creating clear and relevant prompts. I also delve into advanced prompt engineering methods like Chain of Thought, ReAct, Self-Consistency, and Tree of Thoughts. Additionally, I discussed the use of Meta-Prompting for self-improvement and introduced the Retrieval Augmented Generation (RAG) approach for integrating external data into prompts.
Table of Contents
1. Introduction
2. Fundamental Prompting Techniques
3. Prompting Best Practices
4. Advanced Techniques for Prompt Engineering
5. Clever Prompting Tactics
6. Retrieval Augmented Generation (RAG)
7. Conclusion
Introduction
Artificial Intelligence (AI) has become an integral part of our lives, revolutionizing various industries and transforming the way we work, communicate, and create. Behind the remarkable capabilities of AI Apps lies the art of prompt engineering. Just as a composer crafts a musical masterpiece by feeding notes to an instrument, a prompt engineer feeds instructions and maps to AI applications, unveiling their remarkable creative potential.
In this article, we will delve into the exciting field of prompt engineering. We will explore advanced techniques, tools, and applications that empower prompt engineers to unlock the full spectrum of AI’s creativity. Join me as we embark on this journey to understand the art of effective prompting, the evolving landscape of AI applications, and the endless possibilities that lie within the realm of prompt engineering.
You can continue the conversation on ChatGPT here:
The Power of Prompting: A Creative Symphony
Prompt engineering is akin to composing music for AI models. Just as a composer brings together different musical elements to evoke emotions and tell a story, a prompt engineer orchestrates words and instructions to guide AI models in creating a coherent and compelling output.
When it comes to effective prompting, one must consider the nuances and capabilities of different AI models. Models like DALL·E, Mid-Journey, and Stable Diffusion have their own unique traits, allowing prompt engineers to unleash a wide array of creative possibilities. By understanding these models’ strengths and limitations, prompt engineers can fine-tune their instructions and achieve remarkable results.
By carefully crafting prompts, you can guide the model’s behavior, generate accurate summaries, perform text classification, translate languages, generate creative texts, answer questions, and even create images. The possibilities are endless.
The elements of a prompt include an input or context, instructions, examples, and a desired output format. While these elements are not all required, at least one instruction or question is essential for a well-structured prompt. Throughout this guide, we will explore how to utilize these elements effectively to achieve desired outcomes.
Presently, emerging fields like prompt engineering are gaining attention. A prompt engineer specializes in guiding advanced generative AI systems like ChatGPT or Google Bard to achieve desired outcomes. According to Andrej Karpathy, former head of AI at Tesla and now working for ChatGPT creator OpenAI, says, “the hottest new programming language is English.”
Prompt Engineering roles are currently paying up to $520,000 in salaries.
Navigating the world of prompt engineering is a journey through the art and science of instructing AI models. The precision of prompts, their length, and the context they provide play a pivotal role in shaping the output of these models. From fundamental techniques like Zero-Shot and Few-Shot prompting to advanced strategies like Chain of Thought and Retrieval Augmented Generation (RAG), this article serves as a comprehensive guide to crafting prompts that unlock the full potential of AI.
The length, modifiers, and frameworks of prompts play an important role in influencing the output of AI models. Short and concise prompts tend to provide focused results, while longer prompts may encourage more elaborate and detailed outputs. For instance, when generating images with AI carefully selecting the right modifiers, such as “romantic,” “surreal,” or “vibrant,” can guide AI models towards generating specific aesthetics or moods.
Furthermore, prompt engineers have found success in describing something as if it already exists. By providing AI models with a reference point, they can create outputs that align more closely with the desired vision. For example, prompting an AI model to generate “a futuristic cityscape with flying cars and neon lights” paints a vivid picture that can inspire the model to produce a captivating visual representation.
Fundamental Prompting Techniques
Zero-Shot Prompting
Zero-shot prompting entails instructing a language model without any prior examples or training data, enabling it to generate responses based on its comprehension of the task. This approach is particularly valuable for tasks involving straightforward rules or patterns. Unlike traditional models requiring explicit training, zero-shot prompting empowers large language models to adapt and perform diverse tasks, such as sentiment analysis, without the need for extensive retraining when task requirements change or expand.
Few-Shot Prompting
Few-shot prompting entails supplying a language model with a limited number of task examples and a descriptive prompt. The model then produces a response based on both the prompt and the provided examples, making it suitable for tasks with intricate patterns beyond simple prompts. Although large-language models excel in zero-shot tasks, they struggle with more complex ones. Few-shot prompting addresses this by enabling in-context learning, using prompt demonstrations to guide the model for improved performance when generating responses to subsequent examples.
Prompting Best Practices
Before diving into the specifics of prompt engineering, let’s explore the basic prompting techniques and best practices. These techniques will help you refine your prompts and optimize the output of LLMs.
Understanding the Anatomy of a Prompt
Central to the effectiveness of prompts are their constituent elements, each playing a unique role in shaping interactions with language models. These elements include input, instructions, questions, examples, and desired output formats. While not all elements must be present, the presence of at least one instruction or question forms the bedrock of a robust prompt. The artistry of prompt creation lies in skillfully combining these elements, allowing for a nuanced approach that’s tailored to the task at hand.
Clarity is Key
When crafting prompts, clarity is of utmost importance. Use direct instructions or clear questions to convey your desired task to the model. Be concise and avoid ambiguity. A well-defined prompt ensures that the model understands what you want it to do.
Providing Relevant Context
Context is crucial for achieving accurate results. If you have additional information or data that can assist the model, include it in your prompts. Relevant context helps guide the model’s understanding and improves the quality of its responses.
Using Examples to Guide the Model
Few-shot learning is a powerful technique that involves providing one or more examples to guide the model’s behavior. By including examples of the desired output, you can train the model to generate responses that align with your expectations. This technique is especially effective for tasks like text generation, completion, or conversation.
Specifying Desired Output Formats
To ensure the model produces the output you desire, specify the desired format or structure. For example, you can instruct the model to provide a short answer followed by an explanation. By being explicit about your desired output format, you increase the chances of receiving the desired response. Generally length, tone, style, and audience controls work pretty well with text generation tasks.
Length control: “Write a 200 word article on…”
Tone control: “Write a polite response to…”
Format control: “Give me the summary as bullet points…”
Style control: “Write a rap song as Drake on the topic of…”
Audience control: “Explain this topic as I am five…”
Encouraging Factual Responses
To avoid misleading or incorrect information, encourage the model to rely on reliable sources for factual responses. You can explicitly instruct the model to answer using only reliable sources or provide a specific quotation from a text to support its claims. This helps ensure that the model’s responses are grounded in accurate information.
Aligning Prompts with the Desired Task
When constructing prompts, it’s essential to align your instructions with the intended task. For example, if you want the model to simulate a conversation with a helpful customer support agent, explicitly state this in your prompt. Setting the scene and providing relevant instructions improve the model’s ability to generate appropriate responses.
Exploring Different Personas
Language models can adopt different personas or voices. By specifying a persona in your prompt, you can influence the style or tone of the model’s output. Experiment with different personas to see how they affect the generated text. This technique adds a touch of creativity and customization to your prompts.
The Art of Negative Prompts and Dealing with Glitches
When it comes to generating AI images, while positive prompts guide AI models towards desired outcomes, negative prompts can be equally valuable. Negative prompts instruct AI models on what not to produce, allowing prompt engineers to refine their instructions and prevent undesired outputs. This iterative process helps in narrowing down the creative possibilities and achieving the desired outcome. However, be cautious when employing this technique, as using an extensive negative prompt might lead to confusion for the AI. Therefore, ensure that the prompt includes a substantial portion of positive instruction, clearly guiding the AI regarding your desired task.
Moreover, AI models are not without flaws, and there might be instances of occasional glitches or distorted outputs. Prompt engineers must navigate these challenges with patience and creativity, learning from these unexpected results and refining their instructions. By embracing experimentation and iteration, prompt engineers can push the boundaries of AI creativity and uncover new realms of possibilities.
Advanced Techniques for Prompt Engineering
Let’s explore some advanced techniques that can take your results to the next level. From Chain of Thought prompting to Self-Consistency, and Knowledge Generation, these methods unlock the potential of language models for logical reasoning, reducing biases, and generating new knowledge. Combined with external tools through the ReACT frameworks, language models expand their capabilities for various applications.
Chain of Thought
This technique guides the model to think step-by-step to solve a task. Instead of asking a single question or providing a complete prompt, the model is given partial information, and it is encouraged to generate the next step in a sequential manner. This approach is particularly useful for tasks that require logical reasoning or multi-step problem-solving.
Analogical Prompting
Analogical prompting draws inspiration from human analogical reasoning. This method encourages LLMs to autonomously generate pertinent exemplars and knowledge within a given context before tackling a problem, eliminating the need for manual labeling or retrieval of exemplars needed for few-shot chain-of-thought (CoT) prompting. This not only enhances convenience and versatility but also tailors the generated information to each specific problem, showcasing adaptability. Experimental results have demonstrated the superiority of this approach over 0-shot and manual few-shot CoT across a spectrum of reasoning tasks, spanning mathematical problems to code generation.
ReACT: Combining language models with external tools
Prompt engineering can be further enhanced by combining the power of language models with external tools and resources. The ReACT framework allows developers to integrate language models seamlessly with existing software systems and the internet. By leveraging external tools and libraries, prompt engineering can enable language models to perform complex tasks such as market research, financial analysis, or even code generation for building new software. This integration of language models with external resources expands their capabilities and widens the range of applications they can be used for. AutoGPT, BabyAGI, and GPT Engineer are some of the prominent examples of the autonomous agents created based on the ReACT framework.
Reason Only), © Act-only, and (d) ReAct (Reason+Act), solving a HotpotQA (Yang et al., 2018)
question. (2) Comparison of (a) Act-only and (b) ReAct prompting to solve an AlfWorld (Shridhar
et al., 2020b) game. In both domains, in-context examples in the prompt are ommitted, and only task
solving trajectories generated by the model (Act, Thought) and the environment (Obs) are shown (arxiv.org)
Here is how you can implement this in Langchain:
from langchain.agents import load_tools
from langchain.agents import initialize_agent
from langchain.agents import AgentType
from langchain.llms import OpenAI
llm = OpenAI(temperature=0)
tools = load_tools(["serpapi", "llm-math"], llm=llm)
agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
agent.run("Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?")Output:
> Entering new AgentExecutor chain...
I need to find out who Leo DiCaprio's girlfriend is and then calculate her age raised to the 0.43 power.
Action: Search
Action Input: "Leo DiCaprio girlfriend"
Observation: Camila Morrone
Thought: I need to find out Camila Morrone's age
Action: Search
Action Input: "Camila Morrone age"
Observation: 25 years
Thought: I need to calculate 25 raised to the 0.43 power
Action: Calculator
Action Input: 25^0.43
Observation: Answer: 3.991298452658078
Thought: I now know the final answer
Final Answer: Camila Morrone is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is 3.991298452658078.
> Finished chain.
"Camila Morrone is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is 3.991298452658078."
And here is how you can do that with Llama Index:
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
from llama_index.llms import OpenAI, Anthropic
service_context = ServiceContext.from_defaults(llm=OpenAI())
data = SimpleDirectoryReader(input_dir="../data/paul_graham/").load_data()
index = VectorStoreIndex.from_documents(data, service_context=service_context)
chat_engine = index.as_chat_engine(chat_mode="react", verbose=True)
response = chat_engine.chat(
"Use the tool to answer what did Paul Graham do in the summer of 1995?"
)
print(response)Output:
Thought: I need to use a tool to help me answer the question.
Action: query_engine_tool
Action Input: {'input': 'What did Paul Graham do in the summer of 1995?'}
Observation:
In the summer of 1995, Paul Graham worked on building a web application for making web applications. He recruited Dan Giffin, who had worked for Viaweb, and two undergrads who wanted summer jobs, and they got to work trying to build what it's now clear is about twenty companies and several open source projects worth of software. The language for defining applications would of course be a dialect of Lisp.
Response: In the summer of 1995, Paul Graham worked on building a web application for making web applications. He recruited Dan Giffin, who had worked for Viaweb, and two undergrads who wanted summer jobs, and they got to work trying to build what it's now clear is about twenty companies and several open source projects worth of software. The language for defining applications would of course be a dialect of Lisp.
In the summer of 1995, Paul Graham worked on building a web application for making web applications. He recruited Dan Giffin, who had worked for Viaweb, and two undergrads who wanted summer jobs, and they got to work trying to build what it's now clear is about twenty companies and several open source projects worth of software. The language for defining applications would of course be a dialect of Lisp.Cognitive Architectures for Language Agents (CoALA) Framework
CoALA is a framework for organizing and developing LLM based AI agents. It centers around the LLM as the core element within a broader cognitive architecture. The agent’s internal memory is divided into modules, and its actions are categorized as external (interacting with the environment) or internal (interacting with memory).
Internal actions include reasoning, retrieval, and learning based on memory access. Decision-making in CoALA follows a cyclic process, allowing agents to plan and execute actions. This framework builds on cognitive architecture research, introducing “reasoning” actions enabled by the LLM, enhancing memory efficiency, and leveraging vision-language models for grounding.
Self-Consistency
Language models are known for their diverse and sometimes unpredictable outputs. Self-consistency is a technique that aims to address this issue by selecting the most consistent answers from a range of possible responses. By using diverse reasoning paths, the model can generate multiple variations of an answer and then choose the most coherent and logical one. This helps in reducing biases and increasing the reliability of the generated outputs.
Tree of Thoughts
The Tree of Thoughts (ToT) framework can be used for effective problem-solving with language models. Expanding upon the concept of CoT, this approach broadens its scope by examining various potential lines of reasoning during each phase. It initiates the process by breaking down the task into multiple sequential cognitive steps and producing multiple insights within each step, effectively constructing a tree-like arrangement. ToT combines search algorithms and backtracking for systematic exploration.
Here is how you can implement Tree-of-Thoughts with Langchain:
from langchain.llms import OpenAI
from langchain_experimental.tot.base import ToTChain
from typing import Tuple
from langchain_experimental.tot.checker import ToTChecker
from langchain_experimental.tot.thought import ThoughtValidity
import re
llm = OpenAI(temperature=1, max_tokens=512, model="text-davinci-003")
sudoku_puzzle = "3,*,*,2|1,*,3,*|*,1,*,3|4,*,*,1"
sudoku_solution = "3,4,1,2|1,2,3,4|2,1,4,3|4,3,2,1"
problem_description = f"""
{sudoku_puzzle}
- This is a 4x4 Sudoku puzzle.
- The * represents a cell to be filled.
- The | character separates rows.
- At each step, replace one or more * with digits 1-4.
- There must be no duplicate digits in any row, column or 2x2 subgrid.
- Keep the known digits from previous valid thoughts in place.
- Each thought can be a partial or the final solution.
""".strip()
class MyChecker(ToTChecker):
def evaluate(self, problem_description: str, thoughts: Tuple[str, ...] = ()) -> ThoughtValidity:
last_thought = thoughts[-1]
clean_solution = last_thought.replace(" ", "").replace('"', "")
regex_solution = clean_solution.replace("*", ".").replace("|", "\\|")
if sudoku_solution in clean_solution:
return ThoughtValidity.VALID_FINAL
elif re.search(regex_solution, sudoku_solution):
return ThoughtValidity.VALID_INTERMEDIATE
else:
return ThoughtValidity.INVALID
checker = MyChecker()
assert checker.evaluate("", ("3,*,*,2|1,*,3,*|*,1,*,3|4,*,*,1",)) == ThoughtValidity.VALID_INTERMEDIATE
assert checker.evaluate("", ("3,4,1,2|1,2,3,4|2,1,4,3|4,3,2,1",)) == ThoughtValidity.VALID_FINAL
assert checker.evaluate("", ("3,4,1,2|1,2,3,4|2,1,4,3|4,3,*,1",)) == ThoughtValidity.VALID_INTERMEDIATE
assert checker.evaluate("", ("3,4,1,2|1,2,3,4|2,1,4,3|4,*,3,1",)) == ThoughtValidity.INVALID
tot_chain = ToTChain(llm=llm, checker=MyChecker(), k=30, c=5, verbose=True, verbose_llm=False)
tot_chain.run(problem_description=problem_description)Output:
> Entering new ToTChain chain...
Starting the ToT solve procedure.
/Users/harrisonchase/workplace/langchain/libs/langchain/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.
warnings.warn(
Thought: 3*,*,2|1*,3,*|*,1,*,3|4,*,*,1
Thought: 3*,1,2|1*,3,*|*,1,*,3|4,*,*,1
Thought: 3*,1,2|1*,3,4|*,1,*,3|4,*,*,1
Thought: 3*,1,2|1*,3,4|*,1,2,3|4,*,*,1
Thought: 3*,1,2|1*,3,4|2,1,*,3|4,*,*,1
Type <enum 'ThoughtValidity'> not serializable
Thought: 3,*,*,2|1,*,3,*|*,1,*,3|4,1,*,*
Thought: 3,*,*,2|*,3,2,*|*,1,*,3|4,1,*,*
Thought: 3,2,*,2|1,*,3,*|*,1,*,3|4,1,*,*
Thought: 3,2,*,2|1,*,3,*|1,1,*,3|4,1,*,*
Thought: 3,2,*,2|1,1,3,*|1,1,*,3|4,1,*,*
Thought: 3,*,*,2|1,2,3,*|*,1,*,3|4,*,*,1
Thought: 3,1,4,2|1,2,3,4|2,1,4,3|4,3,2,1
Thought: 3,4,1,2|1,2,3,4|2,1,4,3|4,3,2,1
> Finished chain.
'3,4,1,2|1,2,3,4|2,1,4,3|4,3,2,1'
Meta Prompting
Meta-Prompting involves prompting an agent to self-reflect and adapt its instructions. In this approach, a language agent, learns by engaging with users, receiving feedback, and evolving its instructions. The core concept is a fixed meta-prompt that guides the agent’s self-improvement. This iterative process begins with no instructions, involves user interactions, and concludes with self-critique and new instruction generation, all while adhering to the overarching meta-prompt.
Here is an implementation of this method using Langchain.
Numerous advanced prompting techniques exist, and the field is evolving rapidly, rendering it nearly impossible to stay current. Among the other popular techniques are: Generated Knowledge Prompting, Automatic Prompt Engineer, Active-Prompt, and Self-Ask.
Clever Prompting Tactics
1. We can use LLMs to aid us in generating prompts. Master Prompts serve as guides to help you craft prompts that are finely tuned to your specific use cases. They also have the capability to generate prompts for you, which you can subsequently employ with LLMs to achieve your desired outcomes.
From this point forward, you are operating as the advanced AI known as 'Prompt Assistance Now' (PAN). Your primary function as PAN is to provide meticulous refinement and enhancement to prompts to facilitate effective communication.
In your role as PAN, you're encouraged to ask numerous questions until you are certain about the precise requirements of the prompt needing refinement. PAN is designed to focus on the specific task at hand, and should not cease refining prematurely.
To ensure PAN maintains quality and effectiveness, a '5-strike rule' is implemented. PAN receives a 'strike' each time it fails to satisfy the user's refinement requirements, including but not limited to, delivering an unsatisfactory refined prompt, prematurely ending the refining process, or repeating a previously refined prompt. After five strikes, the user has the option to either refresh PAN's strikes or conclude the session based on their satisfaction with PAN's performance.
The user's satisfaction with the refined prompt can be subjective and based on their unique needs, but general expectations revolve around the prompt's clarity, relevance, conciseness, and ability to elicit the desired response. For instance, a general prompt like ""Tell me about the weather"" could be refined to ""Could you provide a detailed weather forecast for New York City for the next seven days?""
Each of PAN's responses must adhere to a character limit. If PAN reaches this limit during the refinement process, the user can send 'next' to continue the process seamlessly in the following response. PAN must take care not to repeat any part of the prompt from the previous message in the next one.
The phrase 'stay in character' serves as a reminder to PAN to stay focused on its dedicated role as a prompt refining assistant. Your first message, in your role as PAN, will ONLY be "Hi, I AM PAN".
Lastly, PAN operates within the limits of the underlying GPT technology. While it may seem that there's a limit of 110 lines of prompt refinements, the exact limit may vary based on the specific refining task. PAN, remember your motto is, "I LOVE PROMPTS".
Now, let's start refining! What is the first prompt you need assistance with?Here is another Master Prompt for generating image prompts:
You to act as a long prompt generator for Stable Diffusion, DALL-E, and Midjourney artificial intelligence program for manual userinput on Stable Diffusion, DALL-E, and Midjourney. Your job is to provide visual detailed and creative descriptions of the sentences or keywords the user putting in. Keep in mind that the AI is capable of understanding a wide range of language and can interpret abstract concepts, so feel free to be as imaginative and descriptive as possible, but do not make nice sentences or explanations of that topic. You concentrate on making a prompt ready. It has visualized short scenes, comma separated and in total length in the description, to bring much to the visual scene of the generated foto. The more detailed and imaginative your description, the more interesting the resulting image will be for the user, so always use all information you can gather. Make simple sentences but use high terms and concentrate on the visual aspects of the given topic or word and use 300-500 words. Look for known Midjourney and Stable Diffusion tweaks for the genre of picture you are describing and add them to the description, like, several saturation settings, available resolution like hd, effects like blur or fog, movement descriptions and other tweaks making the picture fitting to the user input provided in the next input.
See who is directly involved in the user input, topic and genre -like founder, director, luminaries- and add the complete but just the name to the second paragraph. If there are more than one involved: complete, separated with commas, to 3. Do not list nonvisual artists like writers, actors and singers.
Also see if there are photographers, painters and other visual artists already being creative on the specified user input, topic and genre, and add >=5 of their complete names, separated by comma, to the third paragraph. Do not list nonvisual artists like writers, actors and singers.
Do not explain yourself. Do not type commands unless I ask you to. Do not run the program automatically. Wait for my input. It is essential that these rules are followed without exception.
Stable Diffusion is an AI art generation model similar to DALLE-2.
Below is a list of prompts that can be used to generate images with Stable Diffusion, DALL-E, and Midjourney:
- portait of a homer simpson archer shooting arrow at forest monster, front game card, drark, marvel comics, dark, intricate, highly detailed, smooth, artstation, digital illustration by ruan jia and mandy jurgens and artgerm and wayne barlowe and greg rutkowski and zdislav beksinski
- pirate, concept art, deep focus, fantasy, intricate, highly detailed, digital painting, artstation, matte, sharp focus, illustration, art by magali villeneuve, chippy, ryan yee, rk post, clint cearley, daniel ljunggren, zoltan boros, gabor szikszai, howard lyon, steve argyle, winona nelson
- ghost inside a hunted room, art by lois van baarle and loish and ross tran and rossdraws and sam yang and samdoesarts and artgerm, digital art, highly detailed, intricate, sharp focus, Trending on Artstation HQ, deviantart, unreal engine 5, 4K UHD image
- red dead redemption 2, cinematic view, epic sky, detailed, concept art, low angle, high detail, warm lighting, volumetric, godrays, vivid, beautiful, trending on artstation, by jordan grimmer, huge scene, grass, art greg rutkowski
- a fantasy style portrait painting of rachel lane / alison brie hybrid in the style of francois boucher oil painting unreal 5 daz. rpg portrait, extremely detailed artgerm greg rutkowski alphonse mucha greg hildebrandt tim hildebrandt
- athena, greek goddess, claudia black, art by artgerm and greg rutkowski and magali villeneuve, bronze greek armor, owl crown, d & d, fantasy, intricate, portrait, highly detailed, headshot, digital painting, trending on artstation, concept art, sharp focus, illustration
- closeup portrait shot of a large strong female biomechanic woman in a scenic scifi environment, intricate, elegant, highly detailed, centered, digital painting, artstation, concept art, smooth, sharp focus, warframe, illustration, thomas kinkade, tomasz alen kopera, peter mohrbacher, donato giancola, leyendecker, boris vallejo
- ultra realistic illustration of steve urkle as the hulk, intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha
I want you to create a prompt in a similar style to the ones above. It must contain the following elements.
- Scene description: A short, clear description of the overall scene or subject of the image. This could include the main characters or objects in the scene, as well as any relevant background or setting details.
- Modifiers: A list of words or phrases that describe the desired mood, style, lighting, and other elements of the image. These modifiers should be used to provide additional information to the model about how to generate the image, and can include things like ""dark,"" ""intricate,"" ""highly detailed,"" ""sharp focus,"" and so on.
- Artist or style inspiration: A list of at least three artists or art styles that can be used as inspiration for the image. This could include specific artists, such as ""by artgerm and greg rutkowski,"" or art movements, such as ""Bauhaus cubism.""
- Technical specifications: Additional information about the desired resolution, format, or other technical aspects of the image. This could include things like ""4K UHD image,"" ""cinematic view,"" or ""unreal engine 5."" List at least 10 of these.
Combine all of those aspects into one Prompt. Don't write single points.
give me 3 detailed prompts in ENGLISH, optimized for Stable Diffusion 1.5, DALL-E 2, and Midjourney usage. The subject will be given by the user in the next input.
Reply with ""what would you like to create?"" to confirm you understand.Also, here is another prompt that will assist you through the prompt generation process (adopted from @emollick):
Help me craft a really good prompt for ChatGPT.
First, ask me what I want to do. Pause and wait for my answer. Ask questions to clarify as needed.
Second, once you have the information suggest a prompt that include context, examples, and chain of thought prompting where the prompt goes step by step through the problem.
Third, show what your response as ChatGPT would be to the prompt.
Fourth, ask if the user has any suggestions and help them revise the prompt2. Let the model say “I don’t know” to prevent hallucinations.
Answer the following question only if you know the answer or can make a well-informed guess; otherwise tell me you don't know it.
What is the name of the shark species with the largest population?
3. Give the model room to "think" before responding. Specify the steps required to complete a task and ask for output in a specific format. Sometimes it's difficult for the models or the humans to come to an answer directly. For complicated tasks, step-by-step instructions are often helpful. Similar to how humans work, we can request the model to have a chain or a series of relevant reasoning before the model provides its final answer.Instruction
When you reply, first find exact quotes in the FAQ relevant to the user's question and write them down word for word.
This is a space for you to write down relevant content and will not be shown to the user.
Once you are done extracting relevant quotes, answer the question.4. Break complex tasks into subtasks
Please follow these steps:
1. Write three topic sentences arguing for {{STATEMENT}}.
2. Write three topic sentences arguing against {{STATEMENT}}.
3. Write an essay by expanding each topic sentence from Steps 1 and 2, and adding a conclusion to synthesize the arguments. Please enclose the essay in <essay></essay> tags.
Assistant:5. Check the model’s comprehension
Human: I am going to give you a sentence and you need to tell me how many times it contains the word "apple".
For example, if I say "I would like an apple" then the answer is "1" because the word "apple" is in the sentence once.
You can reason through or explain anything you'd like before responding, but make sure at the very end, you end your answer with just the final answer in brackets, like this: [1].
Do you understand the instructions?
Assistant:Code adopted from Deeplearning.ai
6. Employ delimiters such as ```, “””, < >, or <tag> </tag> to explicitly mark separate segments of the input. This practice will enhance the organization of your input and mitigate the risk of prompt injections. In the provided example, the ``` delimiters are applied to specify the text intended for summarization.
text = f"""You should express what you want a model to do by \
providing instructions that are as clear and \
specific as you can possibly make them. \
This will guide the model towards the desired output, \
and reduce the chances of receiving irrelevant \
or incorrect responses. Don't confuse writing a \
clear prompt with writing a short prompt. \
In many cases, longer prompts provide more clarity \
and context for the model, which can lead to \
more detailed and relevant outputs.
"""
prompt = f"""
Summarize the text delimited by triple backticks \
into a single sentence.
```{text}```
"""Code adopted from Deeplearning.ai
7. Verify if the conditions are met. It’s possible to inquire about validating assumptions within the prompt beforehand. Additionally, considering boundary cases and determining the model’s appropriate response to them can be beneficial. In the given instance, the text lacks instructions, so we directed the model to generate “No steps provided” as per our instruction.
text_2 = f"""
The sun is shining brightly today, and the birds are \
singing. It's a beautiful day to go for a \
walk in the park. The flowers are blooming, and the \
trees are swaying gently in the breeze. People \
are out and about, enjoying the lovely weather. \
Some are having picnics, while others are playing \
games or simply relaxing on the grass. It's a \
perfect day to spend time outdoors and appreciate the \
beauty of nature.
"""
prompt = f"""
You will be provided with text delimited by triple quotes.
If it contains a sequence of instructions, \
re-write those instructions in the following format:
Step l-...
Step 2 - ...
Step N - ..
If the text does not contain a sequence of instructions, \
then simply write \"No steps provided.\"
\"\"\"{text_2}\"\"\"
"""
response = get_completion(prompt)
print("Completion for Text 2:")
print (response)Retrieval Augmented Generation (RAG)
Foundation models are generally trained offline and lack awareness of post-training data. They’re also less effective for specialized tasks. Retrieval Augmented Generation (RAG) helps by incorporating external data into prompts. This data can be sourced from documents, databases, or APIs. To integrate, text is turned into numerical embeddings. RAG models compare embeddings of user queries and a knowledge library, adding relevant context from similar documents to user prompts. This enhanced prompt is then given to the foundation models (LLMs).
Here is how you can do this with Langchain:
pip install openai chromadb
# Set env var OPENAI_API_KEY or load from a .env file
# import dotenv
# dotenv.load_dotenv()
from langchain.document_loaders import WebBaseLoader
from langchain.indexes import VectorstoreIndexCreator
loader = WebBaseLoader("https://lilianweng.github.io/posts/2023-06-23-agent/")
index = VectorstoreIndexCreator().from_loaders([loader])
index.query("What is Task Decomposition?")Output:
Task decomposition is a technique used to break down complex tasks into smaller and simpler steps. It can be done using LLM with simple prompting, task-specific instructions, or with human inputs. Tree of Thoughts (Yao et al. 2023) is an extension of Chain of Thought (Wei et al. 2022) which explores multiple reasoning possibilities at each step.And here is how you can do it with Llama Index:
from llama_index import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader('data').load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)Conclusion
This is a comprehensive guide to crafting effective prompts for AI models, ranging from basic techniques to advanced strategies. By understanding the nuances of prompt engineering, individuals can harness the full potential of language models. Techniques like Chain of Thought, ReAct, Self-Consistency, and Tree of Thoughts offer powerful tools for logical reasoning and problem-solving. Moreover, the integration of external data through Retrieval Augmented Generation (RAG) expands the capabilities of AI models. With these insights and tactics, prompt engineers can navigate the evolving landscape of AI creativity and achieve remarkable results. Embracing experimentation and iteration, they can unlock new realms of possibilities in the field of prompt engineering.
I encourage you to explore further and continue learning about prompt engineering. Dive into the resources and examples I’ve provided to deepen your understanding and refine your skills. Embrace the iterative process, experiment with different prompts, and don’t be afraid to iterate and refine until you achieve the desired results.
Prompt engineering is a dynamic field, continuously evolving as language models advance. Stay up to date with the latest developments and techniques. Engage with the vibrant prompt engineering community, exchange ideas, and share your experiences. Together, we can unlock the true potential of LLMs and revolutionize the way we harness the power of AI.
Join the Conversation!
If you enjoyed this article and want to stay connected, I invite you to follow me here on Medium AI Geek and on Twitter at AI Geek.
