Prompt Engineering Alone Won’t Get You Far

I’ve been emphasizing for some time that after covering the basic uses, prompt engineering alone will hit a point of diminishing returns.

The key to unlocking more complex use cases and creating reliable automations is to have the right data.

When you think about creating prompts, consider what information or context you can provide to the AI to help it follow your instructions effectively.

To understand this better, imagine AI as a new employee.

Just like a person starting a new job needs to understand their role, AI requires context to perform well.

If you don’t provide enough context and someone makes a mistake, you can’t fault them if you didn’t explain things properly.

The same applies to AI; it can’t guess what you’re thinking without the right context and information.

So before crafting a large prompt, consider what data sources you can use to enhance it and make it more reliable.

Here are the sources I usually seek out:

1. Internal Databases

Any data you collect that you believe could help AI do its job should be considered.

Don’t just think of databases as SQL or Excel files.

Company reports, Product descriptions, Tickets, Conversations, bug reports, internal logs and even your emails are all fair game depending on the task.

So look through your internal databases to find information that can help make your answers more relevant to your goals and think creatively.

2. APIs

APIs are a great way to bring in structured data for any task you need.

The main advantage of using APIs is that they provide all the data in a structured format, which simplifies the process of building your project without requiring advanced data engineering skills.

Most chatbot building apps out there include a feature that allows you to integrate an API.

However, the downside is that most APIs are paid services because they come from third parties who have taken the time to organize the data you need.

3. Scraping

If you can’t find the information you need in your internal databases and you can’t find a provider to give you this data in structured form, you can search the public internet and gather this data yourself.

Think of scraping as going out to the internet, looking for information that’s relevant to your project, and collecting it in bulk.

You can scrape almost anything you find on the internet.

And this includes social media, news outlets, forum, blogs, and so on. If it’s available on the internet for everyone to see, you can scrape it.

All you need is a bot that can visit the websites you’re interested in and gather all the information from them.

Most people might not realize this, but you can learn a lot about a person or a company by knowing where to look on the internet.

For instance, imagine you’re creating a cold outreach chatbot…

By digging through a company’s website, you can uncover valuable information such as who the decision-makers are, their budget, their top projects, the products they’re launching, and whether they fit into your target market.

Similarly, if you were designing a recruiting chatbot, you could try and infer what positions they are hiring and what roles they might need by examining their blog posts, social media updates, and new website content.

In short, the internet is a treasure trove of insights if you know where to search. And you can only tap into this resource if you understand how to collect and organize this data with data scraping.

The great thing about scraping is that it’s often free, making it a smart choice if you’re working on a budget-conscious project.

But the downside is that it requires technical skills. You need to know how to code and have strong data engineering skills to organize the data you collect from websites.

For those who aren’t tech-savvy and don’t know how to code, there are data scraping services available, but they can be quite costly.

Now, the goal of this post was to make you think and broaden your perspective on potential data sources that can enhance the chatbots and AI automation you’re developing.

Sometimes, knowing where to find the right data is crucial to the success of a project.

And only after you have the right data to begin with, you can then your prompt engineering skills shine.

That’s all for today’s article!

If you liked it, you’ll also like my Ultimate Prompting Guide.

Where I will teach you how to create highly effective prompts and AI automations.

Here’s what you will learn:

• Easy step-by-step approach to create highly-reliable super prompts.
• The right way to create prompts that are highly contextual to you;
• Discover the best prompt engineering guidelines available today;
• The simplest step-by-step approach to create AI automation workflows;
• How to create AI apps with Python

Please check it out: https://lucasaibuilder.gumroad.com/l/ultimate_prompting