Efficient AI Programming: Iterators in LLMP for Complex NLP Tasks
In the world of software development, efficiency isn’t just a buzzword — it’s the cornerstone of innovation. While iterators are a common concept in software applications it can be quite costly when used together with llm tasks.
This is where the Large Language Model Programming (LLMP) framework shines. LLMP leverages the capabilities of iterators within the realm of generative Natural Language Processing (NLP), making it simpler for developers to craft and manage structured prompts for AI-driven tasks. By abstracting the complexities involved in prompt engineering, LLMP allows developers to focus on what truly matters — creating impactful software solutions.
Our previous article introduced LLMP and its core features. Today, we will delve into a practical implementation, focusing on LLMP’s iterators. We aim to provide a clear understanding of how these iterators can be used within LLMP to generate structured outputs efficiently.
As we explore the nuances of this framework together, you’ll gain insights into how LLMP’s iterators can be seamlessly integrated into your NLP tasks, thereby enhancing productivity and fostering innovation. Let’s embark on this journey of discovery and see how LLMP can elevate your programming projects to new heights.
Harnessing ‘for’ Loops in Output Models for Efficient LLM Calls
In the realm of software development, particularly when working with structured data, we often encounter the need to process lists and sequences. This task can become cumbersome when we require repetitive calls to large language models (LLMs) to generate outputs for each item in a list. Not only does this approach strain resources, but it also disrupts the developer’s workflow, making the process inefficient and time-consuming.
The Challenge of Lists in NLP tasks
Consider a scenario where you’re working with a series of book titles, and for each title, you need to generate detailed metadata, including genre, author details, and publication year. Traditionally, this would involve sending each book title as a separate prompt to the LLM, waiting for a response, parsing the output, and then repeating the process for the next title. This method can quickly escalate into a maintenance nightmare, especially as the list of titles grows.
The LLMP Solution: ‘for’ Iterators in Output Models
To streamline this process, LLMP introduces an elegant solution: the use of ‘for’ iterators within output models. By incorporating a simple for-loop into your output model, LLMP allows you to instruct the language model to iterate over a list of inputs and generate a corresponding list of structured outputs in a single call. This feature not only saves valuable time and computational resources but also simplifies the complexity of your code, allowing you to maintain a clean and efficient workflow.
Imagine you could simply define an output model that tells the language model, “For each book in this list, provide me with the structured details.” LLMP translates this into a loop within the LLM’s prompt, handling the iteration internally and giving you a neatly structured list of outputs in one go.
Practical Example: Using Iterators in LLMP
When we’re looking at the capabilities of the LLMP framework, one of its most powerful features is the ability to handle lists of data with ease and efficiency. The use of iterators within output models can transform the way we approach bulk operations, optimizing the way we interact with large language models (LLMs) for structured output generation.
Let’s dive into a practical example to illustrate how LLMP simplifies the process of generating a series of related outputs based on a single input.
Preparing the Input
Suppose we’re working with a dataset of books, and we need to generate a list of potential sequels for a given title. In traditional approaches, this might require running a task prompt multiple times — once for each item in our list. This not only consumes more resources but also adds complexity to our code.
However, with LLMP, we can streamline this process. Let’s use the example from our previous article and create a list of sequals from a book title, author and release year. For this we define our input and output model using pydantic:
from pydantic import BaseModel
class BookInput(BaseModel):
book_title: str
book_author: str
release_year: int
class SequalOutput(BaseModel):
sequels: list[str]And then, we instantiate our LLMP program:
from llmp.services.program import Program
program = Program("gen sequels", BookInput, SequalOutput)When initiating this program LLMP will generate an instruction from the input and output model on it’s own. We can check the instruction generated as follow:
print(program.job.instruction)
>>> ""Given the input of a book's title, author, and release year, generate a list of sequels to the book."Generating a list of sequels
We will use this program to generate a list of sequels that serve as our iterator for example. Therefore, we call our program while passing an input object:
input_data = {
"book_title": "Harry Potter",
"book_author": "J.K. Rowling",
"release_year": 1997
}
output = program(input_data)So as defined in our output model our program will return a list of strings. LLMP’s validation engine takes care of robust output generation and validation. The output, as expected, would be a list of sequel titles:
print(output.sequels)The output of LLMP program is a dotdict so we can call it with dot notation or using dict notation in this case with the key “sequels”.
['Harry Potter and the Chamber of Secrets', 'Harry Potter and the Prisoner of Azkaban', 'Harry Potter and the Goblet of Fire', 'Harry Potter and the Order of the Phoenix', 'Harry Potter and the Half-Blood Prince', 'Harry Potter and the Deathly Hallows']As expected our program generates a correct list of Harry Potter sequels.
Iterating Over the Input List
LLMP further allows us to iterate over a list of inputs directly in the output model. This is done by defining a for-loop within the output model:
from pydantic import Field
class InputObject(BaseModel):
book_list: list[str]
class OutputObject(BaseModel):
book_data: list[BookInput] = Field(rule="for each $book in {book_list}")What have done here?
First we have defined a new input model with a single field book_list as our input for our next program.
Second, we have defined a new output model for our next program using a list of our initial BookInput Model for validation. Furthermore we have add a rule to our new output Model:
“for each $book in {book_list}”
Let’s break down the rule first
A for loop starts with ‘for each’ followed by a counter variable ‘$book’ with ‘$’ prefix. Our rule ends with the sequence we want to iterate through ‘in {book_list}’. We used an placeholder {book_list} as the sequence we want to iterate though. Placeholder can contain any argument defined in the input model an can be used as default values, within rules and as options in multiple choice and single choice output fields. A placeholder will be replaced before executing the generation task. We could also define a sequence directly within the rule without using placeholder.
Based on the output model we would the following program to generate a list for each title in book_list. The resulting list is defined as BookInput so each item should contain book_title , book_author and release_year as its keys.
Let’s now initiate our new program and call it with our list of Harry Potter sequels:
program_iter = Program("gen sequels data", InputObject, OutputObject)
result = program_iter({"book_list": output.sequels})
print(result.book_data)[{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Chamber of Secrets',
'release_year': 1998},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Prisoner of Azkaban',
'release_year': 1999},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Goblet of Fire',
'release_year': 2000},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Order of the Phoenix',
'release_year': 2003},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Half-Blood Prince',
'release_year': 2005},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Deathly Hallows',
'release_year': 2007}]As expected our new program generates a list item for each Harry Potter sequel.
The beauty under the Hood
The beauty of LLMP is that much of the complexity is abstracted away, but for those curious about what happens under the hood, we can inspect the instruction generated by LLMP as well as event and generation logs. As LLMP’s main purpose is to allow and maintain a seamless coding flow while developing, we can quickly define a NLP task and easily use it within our application with reliable type validation for each output key. While this program uses intuitive semantics for its input and output model it is prone to return the desired outputs we can imagine other scenarios where we are faced with more challenging tasks. In this case we can leverage the LLMOps capabilites of LLMP. Each job is stored and indexed within our filesystem and each generation is logged into to job specific generation_log and event_log .
First let’s check the instruction:
print(program_iter.job.instruction)
>>> 'Given a list of book titles, create a list of dictionaries where each dictionary represents a book and includes the title, author, and release year.'Monitoring and Optimizing with LLMP Logs
LLMP gets that, and it comes packed with a logging feature that keeps track of every output and event without breaking your stride. This means you can circle back anytime to see what’s been happening under the hood.
Understanding Generation and Event Logs
With LLMP, every output generated by your program is recorded in a generation log. This log provides a historical record of the inputs provided and the outputs received, making it a valuable resource for identifying and addressing any issues that may arise. We can identify and label wrong generations and create a human-labeled dataset for auto-optimization and/or fine-tuning, which will be discussed in the next article in more depth.
Similarly, the event log captures a detailed metrics of every interaction with the job. This includes the time taken for each event, the success or failure of each generation, and other metadata that can be useful for performance analysis.
Inspecting LLMP Logs
Let’s explore how we can check these logs in LLMP for our previous program_iter
print(program_iter.generation_log())
>>> [
{'event_id': '907ed5df08ed4d45ba6169b8db4a0f2a',
'input':
{'book_list': [
'Harry Potter and the Chamber of Secrets',
'Harry Potter and the Prisoner of Azkaban',
'Harry Potter and the Goblet of Fire',
'Harry Potter and the Order of the Phoenix',
'Harry Potter and the Half-Blood Prince',
'Harry Potter and the Deathly Hallows'
]},
'output':
{'book_data': [
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Chamber of '
'Secrets',
'release_year': 1998},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Prisoner of '
'Azkaban',
'release_year': 1999},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Goblet of Fire',
'release_year': 2000},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Order of the '
'Phoenix',
'release_year': 2003},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Half-Blood '
'Prince',
'release_year': 2005},
{'book_author': 'J.K. Rowling',
'book_title': 'Harry Potter and the Deathly '
'Hallows',
'release_year': 2007}
]}}]and secondly we can also view metrics, token usage and failure rates by inspecting the event log:
These logs are like a detailed diary for your program. They’re perfect for spotting when things go off track or just for understanding how your program’s doing over time.
print(program_iter.event_log())
>>> [
Event(
event_id='c944cc828f224390929ed251c21ad381',
timestamp='20231107180433',
event_type=<EventType.JOB_CREATION: 'job_creation'>,
event_metrics=None,
job_setting={'instruction': None, 'example_id': []},
job_version=0,
example_id=None,
example_version=None,
extra=None,
ref_event_id=None
),
Event(
event_id='907ed5df08ed4d45ba6169b8db4a0f2a',
timestamp='20231107180445',
event_type=<EventType.GENERATION: 'generation'>,
event_metrics={
'verification_type': 1,
'execution_time': 10.300012111663818,
'token_usage': 988,
'model_name': 'gpt-3.5-turbo',
'model_config': {},
'failure_rate': 1,
'errors': []
},
job_setting=None,
job_version=None,
example_id=None,
example_version=None,
extra=None,
ref_event_id=None)]As we can see we have two events in our event_log one job creation event and one generation event, where the second comes with event_metricsas token usage and the model configuration, etc. Despite this two types, LLMP also tracks events for adding/deleting/updating examples, optimization runs, changes on instruction and rollbacks. The event log serves as Version Control system with performance tracking.
What’s Next?
This log review sets the stage for the clever stuff LLMP can do, which we’ll get into next time. LLMP’s logging and retrospective analysis capabilities set the stage for advanced features that we will cover in upcoming articles. Topics such as few-shot prompting, dynamic example generation, and auto-optimization will be discussed, demonstrating how LLMP not only simplifies the initial integration of large language models but also evolves with your project to continually enhance performance.
In our next article, we’ll dive deeper into these optimization techniques, exploring how LLMP can learn from its own logs to refine and perfect its prompts. Stay tuned to see how LLMP can turn a good AI integration into a great one, with minimal effort on your part.
