Building a Menu-Aware NLP System for Real-Time Order Structuring

NLP System for Real-Time Order Structuring

The client provides an AI assistant for Drive-Thru restaurants that uses a custom NLP system to interpret natural customer speech and convert it into structured, kitchen-ready orders. The model handles complex, brand-specific menus with customizable combos, modifiers, and promotional items. It recognizes varied phrasing, synonyms, and marketing terms, and maintains contextual awareness across multi-turn conversations. The system supports slot filling, multilingual input (English and Spanish), real-time menu updates, and upselling logic.

Challenge

Flexible Language Understanding: Customers use a wide range of informal phrases, synonyms, and branded terms to describe menu items. The NLP system must interpret expressions like “double burger with cheese” or “Mega Cheddar” as consistent, structured entries — despite non-standard or creative language.

Context Management Across Dialogue Turns: Orders are rarely linear. Customers often pause, revise, or add items and modifiers across multiple sentences. The system must maintain order context throughout the interaction, updating the structured output dynamically without losing track of prior information.

Menu Structure and Dependency Resolution: Each restaurant’s menu is transformed into a structured, AI-ready format that includes combos, optional and required components (e.g., drinks, sauces), and custom item hierarchies. The NLP system handles these dependencies, detects missing information, and adjusts orders dynamically. It also applies pricing rules — recognizing when individual items form a combo and automatically restructuring the order for accurate pricing and fulfillment.

Real-Time Adaptation and Upselling Logic: Menus change frequently with new items, promotions, or out-of-stock components. The NLP system must incorporate updates without delay and support upselling by recognizing when a customer’s order could be improved or bundled for a better deal.

Solution

1. Custom NLP Engine: A domain-specific NLP model was developed to handle informal, inconsistent, and brand-specific language used in food ordering. Originally built in-house for accuracy and control, the system is now gradually evolving toward LLM-based solutions (e.g., OpenAI) to support greater flexibility and scalability.
2. Intelligent Menu Structuring and Pricing Logic: POS menus were restructured into a machine-readable format with clear hierarchies, combo definitions, and dependency rules. The system accounts for required and optional components, nested items, and pricing logic, enabling the NLP model to automatically restructure orders — e.g., recognizing when separate items qualify as a combo and adjusting the price accordingly.
3. Synonym Mapping and Branded Term Handling: Extensive synonym dictionaries were created to recognize varied customer phrasing, such as “double cheeseburger,” “burger with double cheese,” or “mega cheeseburger.” The system also maps marketing-driven names (e.g., “Mega Cheddar,” “Hero Meal”) to the correct internal product IDs, ensuring consistent interpretation across informal and branded language.
4. Multi-Turn Dialogue and Context Tracking: The system maintains order context across multiple dialogue turns, allowing customers to add, modify, or revise items mid-conversation. This ensures the order structure stays coherent and up to date, even with fragmented or non-linear input.
5. Ground Truth Data and Model Training: Structured “ground truth” data was created by mapping real customer speech to JSON-style order representations. Training combined manual transcription, automated labeling, and segmented datasets by brand to ensure broad coverage and model generalization.

Features

Natural Ordering Experience

Customers can speak naturally using informal, unstructured language, including pauses, corrections, and filler words. The system interprets input without relying on fixed syntax, enabling a smooth, conversational ordering flow.

Combo Detection and Dynamic Pricing

The system detects when items qualify for a combo, restructures the order, and applies bundled pricing, handling dependencies like missing drinks. Combo behavior was customizable per restaurant — either applied automatically or confirmed with the customer based on each location’s protocol.

Support for Branded and Promotional Menus

The system interprets marketing-driven item names and maps them to the correct structured menu entities, ensuring accurate recognition. It also adapts quickly to seasonal updates and promotional menu changes.

Multi-Turn Interaction and Corrections

The system tracks order context across dialogue turns, allowing customers to add, modify, or clarify items mid-order while keeping the overall structure consistent and accurate.

Multilingual Understanding

The system automatically detects the spoken language — such as English or Spanish — and adjusts its interpretation accordingly, ensuring accurate and consistent responses across supported languages.

Upselling and Suggestive Ordering

The system identifies real-time upsell opportunities — such as offering a deal on fries — and presents relevant suggestions to the customer, driven by integration between NLP output and a dedicated promotion logic layer.

Development Process

1. Model Type and Evolution

The NLP system was first developed in-house, specifically for handling fast food orders with informal and varied customer language. It was designed to be cost-effective and easy to scale across many restaurants. As the project grew, the team began moving toward large language models (LLMs), including OpenAI, to make the system more flexible and quicker to adapt to new menus and restaurant setups.

2. Data Preparation and Ground Truth Creation

Real customer audio recordings collected from Drive-Thru interactions at partner restaurants. We used a mix of manual and automated transcription, with timestamps marked for each speech segment.

Transcribed orders were mapped to structured, JSON-style representations reflecting the full logic of the order — including main items, modifiers, dependencies, and updates made over multiple dialogue turns, for example: “Big Mac with ketchup and mayo, a Coke, and fries.”

the dataset was split into training, validation, and test sets with careful segmentation to ensure balanced representation across different restaurant brands and avoid overlap between recordings from the same day or customer session

3. Menu Structuring for NLP Integration

Menus from restaurant POS systems were not designed for natural language processing — they lacked structure, consistency, and machine-readable formatting. To deal with it, the team manually restructured each menu into a format suitable for NLP interpretation. This included:

• Item hierarchies: Mapping main items, categories, and sub-items clearly.
• Combo logic: Defining bundled meals and their components (e.g., main, side, drink).
• Dependencies: Specifying required or optional elements (e.g., a drink is mandatory in certain combos).

Extensive synonym dictionaries were developed to interpret the many ways customers refer to the same menu item — for example, “double cheeseburger,” “burger with double cheese,” or “mega burger.” The system also supports branded and promotional names like “Mega Cheddar” or “Hero Meal,” linking them to the correct internal product IDs to ensure consistent recognition and accurate order processing.

4. Training and Evaluation

NLP models were trained on a curated dataset of real customer interactions, including transcribed audio and structured orders. The data captured a wide range of restaurant brands, menu setups, and natural speech patterns — such as informal phrasing, corrections, and background noise. Training focused on handling language variability, understanding complex menu logic, and maintaining context across multi-turn dialogues for reliable performance in real Drive-Thru settings.

Model performance was assessed using accuracy, coverage of key menu phrases, and precision in handling domain-specific terms to ensure reliable interpretation of food-related language.

Model behavior was continuously tracked through performance metrics, with special attention to critical errors — such as misrecognizing “cheeseburger” as “fishburger.” When accuracy thresholds weren’t met, additional data was collected and the model retrained to improve reliability and reduce high-impact mistakes.

5. Order Logic and Context Handling

The NLP model outputs a structured representation of each order, identifying main items, modifiers (e.g., sauces or toppings), and checking for completeness based on predefined menu rules. For example, if a combo is ordered without a drink, the system detects the missing field and prompts the customer with a follow-up question like “What would you like to drink?”

This logic is handled by a dedicated management layer that ensures every order is complete and valid. The system also maintains context across multi-turn dialogues, allowing customers to revise or add to their order naturally, with real-time updates to the structured output.Technical Highlights

Impact

• High Accuracy in Order Interpretation

The NLP system achieved 85–88% accuracy in converting natural, informal speech into structured orders, reducing manual corrections by over 70% and ensuring consistency across varied customer phrasing.

• Robust Handling of Complex Language

With over 90% recognition of synonyms and branded terms, the model reliably interpreted phrases like “burger with double cheese” or “Mega Cheddar” and maintained context across dialogue turns with over 95% consistency.

• Faster Menu Updates and Real-Time Adaptation

Thanks to structured integration, the system supported same-day updates for new items, promotions, and stock changes — keeping NLP performance aligned with fast-changing menus.

• Increased Order Value through Smart Prompts

Integrated upsell logic, powered by structured NLP outputs, helped boost average order value by 10–12%, identifying upgrade opportunities and suggesting combos in real time.