OrderBot — The final lap
ABSTRACT
Exploring conversational agents to identify problems and/ or enhance experiences in the N5 campus of the Srishti Institute of Art, Design and Technology, while also understanding the kind of conversations between a human and a machine. This includes delving into the history of computing- it’s past, present and speculating the future of AI- in Python!
INTRODUCTION
The idea of bringing a conversational agent to the chaotic environment of N5 seemed like a challenge in the beginning. We started thinking of the areas in the campus that needed interaction.
Identifying problems -
The first thing that we thought of was machines that didn’t come with a user manual and would be tough for a person who isn’t used to them to use. We thought of the tools in the workshop, the water cooler, the printer and so on. We then thought of systems that had some kind of glitch — the ID system, the canteen, the floor plans etc.
We came down to two main ideas — an interactive device to help you locate your classroom, and a device to help with the canteen food ordering service. The canteen seemed like something that needed more assistance and needed to be organised.
The current scenario -
The canteen is one of the places with the highest footfall in N5. Currently, at peak hours, there is a lot of chaos. Each student is in as much of a rush as the next and they all want their food as fast as possible. The canteen uncle has to tend to three main jobs — taking orders, handing people items from the counter and returning the change. We thought about improving the efficiency of this system.
INITIAL EXPERIMENT
OrderBotV1.0
Initial ideas and plans
Our initial idea stemmed common food delivery apps, where you order your food from your phone, and the order goes to the kitchen directly. We even thought of changing the cash system by introducing a card with electronic balance, that the users could swipe at the time of order collection. However, we couldn’t completely substitute the canteen uncle, because the agent was supposed to make his work easier and not completely overpower him. We then decided to make an app — like an interface that sits on the counter itself, to help with taking orders. On observing how the canteen uncle had to run around taking orders, collecting change and also manning the counter- we concluded that the only part of his job that we could reduce was taking orders.
How the system works
After much deliberation, we came up with OrderBot — a conversational interface that takes your order in the N5 canteen. Our system has two main parts — one, a screen with the various options and categories of food that you can choose from, and two, an interactive feature that lets you place your order by simply talking to the machine. We added the interactive feature because using an app adds a certain mechanical touch; when the app is talking to you, it’s closer to the original experience. You navigate through categories, to choose individual food items, and they get added to your order.
At the end, you can even give the machine special instructions that it will pass on to the kitchen. There are two payment methods, cash or Paytm©. The Paytm© option leads you to the QR code that you would generally use to pay for your food, and the cash option generates a unique QR code that leads to your receipt. You show the receipt at the counter, and dig in!
Why it didn’t work
After testing our OrderBot on a few people, we realised that the whole process had become mechanical, it lacked the conversational aspect of it. Even the canteen uncle told us that the interaction with the Bot needs to be a bit informal and natural.
Displaying our ideas to the PG students helped us get valuable feedback. We were advised to install our Bots in different places of the canteen instead of installing all of them at the counter, this would reduce the rush at the canteen significantly. Many people suggested including a feature that would allow the user to order from anywhere in N5. However, people may not turn up on time at the canteen and then the food would be wasted. We were also suggested to include a popular food items and previous order list. Many of the suggestions pushed us to think beyond the conventional interface and to make the OrderBot experience more engaging and interesting.
OrderBotV2.0
Changes in ideas and plans
Taking all this into consideration, we made a few changes while coding our Bot. We included popular items, favourites of the day.
We also added a feature that stores your order history, for the user to refer to while ordering.
To make the interaction more natural and interesting, we added a few general questions like “who are you?” and “how are you?” to which the bot gives an appropriate response. It also has a couple of witty outcomes if you ask a completely random question. If it doesn’t understand what you’re saying, it will reply with an “I don’t understand, what do you mean?” We’ve included the option of directly selecting an item too, if you know the menu off hand.
We also added a function that randomly picks between three or four outputs for each set of statements — to ask if anything else has to be added to the order, and to tell the user that an item has been added. We thought this would remove the monotony of the same response over and over again.
We have also added a help section, if anyone is confused about how to use OrderBot. Typing “help” or “I’m confused” will give you a brief set of guidelines as to what all you can ask for.
By including all this, we thought that now the interaction between the user and the OrderBot would be more natural, appealing even.
What we had to improve
The feedback that we got for our A2 was mostly code related. We were asked to structure the code properly, tidy it up and also add comments in the code so that other people could understand it easily.
OrderBotV3.0
Our final outcome (after much deliberation)
Since the basic structure of our prototype was in place — the flow of the code was decided. Now we just had to improve the functioning of the code, and make it more structured. The simplest way to structure our code was move all the functions into a new module, and import that module into the main file. It would improve readability and shorten the main code. We moved all our functions into a module called OBFunc.py. We moved the contents of the “help” option into a new file. We also moved the long lists that were part of our code to new files, so that they wouldn’t be visible in the code itself. We also added comments to the entire code, to better explain what was happening.
Other than organising the code, we thought of improving its functionality by doing a few things. The primary function of the bot is to record an order and generate a receipt that you can show to Anna and proceed to pay him. Thus, it reduces the load of orders on Anna and the kitchen. We wanted to bring this feature into our coded prototype too. We researched a bit, and learnt how to write data from python into an HTML file. As the bot generates your receipt, it writes the receipt contents into an html file (receipt.html). We used a module called webbrowser() to open the html file in the browser.
The next question we tackled was getting this receipt to Anna. In our initial interface, we had used a QR code that the user scans to display the receipt on their phone. We figured that this method could work even with our coded prototype. Initially, we made one QR code of the receipt.html file, and used webbrowser() to open it. But it didn’t work, because the QR code was linked to one particular receipt, and the receipt generated kept changing. We then found a module called pyqrcode() that generated QR codes within python. The code generates an .svg QR code of the receipt, and opens it in the browser using webbrowser().
Since the two files (receipt.html and receipt-url.svg) were located locally in the device, the webbrowser() couldn’t open them. This came up in the form of another problem which could be solved by setting up a local server using python.
We had to then connect the phone that we were using to scan, as well as the laptop to the same Wi-Fi network. By changing the address of the url to the IPV4 of the laptop, the receipt and QR code both successfully opened.
With all these features added, OrderBot was beginning to look closer to the original idea we had thought of, but still the concept of “voice” hadn’t been implemented. To add a voice to our bot, we imported a module win32com.client() which uses the voice of the windows machine in python. Using a function called Speak() we made OrderBot say all the things that earlier used to be simply displayed.
We were happy that OrderBot was coming to life, but we wanted it to recognise voice! We tried overnight to import different modules that would help, but some completely unrelated error messages would appear and would stop the installation. Finally we downloaded a .whl file to run PyAudio — the module to help us with the speech recognition. How this process works –
There is a function called Microphone() that takes the microphone of the device as a source for the voice. A function called adjust_for_ambient_noise() adjusts the level of the room tone for one second, so that the device doesn’t get confused as to which instruction it is supposed to record. A function called Listen() records a phrase for 8 seconds from the source (here, microphone). A function called recognise.google() uses Google’s web API to change the voice to a string, that can be later used in the code. OrderBot asks you to confirm if you said what it heard. If not, you can re — enter the instruction. Following this, it prints a receipt.
The real life scenario
If our idea was to be implemented in the canteen, this is how we speculate the system may work. There will be a database of the names and corresponding student ids of all the students at Srishti that will be fed into OrderBot. As you start using the device, your order history will keep building, and you can refer to your previous orders to re — order from them. For the receipt generation, OrderBot will give out a local WiFi signal, which devices in the vicinity will connect to automatically. The receipt will then be uploaded to the local server, which all the devices will be able to access. The users can then scan the QR code and show Anna the receipt.
SUMMARY
Designing OrderBot to successfully record your order and make sure your delicious food is just what the doctor ordered is a succession of brainstorming, rigorous feedback, trial and error and as always, returning to the code!
Certain elements stayed true to the original idea- being able to place your order through a screen by talking to it, while being able to browse through your options and even pass on special instructions to the kitchen. Scrutinising our idea through feedback sessions helped us add elements such as recording previously ordered items, popular food items of the day and a few witty comebacks to keep you company! Progressing past a more mechanical conversation to a more natural flow meant adding smaller details- from greetings of the day, informing the user that their item has been added, helping a finicky or a ‘confused’ user, to. Realising this with code also meant equipping it progressively with more sequential and conditional statements. After determining paths that a potential user may take to successfully place an order, improving the functionality of the code came next, making the quality of user experience a memorable one.
See OrderBotV3.0 in action!
