Serperio — the voice apprentice for your computer

Siri and Google Now are great, but as with all mobile apps, they are limited. What if you could control your computer in the same way without any limitations?

Meet Serperio:

We get it. Phones and tablets require a lot of screen tapping to get simple tasks like navigation and weather information up. Voice assistants speed things up and that makes them extremely useful.

With Serperio, the idea is to go beyond merely speeding things up. The goal for this PC and Mac software is to learn from its users and solve real problems.

The goal is to go beyond merely controlling a weather, stock, and reminders app. It’s to go beyond placing calls and sending texts with your voice.

Teaching Serperio How to Accomplish Tasks

Lets say you own a MacBook Pro Retina and you want to add “The Imitation Game” to your Netflix queue. As of 2015, you have to complete a long and tedious amount of actions to complete this digital request:

  • Click on a browser such as Safari
  • Click within the search bar
  • Type netflix.com and hit the enter key
  • Place the mouse within the username field
  • Type the username
  • Place the mouse within the password field
  • Type the password
  • Select a Netflix user
  • Place the mouse within the search bar of Netflix
  • Type “The Imitation Game”
  • Hit the enter key
  • Click “add to queue” if the movie was available.

A user that utilizes Serperio as its digital apprentice could teach it how to do all of the above when asked “Add ‘The Imitation Game’ to my Netflix queue.”

For the developers out there, yes this is doable.

There is already an MIT Open Source GUI automation solution developed by MIT, called Sikuli Script, which accomplishes half of the functionality outlined above already.

Sikuli uses screen scanning to determine where things such as search fields are on a screen, regardless of the operating system, the screen size, and the screen positioning. I have already tested Sikuli’s functionality on Windows and Mac in logging in a user and adding a movie to a Netflix queue.

“Yes but how is Sikuli going to be controlled by your voice?” -asks the silly developer

Use Wit. That’s right. Use Wit.

Wit.ai is a grammar engine solution which takes commands such as “set the thermostat to 70 degrees” and breaks it down into machine category parameters such as “interact with a device called thermostat,” “input 70 degrees.”

Wit.ai can be coupled together with Sikuli so that when a user asks “add The Imitation Game to my Netflix queue,” Serperio will know that “The Imitation Game” is the search text and “Netflix queue” is the operation. And if it doesn’t know this, then it can be taught to know it.

Sikuli is smart. I mean, really smart. And it’s all open source. If it arrives to Netflix.com and the user is already logged in, it can tell that by scanning the screen and it skip straight to searching for the desired movie and adding it (it only takes a little bit of programming to get it to do this, and to get it to work with a password manager).

Serperio Will Know Thousands of Commands Out of the Box

You’ll probably never have to teach Serperio how to navigate Netflix using your voice. This is because the software will come pre-built with understanding how to carry out thousands of commands for various programs and websites.

Serperio will know how to add movies to your Netflix queue, and play them, and manage Photoshop and your antivirus software and thousands of other things without you ever having to teach it or touch your keyboard or mouse.

If you do find an odd task that Serperio doesn’t know how to accomplish, you’ll be able to teach it how. And eventually, you’ll even be able to share the things you teach it with Serperio’s developers who can push that knowledge to all Serperio apprentices for all of its users (after verifying their functionality and quality).

“How will text to voice work?” -another silly developer might ask.

Visit Ivona.com. Select Brian for the voice profile which is located underneath British accents. Boom. There you go. Serperio now has a voice.

As with Wit.ai, there is some licensing cost for using this solution. But it takes care of Serperio’s voice while sounding beautiful and without a developer having to create a custom and expensive text to voice solution.

In fact, Ivona voice solutions are utilized by Amazon. What more do you need?

“But how will Serperio make $$$?” -an investor might ask

Serperio won’t need to show or speak a single advertisement to its users.

Instead, it will make its money off of commissions. Every time someone asks to order, say, Call of Duty for the Xbox One, Serperio will make an API* call to Amazon and order the desired item to the user’s desired location. It will make a small royalty off of this transaction as it will with many others.

*An API stands for application programming interface. When you ask Siri “what is the weather like?” Siri takes your GPS location and sends it off to Yahoo with a request such as “weather in Fairfax, VA.” Then a Yahoo server receives that request, searches through a database, retrieves the temperature and climate information for Siri’s desired location, and then sends it back. Siri then takes care of conveying that information verbally, and visually, as programmed.

“How much does an Xbox One cost?” — a user might ask Serperio

An Xbox One costs $350.00. Would you like to order one to your house?

“Yes!”

Would you like to order any video games or extra controllers with your purchase?

“Yes!”

Anyways. You get the idea. A computer apprentice without any limitations.

But it doesn’t stop there. There are dozens of food APIs such as Ordx and GrubHub which can be used to order food from local restaurants (as well as grab a commission). Serperio doesn’t have to limit itself to the APIs currently available either. It can partner with food delivery stores such as Papa Johns and Plated in order to let users order food using their voice.

While Serperio can be taught to use the Papa Johns website to order anything available on it, it would be best if Serperio worked with such a business to make direct API calls to their ordering server and also earn a royalty for each sale. This would eliminate the need for Serperio to be taught how to use a visual site by developers, as well as the time it would take for Serperio to make such a request to Papa Johns.

Furthermore, with credit card information saved by an API such as Stripe, users’ data will be much more secure.

APIs are a beautiful thing. They can get virtually anything done, such as reserving hotels, flights, and restaurant seats. The possibilities are limitless. In today’s day and age, where the internet of things is quickly growing, a voice apprentice is the next logical step in technology. Such software will be able to connect users to their world, restaurants, and other services instead of merely to weather and sports information.

Serperio’s Many Features

If you want to text your friend from your computer, using your voice, Serperio will be able to control a program such as MightyText to automate the diction and message sending process.

This software will also act as a password, address, and credit card manager that is protected with a master password. Apple and Dashlane already provide such services and they are extremely secure. If properly developed, such a feature could provide users with more security than ever before.

Furthermore, Serperio will be able to connect to WiFi devices such as Nest thermometers and WiFi controlled lights. Whereas mobile users have to open up individual apps and then manage WiFi devices manually, Serperio will be able to manage all of them through a simple voice command.

There won’t be any need to press a key or click a button to activate Serperio either. All that it will require is referring to it as Serperio, and then giving it a command. If the software is not addressed by name it will not process any of the text it hears.

Serperio may also have premium features where it processes digital requests without the need of a running computer. For example, if you want to add a movie to your Netflix queue you would need to have a computer open with a microphone so that Serperio can process the request and complete it.

With a server setup for every user, whether it be at a Serperio server center or through a small local device users setup in their home, Serperio will be able to accomplish these tasks without a running computer. The server will be able to receive commands from a phone or microphone enabled device.

So if you’re on your way home and want to order two mushroom pizzas, all you have to do is open up the Serperio app on your phone and ask where you want to order your mushroom pizza from.

Serperio could also integrate with proofreading solutions such as the Hemingway App to alert writers of complex sentences and other writing concerns, beyond grammar and spell checking.

The beauty of Serperio is that it makes sense just about everywhere. There is no reason that Serperio shouldn’t be running on an Xbox device, or even in a car computer, to process requests and connect its users to services and products like never before.