General Manipulation: What it’s Worth and Why it’s Hard

With autonomous driving on the rise, manipulation is the last frontier in connecting the online worlds of ordering goods and the offline world of transporting and processing them. Digitalization has entered every aspect of life from inventorying a warehouse, to agriculture and dating. Some industries have already managed to go entirely digital, delivering movies, music and books, tracking what you like, making recommendations and adapting their content appropriately. Others do whatever they can to streamline the manufacturing, storage and delivery of goods that are bound to remain physical such as clothing, food, cosmetics or vehicles. Autonomous mobility, both on the street and indoors, is a key technology that will allow physical goods to move through space almost as seamlessly as data. The value proposition is tremendous: once you manage to reliably drive from A to B, there is little difference whether you deliver a screw driver, a crate of tomatoes, or a hand-written note to your tinder date. There is a big stepping stone in all of these interlocked chains of digital interactions that prevents them from becoming fully automatic: physical manipulation.

To take a closer look at such a digital chain, consider your wifi-enabled lightbulb kicking the bucket. After an alert from your wink app, you order a replacement from an online distributor (or the app does). There, an autonomous cart sets itself in motion, prompts a worker to pick the right product and put it into a box, which then finds its way to a (soon autonomous) delivery truck, eventually the new bulb makes its way to your front porch, where you pick it up, unbox it, replace it, and trash the old one. This is neither the beginning nor the end of the story. The bulb has to be manufactured from a large variety of goods coming from all over the world, sent to the online retailer warehouse, and stored at the right spot. At the end of its life, it needs to find its way to a recycling plant where its valuable pieces are extracted, and send back as raw materials to manufacturers. As large as the number of manipulation steps in all of these processes is (in the order of thousands), as low is their individual value, which is bounded by the $3.99 a wifi-enabled light bulb cost. Even if including the cost of your hassle dealing with this problem, the events described above are too rare to justify a specialized solution to fully automate it. Here, “fully” means that not only production, distribution, and monitoring is automated, but also the act of replacing the bulb itself.

Manipulation is indeed a very general term involving activities as diverse as picking an item from a bin or carving a face into a pumpkin that happen in places ranging from a factory floor to a kitchen sink. The kitchen alone is a place for a vast variety of manipulation tasks. Imagine yourself sitting in front of a nicely prepared dinner. This involved retrieving the ingredients from your fridge and other local storage, removing them from their containers, peeling, chopping, marinating, and mixing, placing them onto the stove or the microwave, retrieving dishes and cutlery, moving the cooked food onto the plate and finally setting the table. After your meal, the dish has to be removed, cleaned, placed into the dishwasher, and eventually returned to the shelf.

A nice dinner, with dishes, glasses, napkins and cutlery nicely placed.

These ten steps — plus or minus some depending on how the ingredients arrived at your house — are all fundamentally different in nature, potentially messy, and making it difficult to engineer a single robot that can do them all. At the same time, their individual value is quite small. Lets assume you budget $10 for a single meal and expect to pay most of this amount, say $8, for the ingredients. Budgeting 20c for energy (cooking and cleaning), leaves only $1.80 for manipulation, or 18c per task.

Lets assume, you could purchase a robot that is only able to empty your dishwasher, a task that needs to be done once a day, like so:

A Universal Robot arm emptying a dishwasher.

While you might be ready to pay $5.40 (30*18c) for this service a month, — after all, you spent a thousand dollars for your dishwasher — robotics is not quite there yet to deliver at this price. The ticket price of the system shown in the video above is a little above $40000, requiring more than 600 years to amortize, and still 60 years if the same could be achieved for $4000. With a little good will, this is quite realistic given the low cost of brushless DC motors in mass-market applications like e-scooters or drones.

The value proposition of such an arm would raise tremendously if it could serve other purposes, likely requiring it to remove around in your kitchen. Add another $20000 to your bill for an autonomous cart, and this can be done today. Assuming a not-so distant future in which such a cart — at the end of the day the equivalent of four said e-scooters, a computer and a bunch of sensors that soon will be in every car— could be sold for $2000, we are looking at a total cost of $6000 for a mobile solution that can drive around and handle a variety of objects.

Assuming such a device to amortize over five years, it needs to create services for $100 a month, or $3.30 a day, to be viable. Given the marginal cost of your dinner that would fall into the manipulation category this could be done if the robot could make dinner for two, but given current technology, a robot could probably only help with a few of the involved tasks such as setting and cleaning the table, operating the dishwasher or microwave in the foreseeable future. Yet, a robot that could do those, could also do other things like cleaning up items from the floor, restoring order in your household, dusting, or getting you an espresso from your fancy coffee machine wherever you are in your house. Its the ability to do more than one task, that will make a robotic manipulation system valuable.

A Robotic Materials mobile manipulator delivering a drink and cleaning up trash.

What happens before your dinner? You likely ordered your food from an online grocer or picked it up old-school at a local store. While online grocers are stepping up their game, local grocers are unlikely to go away anytime soon. What about a mobile manipulator that could retrieve all your items for you, ready to pick-up at the store for say, $2 per order? A robot the size of a shopping cart could fill multiple orders at once, driving around the store in circles at day and night. Lets say, 20 people use this service a day, netting $40 a day, or $1200 a month. In this case, a robot that could manipulate everything from a single strawberry to a jug of milk would amortize itself in little over four years at the current $60000 price tag.

A Robotic Materials’ hand manipulating strawberries using impedance control and tactile sensing.

There are more than 38000 supermarkets in the US with 3500 or so offering organic and luxury foods. Getting one such robot on average into every tenth supermarket is quite a big deal, exceeding $200M in volume. Its a big number, but selling a $60k ticket item is no easy task, and the business becomes ten times less interesting when the cost of the robot goes down to $6k. Add stocking shelves, inventory and customer service to the tasks the robot can do, and the equation might change, making up for the lower sales volume in numbers.

A Robotic Materials hand picking bottles out of a bin.

Leaving the dinner table and returning to our lightbulb, a robot that could help fill orders might also make sense in one of the more than 14000 hardware stores in the US. Here, a robot could not only retrieve packaged goods, but also fill specialized orders like o-rings or screws. In both retail examples, the value of mobile manipulation is not only the service fee, but by allowing the retailer to combine online and offline business into a single location. Better yet, a fully autonomous solution for handling goods would allow every corner store to serve customers all over the world, and online giants to become corner stores.

A Robotic Materials hand picking o-rings.

Once a robot is able to handle all the hardware stuff is made off, assembling and maintaining them is the next obvious step a manipulating robot could help with. For example, a robot might replace batteries and light-bulbs in your house (ordered from the hardware store), clean the filter in your dryer, or replace a broken door handle.

An autonomous robot assembling a set of gears modeled after a gearbox assembly task.

Robots like those could also help workers assembling smart phones or car motors. Here, cost are measured against the yearly salary of a human worker as well as harder to calculate performance indicators like machine utilization (a machine amortizes faster if run day and night), reliability and worker retention (feeding a machine with parts from a bin can get quite dull). Robots that could do all this might be even easier to build as industry is more inclined to adapt the environment to the limitations of current robots, than you might be in your kitchen.

Yet, even in manufacturing, manipulation tasks are so diverse and there are so many of them, that its hard to attribute more than a few cents to each, and no individual domain justifies the investment that is needed to develop a general robot solution that is good at doing more than a few at once. But nobody has to. Advances in machine learning, tactile sensing, and 3D perception are are starting to converge into low-cost, capable autonomous hands at low cost. The challenge that remains are developing new business models that take advantage of these new capabilities and not to be afraid of disrupting the labor market, but maintain the US’ leadership in science and technology.