Technically Speaking: Building B.I.L.L. — pt.II
Last year we launched one of our most ambitious projects to date, the Bot Initiated Longevity Lab (B.I.L.L.), a robot-augmented system to clean and repair sneakers.
The culmination of two years of close collaboration with Nike’s innovation team, B.I.L.L. was built as much to extend the life of pre-worn sneakers as to bring the topic of circularity closer to consumers in an exciting way. As B.I.L.L. continues its journey from Niketown London to Nike’s European Headquarters in Hilversum, we sat down with two of the PCH engineers behind its development, Florian Born (hardware extraordinaire) and Christian Kokott (software superpower) to talk about their unique processes, challenges, and the tech that made it happen.
This is a continuation of a previous interview. Read Part I here
The shoe-cleaning process ends with the option to place a patch on the 3D model of the shoe — was this interaction very difficult to achieve?
CK | Yes, this was a key interaction because at this point the customer basically interacts with the robot: so they see their shoe in 3D and they tell the robot exactly which patches to put where. So it had to be fun and fluid, from a user experience perspective, but also reliable.
We have 24 different patch options that the customer can select from and infinite locations on the shoe to patch on, so we built a patch editor specifically for this interaction. Since all the control was on a modern Web-based UI, we used three.js for implementing the patch editor — which is a really cool 3D web library.
An interesting challenge here was the calibration of the physical robot space to our digital editor space. Like, these are both 3D environments, right. But managing how the robots understand the patch location in relation to the shoe meant we had to align coordinate systems to make sure that, when I give it a coordinate on the shoe, both robots understand where this position is in 3D space, and the orientation and rotations are understood in the same way.
The model also had to look exactly like it looks in real life, so when you’re putting patches on your sneaker, it looks how it’s gonna turn out. For this, we created an optic approximation, something that’s good visual quality and, from a computing perspective, still smooth and fast.
At the same time, there are these constraints from the physical process to consider, that don’t exist in the digital version. Like, the robotic arms have limited orientations and positions and, depending where you put the patch, you need to change the path — and then we can’t just rely on the digital approximation of these constraints. We filtered out a lot of things but then we used further mechanisms like a force sensor on the robotic arm itself, to adjust to any imprecisions.
How did you apply the force sensor?
FB | We used it in a number of ways. In B.I.L.L. we have these two robotic arms, we call them the Shoe Handler and the Patch Bot. The Shoe Handler is the arm that picks up the adaptive last with the shoe on its end effector and escorts it through the whole journey. We equipped this arm with a force sensor that we use with the corresponding RSI [robotic sensor interface] for sensor-based motion control and real-time motion planning.
We have 3 pairs of shoe lasts to cover the entire adult shoe size range — small, medium, large, left and right — and each of these lasts is equipped with the passive side of a tool-changer. The first test of the robot is to go into the tool changer and pick up this last. All these positions are taught, but because we are quite lightweight, modular and running in a retail environment for 10 hours a day, it can happen that something in the system shifts by one or two millimeters, and this can throw the robotic arm off if it goes on a blind flight.
So, here we use the force sensor for a smart pick-up approach. Meaning, we go to the known position where the tool changer entry is supposed to be, and then look for the engagement port. And if it’s struggling and realizes it’s putting a lot of pressure on the Z-axis to enter into the tool changer, it makes incrementally bigger circular movements to try and find where to enter. And once it finds the tool changer, it remembers the last position and automatically uses this as the first attempt location for the pick-up next time. So essentially it combines the tool-change action with a smart calibration during each pick-up and return.
We also use the force sensor for the cleaning of the shoe. We have a 3D scan of the shoe [that informs the cleaning path] but this might have imperfections on it, like little artifacts that tell the robot to move a little too close to the brush, and that’s where the force sensor comes into play. So, as soon as the pressure between the brush and shoe becomes too big, the robot automatically corrects its cleaning point to hold a constant pressure. This is important to ensure a consistent cleaning result on each shoe, especially because the shoe is not like a solid block, it’s more of a semisoft object that can deform by applying pressure to it.
Finally, it’s also important for what we call the ‘robotic kiss’, where the two robotic arms come together to apply a patch to a shoe. There’s a kind of an orchestration between both of the arms, where the Patch Bot arm is on a blind flight towards the shoe and doesn’t know when to stop, and the Shoe Handler is holding the shoe and waiting for this pressure from the Patch Bot. Once it senses this pressure, the force sensor helps to ensure it’s enough that the temporary adhesive of the patch actually sticks to the shoe, and at this point the system commands the patch bot to, like, wiggle in the patch and retreat.
We’ve spoken a lot about the Shoe Handler, the robotic arm responsible for carrying the sneaker through its cleaning motions. But B.I.L.L.’s equally important other arm, the Patch Bot, peels the repair patches from one of two dozen reels and applies it to the poised sneaker. Could you go into a bit more detail about the peeling mechanism and the gripper that was developed for this purpose?
FB | Sure! Usually for this kind of task with a single patch, one would build a complex dispense mechanism. But because we had 24 different patches, it would’ve required 24 motors and the machine would have ended up being super clunky. So we tried to think about how to leverage the robotic arm’s flexibility and versatility, to streamline the engineering of other components.
First, we looked at how humans would actually peel off stickers or patches from a reel. On a robotic arm, it’s super hard to peel it straight off and without ripping the tape, especially if you think of robotic mechanisms like vacuum gripping. So we tried to mimic a human hand roll/peeling motion, keeping constant pressure on the tape, using the same needle gripper for rectangular patches and circle patches. For the rectangular patches we also equipped the gripper with a flexible ramp integrated with a suction gripper.
But then the question was, what happens after the peeling? We have an empty spot on the patch reel and to avoid requiring a manual reload, we decided to leverage the robotic arm again and integrated another function into the gripper, which is a little clamp with a contrast sensor. Basically it orients itself according to markers on the side of each reel, and uses them to know when and where to grip the tape, and how much to pull it down so that the next patch is in place.
Apart from that patch housekeeping functionality, this Patch Bot gripper is also important for the scanning we spoke about (see Part I). When the Shoe Handler is presenting the shoe to the camera, it needs to have a homogenous backdrop to make sure that it doesn’t pick random colors or objects from the background that could distort the model. So the Patch Bot also uses the clamp to slot into a mechanism that opens up a screen that provides a white background for scanning. So in the spirit of automation, it’s very multi-purpose.
“It’s an interesting balance of not having to overthink everything to the maximum, but also a challenge to make something chaotic still look cohesive.”
Because it was always going to be customer-facing you had to think of form as much as function. Did that add a lot of complexity?
FB | Hard to say. On one hand, trying to maintain this whimsical or friendly aspect of the machine actually gave us a lot of flexibility in terms of how we designed because we didn’t need to be as rigid about the materials we chose or techniques we used. We could go for, like, some more untraditional ways of doing things. But on the other hand, this came with some issues of finding the right components. Like the ITEM profiles we used for the base gave a very strong structure and would have been easiest to just use for the whole assembly but it might have ended up looking quite boring and industrial to a customer. So we went for a colorful tubing style in all the modules. But then to find and connect these different family systems was also a challenge.
So I think it’s an interesting balance of not having to overthink everything to the maximum, but also a challenge to make something chaotic still look cohesive.
Can you talk me through some of the other key challenges?
CK | I would say definitely the most challenging aspect was capturing this real world object, a shoe, digitally, modifying it and then translating it back into the real world. This is a kind of pipeline that is especially tricky for soft, deformable objects and objects that have wear and tear because it means we can’t work with any standard models. So making the system adapt to basically anything we put into it and making the translation back and forth reliable enough was definitely the biggest challenge.
FB | To kind of continue on that, one of the big challenges from the hardware perspective was to accommodate all the different inputs and variables. While it was always a sneaker, as Christian said, it was in varying states of cleanliness, then we had roughly 21 sizes, the different cuts — high, mid, low — then multiplied by the three different models the robot is designed for, it ends up being over a hundred and eighty possibilities of inputs that can come in. Not to mention the different colors and styles of the shoes and the variety in materials — from leather to fabric or flyknit shoes. This was why we developed adaptable lasts so you could run, say, a high cut Air Jordan One and a low cut Space Hippie 01 shoe on the same system. The same thing applies to the whole programming of the robotic arms. Would the cleaning work for a small shoe as well as for the largest shoe? For robotic arms and systems you want to have a very known input and a very known output to program all these motions, whereas all our motions were completely dependent on the size, cut, material and condition of the shoe.
We also had to make sure it adheres to the CE certification and retail safety standards while being an engaging experience for the customer.
“For robotic arms and systems you want to have a very known input and a very known output, whereas all our motions were completely dependent on the size, cut, material and condition of the shoe.”
There are a lot of different features and components packed into a relatively small footprint. Did you have to pick certain construction mechanisms or new ways of arranging things to fit so much in?
FB | Well, because we knew it had to be installed into a retail environment, the space limitation actually gave us a good starting constraint. We used a very easy and modular structure, the ITEM profile, to make sure that we had a strong base to support the two robotic arms and the other modules, and that already defined our footprint. Then we kind of made our adaptations with another very flexible system from Misumi, where we developed pipe-based frames using a combination of standard and purpose-built connectors. And to potentially accommodate further modules while reinforcing some of the other structures like the last stand [where the lasts are placed and picked up by the robot], we opted for a truss structure on the top.
In essence, what we did was create a simple toolkit of aluminum profiles, steel pipes and connectors, and then used this tool kit to build all the structures in the system.
Were there other safety implications, given that you knew you wouldn’t always be able to monitor who would be interacting with the machine and how?
FB | Yes, for sure. It had a huge impact on how we ensure that the machine is safe to operate because, like, in a factory environment, access is limited to people who are trained to use the machines. In the retail environment, especially a high-traffic area like Oxford Circus in central London, we knew we would not only have adults engaging with it but also kids who are curious and play with and push things around.
So, we went for a multi-layer safety system, which involved firstly casing the machine on three sides with very large polycarbonate transparent sheets, to make sure that you still have the perfect, unobstructed view onto the machine.
We had one side which needed to be open, so the operators could access the machine to load and unload shoes, and for this opening, we integrated a light curtain which, if crossed, triggers the overall safety system and the safety PLC. This, in turn, shuts down the whole machine, so the robots and all other mechanisms stop running to make sure that it’s safe to enter the machine.
But you may also have the situation that somebody’s already inside the machine but for one reason or another, the light curtain isn’t being actively crossed or triggered at that moment. So we added these two laser scanners which shoot a two-dimensional plane through the whole system and compare it with an image of how the machine is supposed to look. If something changes within the machine — for example, a toolbox was left inside the system or a child has snuck in after the light curtain was triggered — there’s no way for an operator to accidentally override the safety system and proceed with a journey.
Finally, we also programmed the stopping of the system in a unique way. When you stop a machine mid-process in a factory, the component might need to be sacrificed and the whole machine needs to be reset to its initial state. The way we programmed our safety system allowed us to go into a ‘safe stop’. So, for example, you can stop the system in the middle of soaping of the shoe, but then resume exactly from that point onwards once the safety concern has been addressed.
CK | When we were here talking about the safety of the machine, one of our colleagues was always saying, like, ‘You need to be prepared for when someone charges at the machine, running’. Like it needs to stop in time. And it seemed like an extreme example but when we were actually operating the machine in Niketown, we had two children and a dog that tried to run into the machine at different times. So in the end that as a safety measure was actually very realistic.
B.I.L.L. was conceived as a robot that aims to bring circularity closer to consumers or communities. How did you try to reflect that?
FB | Making a repair machine that customers can and want to interact with was one aspect, but then there was bringing this whole idea of repair in, in a maybe not traditional way. Like, we weren’t trying to recreate the shoe you bought two years ago to be like new. It was about elevating repair or celebrating something well-worn. Like patching scratches and holes on the upper with colorful patches was already kind of a different approach and a lot of people were actually interested in this concept of, like, using the act of repair as customization. It also gave us, as the in-store operators, a good point to talk to customers about repair.
“We weren’t trying to recreate the shoe you bought two years ago to be like new. It was about elevating repair or celebrating something well-worn.”
Was it a consideration within the system’s engineering as well?
FB | I mean, we tried to always be conscious of the waste created by the system. For example, if you look at the cleaning station, the whole water system is a closed-loop system. Meaning, we filter and reuse the same water up to 8 times to clean the sneakers.
The other aspect — which was especially the kids’ favorite because of how bright and fun it is — was the little wipe-off station, which is made out of shoelaces. We wanted to find ways to reuse surplus materials and managed to kind of create, like, a bit of a rotating car wash brush by threading these bright shoelaces through a reel which then rotates to remove excess foam off the shoes after each cleaning cycle.
It’s one of my favorite parts as well, especially for the conceptual reinforcement, so you understand the goal of circularity right on the machine.
FB | Exactly.
Seeing the potential of a system like this, how would you make it more accessible?
FB | I think, to give customers more access to repair services, it’s a lot about the scalability of the services. I would continue this modular approach, and isolate the stations or functions within the B.I.L.L. system. Through this project, we figured out the motions needed to clean and patch sneakers and these can then be extracted and replicated without needing to be part of a whole retail experience that B.I.L.L. had to be.
There were customers who were only interested in cleaning their sneakers, for example. And you might not think of cleaning as repair, but people had shoes that were so dirty, they felt as though they couldn’t wear them anymore.
Then we had the customers who only wanted to patch their sneakers. We had a lot of people tell us that they would throw away their white Air Force 1s as soon as they got creased. So, if we can add something unique to those practically new sneakers that makes someone keep them for longer, then we’re also extending the life of a sneaker in a meaningful way.
Thank you, Florian and Christian, for talking us through the details of this great project. A big thank you also goes out to Nike’s innovation team for the creative momentum that brought this concept come to life. We look forward to continuing this collaboration and creating more exciting pieces together in the future.
Looking to explore new ways of applying technology for circular experiences and manufacturing? Get in touch!
Interview conducted and edited by Gabriella Seemann and Dev Mishra.
PCH Innovations is a Berlin-based, creative engineering studio for exploratory technology.
