Logistics and improvisation in hardware creation
Manufacturing a new product is nothing more than taking the process one step at a time. Hopefully each step isn’t a stumble.
TL;DR: The badges arrived on time and the conference went great! The path to production was much more difficult.
Hackaday held a conference in Belgrade on April 9th, 2016. The event was first conceived at the end of 2015 and I was in the right place at the right time. I had the opportunity to pitch my suggestion for a badge design. With a short 4 months until the start of the conference, I got a green light to start production right away. I was given a moderate price target but strict budget limit. I also had wide-ranging design freedom. When I have creative freedom, I feel the rest is simply engineering constraints.
My lack of experience in high volume production was my first source of mistakes. My assumptions about human nature were my second. Let’s take a look at what went wrong.
December
Plans on a screen
The basic idea was simple: 8×16 LED matrix and a few buttons. It had to be hacking-friendly as well, so I added a USB connector to make it ready for the bootloader firmware. Two AAA batteries would provide sufficient energy for many hours of operation.
Kingbright makes beautiful and efficient 8×8 LED matrix blocks, so I chose these for the display screen. Unfortunately, the price tag of these light-up components consumed half of the budget. It was the low price of the remaining components on the BOM that saved the day.
With no time to waste, I made the PCB project and ordered the prototype printed circuit boards. A three week delivery timeline (plus the Christmas and New Year holidays) meant I had my first functional model in mid January. Here it is, without LED matrix blocks:
January
Modification
Seeing a model on screen gives a rough idea of how the board will feel and work. Actually holding the device in your hands is an entirely different experience.
As soon as I saw my prototype, I knew I had to make certain modifications. The three small buttons became four large buttons, which allowed a “joystick mode”. I also needed an extra RESET button next to the ON/SLEEP button at the top. I extended the lower part of the PCB to make it more comfortable to hold while pressing the buttons. Finally, I added an infrared transmitter and a receiver to allow communication between boards.
I moved onto the next PCB project while Alek (Aleksandar Bradic) designed a nice conference logo. During that time, Hackaday sold early bird tickets providing us with the visitor estimate. We needed a total of 250 badges, which informed part purchasing. It turned that even this estimate was low. We had to produce 65 units extra after the fact. That’s a story of its own.
Suspicious, but not worried
Now I could order the components and start organizing the pick-and-place assembly. I have limited experience in high volume production, but luckily I found some help. I found an assembly company close to me in Belgrade, Serbia. They often manage PCB outsourcing to China.
Our first interaction brought unwelcome news: we were in the middle of Chinese New Year. This would last until the February 17th (2016), so we had to wait for a whole month to get boards. The assembler could receive prototyping PCBs before the holidays, but all other items had to wait.
The conference deadline of April 9th still seemed far away, so I wasn’t nervous about the pace of production yet. By the end of January I had the second, and hopefully the final prototype:
This board worked fine and I got good feedback from Hackaday crew. The next step was ordering components. Everything seemed promising, except for the lack of one item: Kingbright’s LED matrix blocks. I could get about one quarter of the required quantity. The rest would be available in nine weeks from the factory. That was the reply I received…eight weeks before the conference.
February
Lead time issues
The lead times were disappointing and pointed to a particular reality. I couldn’t count on the timing of Chinese pick-and-place assembly. Between the production and testing time and the part delivery times, we’d never make it. By the time the parts were received, they would start too late to hit our delivery time. Not only that, I wouldn’t have the control over the production, which was on the other side of the world. If anything went wrong — even a day of delay — I’d be doomed. I’d have 250 badges a few days too late and an equal number of disappointed conference visitors.
There were two problems I had to solve as soon as possible. The LED matrix blocks were the priority. I asked Chris Gammell for help, and from that moment on we kept in constant communication. Each day we checked distributor stock throughout the entire planet. We found a grand total of 124 display blocks.
That’s enough for 62 badges.
I keep reminding myself that we’re not in panic mode yet. That didn’t last long.
Panic drives solutions
The epiphany came a day later. All the displays we were looking for were the “common anode” variety. If we expanded our search to common cathode types as well, perhaps we could find more? This required re-routing the PCB, which also means multiple versions of the board. In turn, that could also require software that adapts for different displays.
Scouring the planet
Our broadened search was much better: 320 blocks! That’s an additional 160 badges. So including two types of PCBs, we can cover 222 badges. Since we were set on using the blocks, we needed to figure out how else we could expand the search. When we included green display blocks, instead of only red ones, we found 76 more green blocks. That’s enough for 38 badges. So we’d have non-uniform blinking badge colors.
We figured if someone asks why some of them were green we would respond, “Those are for the special visitors”
Adding up our score
62+160+38=260 badges (520 total blocks)
That’s how many we could get from four different distributors (Farnell, Mouser, SOS and DigiKey). After our orders, there were no TA(C)15–11SRW(G)A LED display blocks available in the whole world! At least not available in less than 9 weeks.
There were other “options”
Above is the shorter, simplified version of this story. There were also several less known distributors which offered odd numbered quantities (ie. 343 available blocks). This was not the problem. The problem was the high price tag and the fact that the (odd) number matched a reputable source (ie. distributor X, distributor Y and Mouser Europe all offered 343 blocks). Suspicious, to say the least. These were likely brokers who were not submitting reliable info to the part feeds.
We verified our suspicions when we finally ordered the whole stock from Mouser. All similar stock listings disappeared. How mysterious!
Dealing with badge variations
While part brokers are an article unto themselves, we could now produce 260 badges! We now had 3 different “versions” of the badge:
- Common cathode with red blocks
- Common cathode with green blocks
- Common anode with red blocks
The secondary board that works with a common cathode display blocks was next. This also had a provision for auto-detecting the type of display. We could stick with one common version of firmware, which simplified programming. This also made badge hacking a more accessible activity. We wanted to encourage all conference attendees to take part.
The PCB modifications came at the last minute for the common cathode displays, as the PCBs for common anode ones were already ordered.
Here are two prototypes:
PCBs
Because of the various designs we implemented, we had to make urgent orders for PCBs. For some reason, the outsourcing company procrastinated. They told us, “No need to rush, there’s plenty of time!” Also, “The Chinese timezone is six hours ahead, they have finished their work for today. So it’s also OK if we order it tomorrow”.
By that logic, if we order it tomorrow, wouldn’t they get it the day after tomorrow?
Even before we found out about the holiday (February 17th), we were doomed. We had doubled our prototyping PCBs because of the sourcing issues on the displays. We needed to test both versions and then order two different series of PCBs (and manage the differences through assembly).
Again, time was our enemy. We were already ten days past the scheduled start. It wasn’t until the last day of February when the company finally accepted our order. By that time, we needed to do a “blind” procedure: manufacturing without testing. Scary.
So how did our gamble turn out?
We were lucky…this time. Fortune favors the brave, but not universally.
It doesn’t mind the brave wasting their nerves, mind you. But anyone involved in electronics manufacturing understands those are a commodity anyway.
March
SMD assembly
Since pick-and-place assembly in China was out of question, I had to find a solution here in Belgrade. The remaining time had simply run too short.
After several contacts, I found out that most local companies rely on outsourcing orders to China. Very few actually do assembly here in Belgrade. Moving the order to Chinese assembly would take too much time.
I found one company which does the assembly in Belgrade. Moreover, I figured out that the owner of that company was an old friend of mine! It looked like I was in luck again. But when I tried to contact him, I hit a wall. Nobody let me to contact him directly, so I suspected something unusual was at hand in the company. “Some sort of reorganization” was the blurred answer from his workers. “Try contacting us next month”.
Unshaken, I kept trying local vendors. At the last moment, a young company named SLT appeared out of nowhere. Can they do it? Yes, sure. Such a small number of units? Of course. Urgent delivery? No problem. I couldn’t believe my ears. Where’s the catch? No catch.
Miracles happen, but should never be planned for. They did assembly without issue and the project was saved at the last moment again. Here is the delivery, on schedule, ready for manual soldering of battery holders and MCU programming.
April
Final check and programming
From the beginning, I insisted on having one spare week before the conference. Buffer weeks pay off, every time!
Since the badge had to be hacking-friendly, its firmware had two parts. The first consisted of the bootloader and kernel (to those familiar with PCs, this is like a BIOS). The second part of the firmware was the demo program. It was a moving message display, Tetris game and an accelerometer demo.
The first part was programmed by the external programmer (PICKIT 3 in this case). Once the bootloader is in place, the rest programs via the USB port. This makes it more friendly for loading user-generated images later.
The USB port is where another problem appeared. In about 15% of the units the Micro-B USB connector was poorly soldered. It was SMD type, so it was “floating” during pick-and place assembly. The part ended up misaligned after reflow, which is heating the board to the point of solderpaste melting into solder joints. After review and discussion, SLT accepted that they were responsible for repairing those units. Like the other steps in the process, this took more time and further ate into our buffer week.
We later learned these USB connectors are problematic more generally. Even after swapping to an USB connector with through-hole alignment pins, we encountered issues. The next badge, made for Hackaday Supercon in Pasadena, had similar misalignment and bridging. A different (US based) company did the pick-and-place assembly, yet the problem reappeared. Obviously, these types of parts need extra care.
260 times the wrong part
When I tested the first sample, I found another problem: the infrared transmitter didn’t work. After a thorough investigation, I found the reason. I was careless and failed to double-check before ordering the parts.
The supplier did not have enough BAT43 Schottky diodes, so I ordered BAS16. I thought that there was no need to test it on the prototype. After all, “it’s just a dumb diode, what could go wrong?”
It turns out the diode was not dumb. In retrospect, not checking the diode sure felt dumb.
BAS16 is not Schottky, but a standard silicon diode! In this case, it was used to modulate TXD signal with a 40KHz carrier signal, made with MCU’s CCP2 peripheral. Check out the schematic available on the project page. Viewing the diode (D3) and resistor (R5), you will see why it has to be Schottky type. It modulates the transistor (T10), which then drives the infrared LED (D1).
Finding other Schottky diodes was not a problem, but replacing 260 items was. Again, time was our enemy and this was quite a time consuming process. Everything turned out OK, again thanks to the spare week.
Delivered
All badges were ready two days before the conference. Programmed, tested, batteries loaded — and an entire day left to spare! What more could one ask after such a stressful production?
I discussed more about the internals of the badge and the build process at my talk during the Hackaday Belgrade conference. View that here:
Looking to read more like this? Subscribe to the Supplyframe Hardware mailing list for more articles and news, straight to your inbox.
