IDES of DEF CON: Badge Post Mortem

The IDES OF DEF CON is/was an independent electronic badge we manufactured for DEF CON 25 at Caesar’s palace. More details about the badge are available at https://dc25spqr.com. This post is mainly about what it cost and what happened during the manufacture of the device.

We just got in about 30 minutes ago from a long drive through the Mojave desert. The entire trip was a blur, but here’s what we can remember. #badgelife. YOLO.

We started this project in August of 2016, just after DEF CON 24 with just $500.

Financial transparency

We raised $30,015 over the course of six months.

We had two non-kickstarter sponsors who graciously offered us larger than normal sponsorships:

• $5,000 from Avast. We gave them 11 badges, one of which ended up on Gary Kasparov’s neck(!), one went to the person who helped us get the deal signed.
• $1,700 from Phobos Group. We gave them 3 badges.

$23,315 came from Kickstarter ($21,448.15 after fees).
No idea on taxes yet, we’ll figure that out soon enough.

This includes sponsors who bought a sponsor package on Kickstarter:
$2,500 from Hacker Warehouse
$2,500 from Urbane Security
$1,000 from Red Team
$1,000 from Grimm
(Thank you to all of our sponsors. We couldn’t have done this without you.)

I feel that we had too many sponsors.
Six was way, way too many.
Next time we limit this to 2–4 at much higher price points. ($5k and above)
We should try to raise at least $75,000 next year and produce 500 badges.

Additional Income while at DEF CON, directly selling to people in our room or in person was over $3,130.

eBay Sales of partially workings or boards in a less than perfect state brought in a final $1,197.50.

We built 225 boards. Build cost per unit was around $113.
Over 34% had one or more failures, and this number was possibly higher than that including boards that had already shipped to people which we reworked at DEF CON. We should of tried to reduce cost further here. COGS was too high. Next time, China?

We spent around $25,852 on manufacturing, and had additional expenses in excess of $6,580.
Most of this was prototyping expenses, equipment expenses, or the team/sponsor dinner. We are exiting this project with a net gain of about $700 and we are up a year worth of experience, a stereo microscope, a hot air station, lots of SMD rework tools like tweezers and probes, many dev boards, and badges for everyone on the team.

After five days of reworking, we managed to reduce the failure rate from 34% down to 9%
I took a full electronics lab to DEF CON, set it up in my hotel room, and people came by for repairs. It was grueling. Of the remaining boards, around 20 of them could not be reworked, we gave away three of them, and the rest have been sold off on eBay. A few scant boards remain in my house.

Things we did right

• Lots of stuff completed on time and on budget. The badge, challenge coin, packaging, and lanyard were great. Great art.
• Bill’s work on the low-level drivers and updater code was nearly perfect and he even patched a bug while at the con, involving updater issues on different compiler version and the updater itself being overwritten. Amazing.
• Joining the #badgelife slack was invaluable. Thanks to the incredible organizing by wdm (Whitney Merrill) we all had access to a great Slack (#badgelife) where we were able to coordinate and learn from each other.
• On Slack we managed to beg for various parts during the con. When Macrofab was forced to send back our batteries on UPS ground (IATA regulations), we bought some from other #badgelife members at B-Sides, making 26 more boards available for sale.
• Quickly learning SMD reworking (Thanks to watching many, many youtube videos from Louis Rossman) — An excellent rework job rescued the project, but cost us $1,000 for a hot air station, $700 for a stereo microscope, and around $100 for hot air tips. I am fairly certain I leveled up on SMD repair. However, never again will I use a cheap Chinese hot air station. The difference was huge — It was like going from a Radio-Shack soldering iron to a temperature controlled Weller.
• When it came time to QA the boards for reflow and debugging, I had the bright idea to print out little QA slips and attach them to every board. The QA checklist was invaluable in both triaging repairs (speed = more sales = more money), selling broken boards, and producing a quality product. Do this again!
• Learned very quickly how to repair boards based on our schematics. It got to the point where we would intuitively know what was wrong with the board within a few seconds. Examples: No display / video noise? Oh, that’s SPI2_SCLK, SPI2_MOSI that need reflow… Radio slow to respond/retransmitting? Reflow VR_PA and RF_DIO. No LEDs? bypass/short and test the chain until you find the last good LED, then replace.
• Pretty much nailed most of the fight code; Minor bugs persist.
• People loved it. Graphics were great.

We also received some excellent press coverage…

• The Parallax: https://www.the-parallax.com/2017/07/25/hackers-love-badges-ides-defcon/
• Engadget: https://www.engadget.com/2017/07/31/def-con-badge-culture/
• Avast: https://blog.avast.com/pirates-at-defcon-avast-listen-to-the-roman-lighthouse-siren
• Avast “inside scoop” interview: https://blog.avast.com/ides-of-def-con-inside-scoop-on-best-unofficial-electronic-badge-def-con-25
• Some sort of aggregation of all of these: https://community.spiceworks.com/topic/2022415-the-ides-of-defcon-an-unofficial-multiplayer-fighting-conference-badge

We had some great art from Matt, who did not attend DEF CON…

Matt’s Character sheet for all of the fighters (nice, huh?)

Things we did wrong

We wrote all of the code for a leaderboard and we never got one online in the HHV.
I was so busy with board repair that the leaderboard agent device and it’s associated raspberry PI didn’t get deployed anywhere at DEF CON. Dammit. Next time when someone offers to help you with something during the con, keep better track of who’s offered and chase them down at the con.

In fact, I never even went to the HHV because a) I feared being mobbed by people who wanted a badge b) Setting up $10k worth of debugging equipment and optics in the HHV and breaking it down every day was going to suck ass c) too loud of a space to work in. d) Probably was going to have to deal with too many n00bs asking questions while we were attempting to get ourselves out of a $15,000 dead-board hole.

For our top Kickstarter tier, we included too many badges.
The top KS tier, for example, included 20 boards. This nullified the effect of being a sponsor, essentially creating a product-purchase relationship and not a sponsorship. We did extra work for sponsors (logos, managing expectations, etc.) who ended up with badges (2500 / 20 badges = a top-tier sponsor actually paid $125 each) I don’t mind that they got badges, but Sponsorships should not be pre-orders. There has to be a good balance here.

We did not keep good track of sales while at DEF CON.
We had so many sales happening in the room that while I had one eyeball on the microscope and one on the money, our count is probably abit off. I had a couple of sheets of paper for tracking but things got hectic, fast. At one point I had a line going out of my hotel room of people wanting badges.

We did not have enough screens to go around and our design used large screens which broke when they came in brutal contact with other badges. There was a good area here to make a profit!

We did not make enough devices to significantly reduce costs.
225 is a bad quantity for electronics — 500 or 1000 is a much better place to be. But, we were scared that we wouldn’t get our money back! More boards, more risk. Additionally, create a waiting list with FULL contact information so you can reach people. Differentiating between hacker-handles and paypal/venmo legal names sucks.

While we were repairing boards onsite, we lost much of our day at DEF CON running around delivering badges.
Next time we should of had a central pick-up point / don’t ship UPS to people (saving money) and assign a team member to deal with repair tracking. Also pick one. Either have people pick up, or ship the damn things. Mixing the two is a recipe for a serious headache.

For some strange reason we also had a number of people that simply didn’t pick up their badges or respond to multiple requests to pick them up. I blame Kickstarter for this. We couldn’t find these people because they didn’t fill out surveys. I fully expect them to come back to haunt me months later looking for badges which we sold to other people because they didn’t respond. There could be a number of refunds in the upcoming months.

It was also difficult to deal with people who had badges lost in the mail, I’ll spend the next week looking for those badges, and may have to issue refunds dragging the losses down further.

Anyway.

Manufacturing Learnings / Issues / Concerns

Poor component selection of the MCU and LEDs ruined many boards and reduced yield.
WS2812B RGB LEDs (aka Neopixel) do not survive reflow, and fall apart at the slightest visit from a hot air station. Had we used something else, I would not of spent days in a room. I WILL NEVER USE THESE AGAIN. I would only consider using them again if they were hand-soldered. No one seems to know what the right range is for removing these devices without issue and the data sheet lacks any sort of temperature guidance.

Using the KW01 chip caused more problems. Time and time again the footprint bit us in the ass, creating dead boards, and lots of time lost in prototyping spins. I WILL NEVER USE THE KW01 AGAIN.

Next year we want to jump on the BLE bandwagon and use the Rigado chip or whatever the other teams had. Standard badge comms will be awesome and allow for many more features.

We were not explicit in our programming instructions to Macrofab, causing boards to be reflowed(!) or rejected as failures even though they were fine.
We had a large number of MCUs stuck in a loop. Had we specified: “If you can’t program the board, telnet into port 4444 and type ‘kinesis mdm mass-erase’ and everything will be okay” our reject rate would have been way lower.

Lithium Battery shipping times, because you are forced to use UPS ground or cargo airlines only, added to our troubles.
If you are using LiPos, be sure to allocate enough time to ship them. IATA regulations force you to use slower shipping options.

Macrofab repeatedly failed to send back all parts requested, costing us time and money. Update: Macrofab refunded $1500 to us because of issues. Thank you Macrofab.
I really like Macrofab. I do. I want to do everything to support them. I think they are the best low-cost fab in the country, but we had problems which were a combination of our inexperience and lack of communication in general. We will fix this in the future. Again and again the words, “Please ship everything you have back to us.” Turned into: “We forgot to ship you these parts because we forgot.” Eventually we were doing 2nd day air from Houston to Las Vegas just so we could pull through.

We are going to work with them again. we are just going to be much, much, better about the chips we use and the way we communicate with them.

There was also one prototype where we asked Macrofab to verify the chip footprint, they said it was OK, but I was off by 22/100’s of a millimeter.
This is entirely my fault, because I made the mistake by misreading the datasheet, but it is also their fault because they should have told me it was wrong.

Our inexperience at building boards at scale meant that we did not build in and/or utilize proper DFM (design for manufacturing) techniques.
We should have written more tests into the board. I had to add a last minute routine to do button testing into the board’s LED test code. That code was extremely valuable and I will do it again.

Putting the JTAG connector under the screen was absolutely stupid. Don’t do that again.
Having to remove the screen just to reflash a board was the most irritating thing, ever.

Conclusions

Are you doing this next year?
Most likely, yes. We still have lots of great ideas and now have better equipment to work with. We also know what -not- to do which is more valuable than anything else.

It was an extremely rewarding experience to walk around and see these devices hanging off people’s necks running many copies of our code. It would have been better had we made about 2,000 more of them so we will attempt to make many more next year, shooting for a $40–60 price point instead of $120.

I still believe that this project was a success for our team and our sponsors. In the end we didn’t lose any money (well, we did if we count the cost of getting to and from DEF CON, rooms, and our time, but who’s counting.)