wtfBeacon (how Shortwave works)
Earlier this year, we (myself, Matthew Kulp, Ethan Sherr, and Alex Whittemore) built a little app called Shortwave that allowed you to chat anonymously with other people within 100 feet. We had a few reasons:
- It seemed weird that we’re all walking around with super-fast network-connected devices with ambient sensors (we used to call these “phones”), yet said devices are only dimly aware of one another.
- Nearly everything nowadays has a signup flow and a contact list — we wanted to play with a social model that didn’t.
- Hacking our way around the limitations of nearby networking on iOS seemed like great fun.
We launched Shortwave back in March and had a blast watching it grow internationally — there’s probably a whole blog post to be written on what we learned about anonymity and physical/digital social dynamics. I also started getting emails (and comments) from people building similar apps that were curious about how, exactly, Shortwave worked — and how they could build something similar. So that’s what we’re going to go over here — the limitations of the CoreBluetooth and iBeacon APIs on iOS, and how to partially overcome them. This isn’t in any way exploitative, and there’s no cheat code to magically lift the restrictions Apple has placed on these APIs, but through a clever combination of design and engineering, you can build a pretty magical experience.
To do this, though, you’ll need more than an iBeacon. You’ll need something we came to affectionately call the wtfBeacon.
There are a lot of small details in this approach, and some of them are kind of easy to miss. I’m mostly writing this to keep track of them myself, but in case it’s useful to anyone else I’ve done my best to explain what we built. Feel free to call me out if anything seems wrong or unclear.
Also, here’s a gist with some example code. It’s mostly meant as a guide, and as this was the first Objective-C project I’ve done it’s probably really, really bad code.
We want two users to be able to spontaneously start chatting, without either opening the app. If one user is sitting in a coffeeshop, and another walks in, we want both of them to be notified that the other is there. Even if only one user opens the app, we want them to see that there is another user in range, and any messages they send should be delivered to the other user. To reiterate — we do not want to require both users to have the app open and in the foreground to be able to see one another.
Multipeer is a mesh networking API that uses infrastructure wifi (a wifi network both devices are connected to), ad-hoc wifi (a wifi network broadcast from one device to another), and bluetooth to detect nearby devices (this is what Firechat uses). Most importantly, it can work across these modes, enabling a device (with only bluetooth enabled) to communicate with a device (with only wifi enabled) through a device that has both enabled. Unfortunately, there are some pretty serious limitations to the Multipeer Connectivity framework (although I haven’t looked at it on iOS8, perhaps it’s improved?) — basically, you couldn’t establish connections unless both devices were in the foreground. For us, this meant that both users would have to open the app in order to even see the presence of a nearby user — we didn’t like that.
So we decided to instead use Bluetooth Low Energy to associate nearby devices. The range was a little less than Multipeer, but the support for background networking is much better. There’s been quite a bit of hype around iBeacon lately (although it seems mostly to be a marketer’s fantasy at this point, and we’ve seen very few cool consumer applications), and this was the first component of the core of Shortwave.
Skip this if you know how iBeacon works
Bluetooth Low Energy devices send out small packets of data to alert other nearby devices of their presence. Usually, these packets contain data about the advertising interval, the calibrated power level, available services, etc. However, there is also a Manufacturer Data field type (0xFF) that you can use to broadcast 31 bytes of “generic” data — anything you want. This is cool because you can transmit, say, your battery level to every BLE-compatible device in range, without connecting to anybody. You’re already broadcasting packets to indicate your presence to nearby devices, so why not piggyback a little extra metadata on those packets?
Side note — when I was building low-cost ambient sensors at Robin, we used the ManufacturerData payload to transmit small amounts of sensor data (temperature, light, acceleration, etc) that could be picked up by any BLE-compatible device in the area — in our case, a raspberry pi with a BLE usb dongle
iBeacon is just a specification (created by Apple) that says “if you arrange these 31 bytes in a specific way, iOS devices will recognize this packet, and wake your app up to process it”. There’s a great StackOverflow answer that describes the structure, which looks like this:
Basically, you have three identifiers that you can play with — a uuid, a major, and a minor. UUID identifies your app/product uniquely, and major/minor is unique per beacon. For example, if you were deploying iBeacons in grocery stores, you might use the same UUID for all beacons, change the major value for each store, and change the minor value for each beacon in each store.
The Beacon API is part of CoreLocation, and it’s pretty simple — you can tell your app to broadcast as an iBeacon, and to listen for iBeacons. To broadcast as an iBeacon, you just provide a uuid, major, and minor — to listen for iBeacons, you provide a uuid and optionally a major/minor (if you want CoreLocation to “filter” the beacons received based on major and minor) — as well as a callback to be run when a beacon is discovered. There are plenty of good tutorials out there on this.
There’s an important abstraction that comes from the fact that iBeacon is a part of CoreLocation, not CoreBluetooth. In iBeacon-land — you talk mostly in terms of “regions”, and less in terms of “devices”. A region is defined by the combination of uuid, major, and minor described above — a region can refer to a uuid+major+minor, a uuid+major+(any minor), or a uuid+(any major)+(any minor). Any iOS device listening for this region has a state which is either “inside” or “outside” .
In practice, you’ll define callbacks that will be run when the state of a region you’re monitoring changes (when you enter or exit it). If your app is in the background, it will get about 5 seconds to do almost anything (no UI, of course) before being killed again. Even if your app has been killed, or the device has been restarted and the user hasn’t launched your app, it will be launched and your callback will be run.
Generally, you’ll want to spend these 5 seconds ranging beacons. Ranging in a region gives you all of the actual devices that are in that region, each with UUID, major, minor, and rssi (signal strength, power-law to distance). It’s important to note that unless you specify major/minor when you create a region to monitor (which limits that region to only that beacon — not what we wanted), you won’t know the major/minor when you enter the region. You’ll need to range to get these values.
On a high level, that’s how iBeacon works. It’s relatively straightforward to build a system for iOS devices to communicate unique identifiers to other iOS devices via iBeacon. Make up a common UUID, have each device broadcast a unique-per-device major or minor “userId”, and have all devices listen for all beacons. When you enter the region defined (only) by your UUID, start ranging beacons to get each beacon’s major and minor.
How this works in Shortwave
Let’s put this in the context of Shortwave. We’ve got a single, global UUID that every beacon broadcasts on. Every user broadcasts a unique combination of major and minor. We just concatenate major (2 bytes) and minor (2 bytes) and treat the combination as a single 4 byte integer — we’ll call this our userID.
When you open Shortwave, you immediately begin broadcasting as an iBeacon. Any other Shortwave devices in range pick up on this, waking or launching the app if necessary (if the app is in the background, then the 5-second time limit starts here). Each device then begins ranging beacons. Anytime a new beacon is discovered, the receiver propagates the id pair (it’s own ID, as well as the ID of the beacon it just discovered) back to the broadcaster (we use Firebase, because it’s awesome). The broadcaster is now “linked” with the receiver — any messages the broadcaster sends will be propagated to the receiver (and vice versa). “Old” users-in-range are pruned after about 30 seconds after the last sighting.
This is a good start, but there are a couple of problems.
Problem 1: Discovery
I started the previous section with “when you open Shortwave…”, but how do we actually get people to this point? If two Shortwave users walk into a bar (ha!), how would either of them know to open the app? iBeacon can’t broadcast in the background, so that’s out. Can CoreBluetooth broadcast in the background?
Solution 1: CoreBluetooth
The answer is: “sort of, and it’s good enough”. When your app enters the background, the bluetooth stack in iOS strips out quite a bit of your payload — what’s left isn’t a fully-formed iBeacon-compatible packet, so it won’t be picked up as an iBeacon. Bummer.
Although we can’t use CoreBluetooth to transmit an iBeacon-compatible packet in the background, it can still be useful. Users don’t actually need to know who is nearby until Shortwave is in the foreground — it’s enough to know simply that “a shortwave user” is nearby.
Implementing this with CoreBluetooth is pretty simple. You define a service UUID (ours is the same as our iBeacon UUID), and start advertising. You also start listening for other devices with that service UUID, and register a callback to be run when one of these devices is discovered. Like the iBeacon region entry callbacks, these will be run even if your app is in the background.
When a new device is discovered, Shortwave is woken in the background and we send the user a local push notification alerting them that there is a Shortwave user nearby. If they act on this push and open Shortwave, they begin broadcasting as an iBeacon, and the process described above links them to the other device.
Because you share the same peripheralManager between iBeacon transmissions and regular CoreBluetooth transmissions, you need to continuously flip between broadcasting using iBeacon and CoreBluetooth. On Shortwave, we used an interval of 1 second. When Shortwave goes into the background, it stops flipping between these states (no broadcasting as an iBeacon in the background) and simply broadcasts at a low interval using CoreBluetooth.
Problem 2: Region Collisions
There’s an issue with the above iBeacon method that can cause some receivers to fail to link with broadcasters, and it has to do with the way regions are defined. To illustrate this, let’s imagine that Matt, Ethan, and Alonso are all in the same space. Let’s imagine that Matt brings Shortwave into the foreground, and begins broadcasting as an iBeacon. Here’s what happens:
- Matt emits an iBeacon packet
- Alonso’s region entry callback is run, he ranges iBeacons nearby, grabs Matt’s userId, and links himself with Matt
- Ethan’s region entry callback is run, he ranges iBeacons nearby, grabs Matt’s userId, and links himself with Matt
Now let’s imagine that Ethan brings Shortwave into the foreground. Here’s what you would expect to happen:
- Ethan emits an iBeacon packet
- Alonso’s region entry callback is run, he ranges iBeacons nearby, grabs Ethan’s userId, and links himself with Ethan
- Matt’s region entry callback is run, he ranges iBeacons nearby, grabs Ethan’s userId, and links himself with Ethan
However, in this example Alonso fails to link with Ethan, while Matt links with Ethan properly. Why?
The “region” that receivers are entering is only defined by a single UUID, which means that there’s only one region that any Shortwave user can be “inside” or “outside” at any given time. Alonso is already inside this region, which means that his region entry callback won’t be run again. No region entry callback means no ranging, no ranging means no major/minor, no major/minor means no userID, which means no linking.
This might seem like an extreme example, but it actually happens quite frequently when you have multiple devices in a single space. No matter, I’m sure we can build something epically ridiculous to address this.
Solution 2: Wakeup Regions
The root problem is that we can’t be woken up on entry for a region that we’re already inside. The solution is to introduce more regions that only serve as wakeup regions. They don’t transmit identifiers, they only exist to fire other devices’ region entry callbacks.
iOS imposes a limit of 20 beacon regions at a time. We refer to the beacon region detailed above (the one defined only by UUID) as our Identity Region, and we created 19 other regions to serve as Wakeup Regions. Every Shortwave device registers region entry callbacks for both the single Identity region, and the 19 Wakeup Regions. No matter what region actually calls the region entry callback, ranging only ever happens on the identity region. It’s common for an app to be woken up by, say, Wakeup Region 15, and then immediately start ranging on the Identity Region to get the userId of that device.
We’re now flipping between three broadcast states while our app is in the foreground:
- Broadcasting our userId on the iBeacon Identity Region
- Broadcasting anonymously with CoreBluetooth to handle both-device-backgrounded wakeups (CoreBluetooth)
- Broadcasting on an iBeacon wakeup region to ensure region entry
Again, when our app enters the background, we kill the iBeacon broadcasts and use CoreBluetooth only.
While not strictly necessary, it’s a good idea to keep track of what wakeup regions you’re currently in (caused by other devices broadcasting on them), and what wakeup region you’re currently broadcasting on. This will make sure you avoid “interference” from another device that might be broadcasting on the same wakeup region.
Bringing it all together
With the above solutions, two users can walk into a bar, both with their phones locked and in-pocket. They’ll both get a notification that there is a Shortwave user nearby. As soon as one of them opens the app, they’ll be linked with all of the users around them, and any messages will be magically delivered within 100 feet. Pretty sweet, I think.
I hope this becomes obsolete
I get why Apple doesn’t allow us to broadcast in the background — even with user consent, the power drain can be enormous. That said, I think we need an API for extremely limited background iBeacon broadcasting. Something on the order of 1/minute would be absolutely fine, and would open up all kinds of cool applications while keeping power consumption low.
While we’re on the topic — forming a connection with a peer using the Multipeer framework while that peer is in the background would be awesome. If anyone has been able to do this, let me know!
@alonsoholmes on twitter, hello at alonso.io on emails.
Also, our new-ish project is called Hashtag — it’s like a global, public Slack (or, you know, IRC), and it’s in super-super early beta. I’ll be hanging out in #wtf-beacon — come say hi!
Also, a couple of people have expressed interest in an easy-to-use framework built on this. If this sounds like something you’d find useful, let me know!
I sometimes consult on projects like this. If you’re looking to build something that does batshit crazy stuff with iBeacons, we might be able to build something cool together. firstname.lastname@example.org
 — Although this state is briefly “undefined” while your device’s bluetooth stack is booting, it changes to “inside” or “outside” shortly after
 — Your app delegate needs to conform to the BeaconManager protocol, in order for your app to be woken up in this way.
 — They won’t boot your app if it’s been killed, but the iBeacon callbacks you set up earlier will take care of that.