Filtering LoRaWAN traffic on gateway level
If you have ever deployed or are planning on deploying your own LoRaWAN network, you have likely had to decide how to get data from your LoRaWAN gateways to the Internet, where your application runs.
A typical PoC (proof of concept) will run in a fairly controlled environment, where gateways have wired backhaul, data volume is virtually unlimited, and is negligible in the PoC budget.
Once you turn your PoC into a PoV (proof of value) or start deploying a production network, sooner or later you will run into a situation where you have to pay per volume of data on the gateway backhaul. At that point, you will start figuring out ways to reduce the backhaul usage. That is where filtering of LoRaWAN traffic might cross your mind — and rightfully so.
Real-world experience with LoRaWAN filtering
Some of Avoguard customers are deploying LoRa gateways in areas without wired internet access, and sometimes even without access to electricity. Especially in agricultural applications, this is more common than not.
To address this problem, their gateways are communicating to Internet over
- 3G
- 4G
- satellite
- point-to-point radio
backhaul. This allows for great deployment flexibility, but comes at a cost. Every byte sent or received over these links has a price. We have seen a lot questions coming up about these costs, all based on inaccurate assumptions and expectations of low gateway backhaul usage, stemming from the low bandwidth nature of LoRaWAN devices.
In this article, I will provide answers to these common questions, and outline how you can avoid large cellular data consumption bills by edge computing — by filtering traffic on your LoRaWAN gateways. I will also detail some of the caveats you should be aware of before selecting your filtering method, and recommend specific cases in which to use one or the other approach.
Basic filtering
A basic level of filtering is to filter only valid LoRa frames. These are frames that are consistent on the physical layer, and have a valid header and CRC. Most gateways and configurations do it by default, in hardware.
Filtering valid (authorized) traffic on higher levels would require the gateway to store per-device security information, which would compromise network security, and significantly increase backhaul usage due to the need of sharing the security information.
A naive approach
A naive approach is to ask to simply filter only “useful” traffic at the gateway level, so that only traffic address to the specific application gets communicated over the backhaul. This sounds like a valid, simple request. However, due to the data-saving features of the LoRaWAN protocol, roaming and device mobility, it is not practically achievable. To filter LoRaWAN traffic with 100% accuracy, with no false negatives, and no false positives, you will have to make some compromises.
Special case:
DevAddr filtering for stationary devices and single gateway, no roaming
This special case applies only to networks that meet all the following criteria:
- your devices are stationary
- there is only one gateway in their range
- you do not want the gateway to forward roaming traffic
If your networks meets all of these, a simple and fairly accurate filtering is possible. A gateway can be instructed to only forward traffic for white-listed DevAddrs (see explanation of what DevAddr is in Filtering by DevAddr).
Downside — scalability: This solution does not scale with number of gateway or devices.
Suitable for: Isolated individual gateways
Unsuitable for: Anything else than isolated individual gateways
If you are looking to filter traffic in any other case, you will need to make compromises. Furthermore, you need to be aware of the side effects the filtering will have. The following are some of the methods that you should consider for larger scale networks.
Filtering by DevEUI
A typical modern LoRaWAN device is assigned an identifier called DevEUI. A DevEUI is a globally unique identifier from an IEEE identifier range. The global uniqueness is guaranteed through a centralized assignment of the ranges by IEEE. Naturally, because DevEUI is unique per device, you would want to filter traffic by this value-list DevEUIs that are allowed on your network. The challenge is that a DevEUI is not part of regular data traffic that you want to filter. It is only communicated during the OTAA procedure, and there’s no simple way a gateway can on its own correlate the regular traffic to the OTAA events and the DevEUI.
The only way to compute a DevEUI from an uplink packet is to perform a lookup using the device’s current session information.
Downside — security: You are going to be distributing sensitive security information to the gateways. You need to have trust in the gateway’s security.
Downside — traffic overhead for mobile devices and multi-gateway networks: You need to continuously update session information on all gateways that are currently in the proximity of the device.
Suitable for: small private networks, use-cases with low / no security requirements
Unsuitable for: large public networks, highly secure networks
Filtering by DevAddr
In order to reduce overhead and increase security, LoRaWAN devices do not use the DevEUI when sending or receiving data. Instead, they use a DevAddr — a shorter, 32-bit address that is assigned to every individual device, but is not guaranteed to be unique. In fact, with the latest changes in NetID allocations, chances are smaller networks will have to reuse DevAddr values. For example, companies with Adopter status in the LoRa Alliance get allocated Type 6 NetID, and have address space for at most 1024 unique DevAddr values.
If DevAddr values are reused, there is no way to tell a gateway to filter data only for one device just based on the DevAddr. The gateway needs to know the current session information, which makes this a case identical to Filtering by DevEUI.
Filtering by NwkID
Up till now, we considered a DevAddr is a value without structure. However, a properly constructed DevAddr does have a structure. The 32 bits of a DevAddr is structured as follows:
- NetID Type prefix (1 to 8 bits long), always ending with a zero bit
- NwkID (6 to 17 bits long, depending on NetID Type)
- NwkAddr (7 to 25 bits long, depending on NetID type)
Let’s explore what these values mean and how they are assigned.
NetID Type prefix is linked to the NetID type you have been assigned by the LoRa Alliance. The less bits it has, the more space you have for NwkID and NwkAddr.
NwkID is a value that for Type 0, Type 1 and Type 2 networks directly maps to a NetID. For other NetID types, the mapping of NwkID to NetID is ambiguous.
NwkAddr is the individual address that you as a LoRaWAN network operator can assign to your network’s devices. For optimal processing and best possible filtering results, your NwkAddr address space should have a uniform distribution. In other words, you should avoid duplicates for as long as you can. Once you start having duplicates, you should have a uniform distribution of the duplicates.
As you see, in order to filter by NwkID, you need to start by generating DevAddr that are properly constructed based on your NetID.
If you are owner of a Type 0, 1, or 2 NetID, and assuming you properly used your NetID to allocate DevAddr values to your devices, you can filter traffic on your gateways by the NetID. Also, your address space is fairly large, you don’t need to be too concerned about duplicates unless you are running a very large network. For example, a Type 0 NetID has over 30 million unique DevAddr available.
If you are owner of Type 3 NetID or higher, the mapping of NwkID to NetID is not unambiguous, and a single NwkID could map to multiple NetIDs. Therefore, filtering by NwkID can have false negatives — records that should be filtered, but are not.
Downside — legacy ABP devices: Historically, some early ABP devices came out with a preconfigured DevAddr, that you cannot change. If you migrated ABP devices from another network, chances are they kept their previous DevAddr. This will also be the case for migrating OTAA devices that would need to re-join to your network after the migration. If you need to run such devices on your network, NetID filtering will have false negatives, and will drop traffic from these devices.
Downside — roaming: If you plan on using your gateways to communicate with roaming devices, NetID rules from above will need to apply to all roaming devices, and, your gateways will need to keep an up-to-date list of NetIDs that are currently allowed to roam. Furthermore, not all devices with a given roaming NetID will actually be using roaming / accepting roaming charges. So in order to perfectly filter roaming traffic, your gateways would need to have an up-to-date list of all currently active roaming sessions for a given NetID.
Future downside — Type 3 and higher NetIDs: For NetIDs of Type 3 and higher, your NwkID might not be unique, but could be the same as other NetID. This is not the case with the current allocation list, and is unlikely to be the case any time soon, so you don’t need to be concerned about it at this point.
Suitable for: networks with proper NetID usage, networks with Type 0, 1 or 2 NetID
Unsuitable for: networks with inconsistent / unstructured NetID / DevAddr usage
Filtering by NetID
A NetID itself has a structure of its own. It is structured as follows:
- NetID type (3 bits)
- RFU bits (15, 12 or 0 bits reserved for future use)
- ID (6, 9 or 21 bits)
To filter by NetID, the NetID type and RFU bits will not really help you, you need to extract the ID part of the NetID, and combine it with NetID type.
The limitation here is a NetID is not communicated in full in regular LoRaWAN uplink / downlink messages. You need to compute the NetID from the NwkID, which is part of a properly constructed DevAddr. This method is therefore identical to Filtering by NwkID.
Advanced filtering for networks without Type 0, 1 or 2 NetID
If you don’t have your own NetID, chances are your LoRaWAN network server vendor has one. You could ask your network server vendor to subdivide the DevAddr space it owns. Type 0, 1 or 2 NetIDs provide enough space for reasonable size of subdivided networks. Filtering then can be done based on the proprietary subdivision scheme.
However, if you plan on using roaming in the future, you should seriously consider getting your own NetID.
Downside: any traffic coming from a 3rd party network via roaming will need to go through your NetID provider’s roaming hub, which will deliver the roaming messages to the associated subdivision networks.
Suitable for: networks without their own NetID, or with limited DevAddr space
Not suitable for: networks that want to use roaming without network server vendor’s hub
Advanced filtering with structured NwkAddr
In advanced use cases, where none of the above methods are efficient or applicable, you can use the DevAddr or NwkAddr with any other structure or formatting, or / and use additional function to construct and classify the DevAddr / NwkAddr in addition to what the LoRaWAN standard specifies. You can contact me if you would like to have some suggestions regarding design of such functions.
Suitable for: when none of the other filtering options provide good enough results, or you need one more level of filtering
Machine learning based filtering
It wouldn’t be a proper IoT post if I didn’t speak about machine learning (artificial intelligence). Of course, with the aid of machine learning, one could extend the filtering beyond just the deterministic analysis of the structure of the data frames.
Using additional information about radio signal properties, and by tracking time-wise behavior of devices, you could build a filtering mechanism that learns the typical behavior of a device / device type, and filters traffic according to more complex rules.
Caveat: Monitoring gateway traffic when filtering
For large network, consider the value of analyzing the full radio environment of your network. With filtering, you will be limiting the data that you are getting into your cloud. Deciding to throw away that information limits the insight into your network operation, and limits your ability to predict future problems, and diagnose existing problems.
This information does not need to be collected in the granularity of individual uplinks / downlinks, but can be collected as statistical information about the radio environment, such as number of uplinks and downlinks on given frequencies, or amount of non-LoRaWAN / non-LoRa traffic, and traffic in foreign LoRaWAN networks.
Conclusion on LoRaWAN traffic filtering
LoRaWAN networks allow for number of very robust methods of traffic filtering and saving of backhaul traffic usage, and reducing your data bill. NetID / NwkID filtering is the intended filtering method, but there are other methods that can yield better results in specific scenarios. There are also additional, proprietary methods that go beyond the LoRaWAN specification.
If you need help choosing the right filtering method, please do reach out to me, or leave a comment.
