Abstract
We developped a low power IoT Edge system using RaspberryPi PICO-W(rp2040) + W25N01 NAND flash memory to store sensor data with unexpected power loss.
The IoT edge that terminates an ANT+ sensor attached to a vehicle(e.g., bicyle) that is typically not equipped with a battery, transmits data to wireless network, and stores the data into Local Storage.
The mostly development requirements for this IoT Edge system are as follows.
- Unexpected power loss, microcontroller system specific issues.
- Local storage NAND flash memory rewritable count issues.
+ No MBR space required.
+ No garbage collection required. - Application should start immediately after power on.(3–5 sec)
- Compress 100 Hz data from sensors.
This article introduces a Telemetry System that does not use MBR of file-system , and Garbage collection, does not require mount time, and is highly resistant to unexpected power loss.
Considering vibration, dust, and unexpected power failure, NAND placed on an extension board was adopted as Local Storage.
Each file system has Pros and Cons, in this development, target board was PICO-W , so the range of options was limited to begin with, and Access Control, Security, etc for an ordinaly file system were not necessary, so optimized Read/Write Process was developed from scratch.(Almost No-FilesySytem)
How to Fast Mount/Boot
Log-structured File System requires meta-information for each piece of sensor-data to be stored, so all blocks and pages must be read at boot-Processing time, which means that the boot process is time-consuming and costly.
This Architecture accelerated boot(disk mount) solution takes advantage of asymmetric use cases, while the write use case requires a fast boot(i.e., latency constraints are high, such as need to start writing immediately after power-on), the read use case is under a stable power supply and has almost no latency constraints.
- Payload header per Layer
- Multiple layers are compressed and stored on 1 page and flushed 1 page at a time.
- Means that the boot process reads only the first 64 bits of the page and checks for the exists of data.
Characteristic
In this implementation “Unprocessed:0xFFFFFFFF” and “Written:0xDEADC0DE” statuses are set in each page header to facilitate management of sensor data written to the NAND flash memory.
The “Unprocessed” status indicates that ERASE processing has been completed, and “Written” indicates that data writing has been completed.
These pages statuses are allocated as a Bitmap in the first 64 bits area of the Block header using 0/1 = OFF/ON.
typedef struct payload_hdr {
uint32_t magic:32; // 0xdeadbeaf, 0xdeadc0de, 0xffffffff
uint32_t time:32; // system time
uint32_t comp:1; // 0:no-compress , 1:compress differencial,
uint32_t type:5; // 0: DC,
// 1: ANTP(SPD), 2: ANTP(PWR),
// 3: ANTP(CAD), 4: GYRO,
// 5: ACCELEROMETER,6: TEMPERATURE
uint32_t padd:26; // padding to 128 bits(reserved)
union {
struct {
uint32_t length:9; // ANT data length
uint32_t padd:23;
} ant;
struct {
uint32_t dim:3; // 0: x, 1: y, 2: z
uint32_t count:11; // count: 1024
uint32_t base:16; // base value
uint32_t padd:2;
} gyr;
struct {
uint32_t count:11; // count: 1024
uint32_t base:16; // base value
uint32_t padd:5;
} tmp;
};
} payload_hdr_t, *payload_hdr_ptr;for(auto offset = 0; offset < PAGE_SIZE;) {
compress_ptr cp = (compress_ptr)&rbf[offset];
if (cp->magic != MAGIC) {
DRIVER_INFO(
"magic ..(%08X) offset: %d", cp->magic, offset
);
break;
}Error determination process conforms to the flash management systems of many previous paper, and is based on the idea that write errors can be localized to the block/page being processed in the event of an unexpected power loss.
Wear Leveling
Place a payload header on each layer, and use differential data compression on the payload to prevent unbalanced access to specific blocks.
These simple methods can eliminate the need for a file systems when storing time-series data such as telemetry data, and more.
- Reduce DMA times
- Reduce processing used MCU cycles.
- Improve system Performance.
- Against to unexpected power loss.
Against to unexpected power loss
The following restrictions allow simple use of HW interfaces without infringing on other companie’s IP, and supported against to unexpected power loss.
- Writing is limited to sequential writes of sensor data to BLOCK and Page.
+ Random access to specific data(file) is not supported. - Validaion of write errors is performed only when the read processing.
+ Garbage collection is not required. - No virtual address, and no inode design, only support the full scan in read processing.
- No file system mount process, i.e., Fast mount
Other file system
File systems that can withstand unexpected power loss include the Linux standard ext4, which is a journal system, and the JFFS file systems with a log structure.
In the case of ext4, the porting cost to PICO is high, and the file-system itself uses too many PICO resources.
The JFFS file-system takes a long time to mount, which eliminates PICO’s advantage of being able to boot immediately after Power ON.
Embeded file system such as FAT and YAFFS were not used for reasons of unexpected power loss.(The file systems compatible with PICO-SDK include no-OSD-FatFS-SDK, renesas, kyoto software research, etc.)
Compress the 100 Hz data from sensors
100 Hz data from gyro, angular acceleration, and temperature sensor data, in differential compression format to increase the data integration rate per NAND page.
[INF]loggerfs_flush.cc:261 : block: 1 , page: 0 to block: 1 , page: 1, compress:(3864 -> 1977, gyro: 644, antp: 0, flags: 00000020), time:2345
[INF]loggerfs_flush.cc:261 : block: 1 , page: 1 to block: 1 , page: 2, compress:(2136 -> 1113, gyro: 356, antp: 0, flags: 00000020), time:2345
Compression with this technique reduces the DMA size required for a single write by as much as 50%.
Measure performance about SPI/NAND interface
Conclusion
In this article, we introduced an approach to unexpected power loss that uses SPI/NAND directly and does not require the file-system(any modification).
We hope this article will be an example of a use cases, such as telemetry user application.
