Building a more useful & sustainable Secure Data Kit in Q3
Q3 was a big quarter for Secure Data Kit. We focused primarily on adding more XLSForm support to our legacy SDK Android apps, and optimizing the performance of SDK to provide our customers with a quicker more stable platform.
Added support for Nested Groups and Repeaters in SDK mobile app
If you’re familiar with XLSForm — a format to build forms in excel for data collection, widely-used in the global health community — you know that they’re super robust and allow for some really advanced form design. Secure Data Kit supports most XLSForm functionality, and as of Q3 we added support for nested groups and repeaters. What does this mean, and why is it useful? I’m glad you asked.
To understand the nested groups functionality we first need to understand the concept of a group. A group is a set of questions that’s grouped together for a variety of reasons. You may want to group questions because they all have to do with a similar topic (like demographic information), or because you want to apply certain logic to an entire group of questions (like ‘only ask this group of questions if the respondent has given consent’). A nested group is a group inside a group. This is useful to further organize your form and apply even more advanced logic where necessary.
Repeaters are similar to groups, except you can set that group to repeat based on any number of criteria. For example, you may ask a respondent how many children live in their household, and then proceed to present them a group of demographic questions, repeated for each child. Repeaters live in their own datasets in SDK, and are automatically linked to their parent record with a unique identifier. Prior to supporting repeaters, we required that our customers create separate forms and their own unique id system to link related records across forms, but now that’s all done for you behind the scenes.
Performance Optimizations
We’ve released some optimizations to Secure Data Kit that will bring further stability and increased performance to the platform. These improvements can’t be seen, but will give you peace of mind knowing that you’re interacting with a quicker, smarter and more secure platform.
Before we dive into each update, let’s first talk about workers (this is going somewhere, I promise). What’s a worker? A worker is like a tiny little robot that performs tasks all day long. How does it know what to do? Well, we tell it what to do! Prior to these optimizations we had one worker that was super busy, and was assigned to do several different tasks:
- Store submissions — when we receive submissions from the field the first thing we do is store the submission file (an XML file) on a server for easy access, and to use as a backup if we ever need it.
- Process submissions — once the file is safely stored, we process the submission by parsing out the relevant data and making it accessible in the Secure Data Kit platform.
- Export and email files — each time a user requests any type of file (whether it be a report, a dataset, etc) that file is generated and emailed to them.
Talk about a full plate! Sometimes the worker would get overloaded. We understood, because we gave it nearly 100,000 submissions to store and process each week, and many clients were exporting data and running reports several times each day. When this worker started getting overloaded, things began slowing down everywhere because everything was interconnected, and it wasn’t a great experience for our customers. You may have experienced the affects of this yourself — ever tried to upload a form and nothing happened? Ever try to generate a report, and you never got it? Yep… that was the overloaded worker.
The good news is, in all this we never lost data (it was just stored and processed slower than we’d like). But it still became clear that we needed to make a change to provide our customers with a great experience, and add stability to Secure Data Kit. That’s why (drumroll please…..) we moved all of these different tasks to their own workers, making individual actions within Secure Data Kit function independent of one another.
So, how does this impact your interaction with Secure Data Kit? Now that these different tasks have their own workers, no one worker is overloaded anymore. That means we can store submissions all day long without ever slowing down the worker that processes submissions, and we can process submissions all day long without ever slowing down the worker that exports and emails files. You’ll notice Secure Data Kit moving a little quicker because it’s working smarter.
In addition, we’ve also optimized the way the SDK2 mobile applications pull data down from the server. Prior to this update, SDK2 attempted to pull all the data down at the exact same time. This worked in most cases, but really large datasets would sometimes struggle or even fail (if you’ve seen the error message ‘Unable to add survey records to your device,’ this is what you were experiencing). We now pull the data down from the server and add it to your device 150 records at a time, rather than all at once. This puts much less stress on the platform and gives us a much higher success rate when adding records to the device.
We make updates like this all the time, but they often go unannounced. Given how impactful these changes are going to be for both stability and speed we wanted to make a special point to share them with you!
If you’re interested in reading more detailed explanations of any of the items mentioned in this post, check out these resources written by the developers who did all the work!
For a glimpse of what’s coming in Q4 2018 (and beyond!), check out Secure Data Kit’s product roadmap here.
Originally published at https://www.linkedin.com.
