Open release notes, from day “Hello, world.”
March 25, 2015—In an effort to keep everything we’ve been up to out in the open, here are the previously archived release notes from day 1 until we started blogging on Medium.
We hope they’re useful. If nothing else, it’s fun for us to look back over the last 5 months.
Taylor Downs, core contributor at openfn.org
Version 0.9, Faster 2015–02–02 17:14:13Z
This is a major release today including upgrades to the importer itself?lots of speed and stability improvements?along with a bunch of UI enhancements. When the touch-enabled mapping page is done (we’re shifting away from drag and drop) and we’ve finished the PayPal payments section, we’ll be at our 1.0, and launch on the Salesforce Foundation, ODK, and SurveyCTO blogs.
Incredibly sad news 2015–02–02 07:13:11Z
Gaetano Borriello, the “visionary and leader behind ODK” passed away on February 1st, 2015.
Version 0.85, More real estate on map edit page, limitation on lookups 2014–12–24 13:56:41Z
We’ve made more big changes to the map edit page so that more ODK fields are visible and the floating top bar takes up less room. Also, Katie has IDed a limitation in the lookup matching functionality. As it stands right now, the API name of the relationship field must be the same as the API name of the destination object from which you’d like to choose a matching field. This is obviously not intended and limits the functionality fairly seriously. I’m digging in right now and hope to have this sorted in the next few days.
Version 0.81, New UI 2014–12–22 16:58:18Z
Just pushed out about 60% of the new UI, and am excited to gather feedback. First and foremost, we’re trying to remove clutter from the map edit page. Give me a shout if you’ve got suggestions.
Version 0.71, Rollback 2014–12–09 18:07:06Z
We’ve had to roll back to an earlier version as the new XML parsing that allowed for group handling junked up the “create new mapping” button on the UI. Integration code is still stable, but not much use if users can’t add more mappings. Hope to have this foxed by the AM.
Version 0.7, Groups now working 2014–12–08 17:13:49Z
Thanks to lots of Stuart Corbishley’s brain hours, the ODK integration now handles groups in X forms. This is a big step forward. We’ll be merging in photo-handling functionality tomorrow as well, if all goes to plan. Stay tuned!
Version 0.651, Ecosystem update 2014–12–04 15:18:02Z
Have come across these three tools for designed to help users to find the right tech:
- Humanitarian Nomad Online Selection Tool
- mHealth Software Systems — Google Fusion Tables
- Medic Mobile’s Matchmaker — Just spoke with Amanda at Medic and we’re hoping to fold some of this design into a selection wizard on the site next year!
Version 0.65, User accounts, credits 2014–12–04 10:28:28Z
We’re piloting a fee structure based on the number of enabled maps, as this is more closely related to actual resource consumption. To enable a mapping, you’ll need a credit. Once a mapping is enabled, you can import manually or switch on the automatic importer. Since a full billing system isn’t yet in place, just contact me when you need credits added to your account.
Version 0.61, Removing destination objects entirely 2014–11–28 13:03:50Z
Folks, when removing destination objects from the mapping page you need to first remove the mapped fields to that object and then remove the object itself. Early next week we’ll be able to cut out this tedious step. Sorry for the extra work, in the mean time.
Version 0.6, Empty repeat blocks solved 2014–11–28 08:37:00Z
Stu was up late last night to handle empty repeat blocks as generated by ODK Collect. The patch passed all of our stored tests and Livelyhoods’ specific use case. In short:
- There are no known issues with first-level repeat blocks at present.
- Fields within non-repeat block “groups” can only be used when they map to objects that are marked as “repeat block destination” objects. This means that the user can’t access the “future ID1” of that destination object to link child objects to it directly. It’s still possible to link child objects to that group field destination object using a lookup or MD field and matching on a unique field.
Technical pipeline at present:
- Salesforce record update handling
- Full SOAP handling
- Rest API handling
- ODK Photo handling, with KopoKopo as test case
Finally, props to Stu for building out a more robust suite of tests. In the last major update (v.4) we solved one problem while introducing instability elsewhere. Now that our suite of tests is growing along with our functionality, we can be more confident with each new release.
1To insert new nested objects (e.g., Household__c with Household_Member__c) you must first insert the parent object and then the child?i.e., the order matters, and is specified by the digit to the right of your destination object in the mapping page. In our example, you should see Household__c 1 and Household_Member__c 2.
In order to link Household_Member__c 2 to Household__c 1, without knowing any information about the Household__c you use the “future ID” of that object-to-be-inserted. It appears as a new source field in the mapping matrix below:
New [ Household__c_1.ID ]
There must be a relationship field in Salesforce on Household_Member__c that looks up to Household__c. Drop that field into the destination space for New [ Household__c_1.ID ] and the Household_Member__c 2 will get linked to your Household__c 1.
If Household_Member__c 2 is getting its source data from a repeat block, make sure to check the chexbox and you’ll notice that the future ID for Household_Member__c 2 disappears. You can’t use New [ Household_Member__c_2.ID ] as a source for some other destination, because there could be many of them and we don’t have a way of choosing which one to use. If a third destination object must be linked to an earlier inserted object from within a repeat block, you are forced to match based on some other unique field?i.e., you must know something about the Household_Member__c 2 at the time of surveying!
Version 0.52b, Why don’t my records appear instantly? 2014–11–27 09:52:15Z
We’re at an exciting stage in the life of the importer. At present, just over 38,000 survey submissions have been successfully imported. Most of these are from the household survey that is being conducted in Jharkhand, India through KGVK.
So far, we’ve been able to keep costs fairly low: we run a single worker on Heroku for $34.50/mo and maintain a database submission successes and failures for another $9/mo. We check for new ODK submissions every ten minutes because ODK can’t actually push them to us. Once submissions have been fetched, they’re kept in a queue (currently, we use Resque) and then inserted to Salesforce in order of apperance in the queue.
The slowness Vivaan noticed yesterday (thanks for pointing this out, V!) is because a large number of submissions from another user (KGKV) had entered the queue before he tried to import records from Livelyhoods. We’re doing two things to address this problem:
- We’re soon switching to Sidekiq, which will effectively allow us to maintain multiple queues, reducing the wait time per user as the number of users increases.
- In the new year we’ll move off of Heroku to AWS, which will allows us to fire up more workers for less money. No need to do this just yet, as we really only have 3 users. In the future, however, we’ll be able to scale up more easily and keep lag down.
It’s also worth noting that we’re still playing around with the best way to keep performance high and make the service affordable for everyone. At present, we’re asking folks for $10/mo. (well below the DIY cost of $43.50) but are considering breaking this out into non-profit and for-profit pricing, or pricing based on the actual system load?i.e., if you don’t need real time syncing and only have a few submissions per month you’d pay a lot less than someone with several hundred submissions per day who demands that we pull very frequently from ODK.)
Please let me know if you’ve got thoughts around how to make this simple, fair, and affordable while ensuring that we keep performance high.
Version 0.52a, Clarification on UUIDs 2014–11–27 09:41:13Z
Check the “UUID” checkbox beside /meta/instanceID and then map that source to a unique destination field (usually called “ODK_Key__c”) on each object you’re inserting in Salesforce, even if it’s a child object that’s getting data from within a repeat block.
If the destination field in Salesforce is set to unique and the altered “UUID /meta/instanceID” is used, you can be sure that no matter how many different mappings are made or how many times you manually pass the importer over the same data source, you won’t get duplicates in Salesforce.
Version 0.52, Enketo submissions 2014–11–26 17:33:29Z
It seems that Enketo passes data to ODK Aggregate differently than ODK Collect does. With Enketo, empty repeat blocks are not actually empty; they get nil values passed in to a single item in the array. This explains why we saw “ghost records” showing up even when there weren’t any loans in KGVK’s big household survey, for example. According to the data source, there really was a loan — Enketo had passed in a blank loan.
We’re currently working on handling truly empty repeat blocks (for which ODK does not pass data, rather than passing a “nil”) for an implementation with Livelyhoods. Hope to see that working tomorrow evening.
Version 0.51, Douglas from Wikimedia South Africa 2014–11–26 13:12:07Z
Just had a great conversation with Douglas from Wikimedia South Africa. Over the next few weeks as we develop listings we’ll be looking to find our particular style of “authoritative narrative”. We’re trying to pull out the value words, jargon, and complexity from the 400 character tile listings. Please feel free to submit new listings or update them while trying to adhere to this “first sentence of Wikipedia” style.
Version 0.5, Functional listing forms, clarification on RecordTypeID, re-design 2014–11–26 09:11:17Z
You’ll now see links to two Formstack forms in the marketplace. Feel free to add new listings, or update existing listing content. We’re thinking through the costs and benefits of two different styles of listings. If we allow developers to post and manage their own listings, they’ll inevitably feel more like advertisements, but it will cost less to maintain the site in the future. Alternatively, we could write each listing ourselves, and strip down to super objective functional strings of 400 characters for the tile. The latter would make for a much better (but more expensive to maintain) experience, and for now we’re settling on a happy medium… listings can be submitted by anyone, but they won’t be posted until they’re reviewed and sometimes re-written by staff. Over the next few weeks we’ll be cleaning up existing listings and would love feedback both on the content and the presentation of this information.
RecordTypeID needs the record type label as its data source, not the ID or the API name, as previously specified. Apologies for the confusion here.
Version 0.42, welcome page, logout glitch fix, repeat block ghost fix, UUID fix 2014–11–24
We’re trying out a new welcome page and hope to open the site up publicly in a few weeks. Trying to condense everything into a few short sentences is a great exercise. I’m excited but daunted by the technical challenge on the modular integrations side of the platform. Jon’s work at the outset was constrained a bit by the need for a single working integration (ODK to Salesforce, one way) and Nic and Stu are now charged with genericizing the importer in such a way that new products can be introduced as endpoints more easily in the future.
- If you’re using the UUID functionality, please make sure that you’re mapping that source to a unique field on each Salesforce object.
- The repeat block ghost record issue (inserting a record where there was nothing in the repeat block) has now been resolved.
- It’s now possible to log out.
- It’s now possible to access the site without a login.
Version 0.41, product listing management 2014–11–14 15:00Z
Hey fam, this is a cool one. Now using Force.com to manage product listings. More soon.
Version 0.4, ODK/SF upgrades, product listing mockup 2014–11–13 17:09Z
Hey fam, quick features update on the latest release to production:
- Error handling is now accomplished through a “process log” which allows the user to see all attempted submissions, with their IDs, and whether or not they succeeded or failed.
- You can now map to record types, using the record type name (that’s the API name, not the label) in your ODK source.
- You can see the last attempted submission from ODK (the “cursor”) and clear this cursor position (so that the bridge attempts all the records in the ODK database again). Combined with the UUID functionality described in my last update, this means you can be completely sure that each submission hits Salesforce or shows up in the error log.
- You can change the insert order for Salesforce objects by dragging the objects left and right.
- You can refresh Salesforce objects to pull newly created fields from Salesforce into your existing mapping.
- The repeat blog bug has been fixed.
- There is a semi-functional (it is searchable for text, and nothing else) mockup of the marketplace on the site.
Roadmap update, in order of priority:
- Update existing records in Salesforce by specifying a unique ID field to match on.
- Handle photos, video, and audio either as attachments on your Salesforce records or as URL fields that link to the content stored on ODK aggregate.
***Today we found and replicated a strange bug with the lookup matching picklist field. It is no longer appearing and we are on high-alert. Please drop me a line if you notice that the picklist which populates your matching options for a lookup field goes blank while creating or editing a mapping.***
Version 0.3, Unique IDs for each destination record 2014–10–29
ODK Aggregate generates a single unique string for each submission. We’d been using this “metainstanceID” to map to “ODK Key” fields in Salesforce so you have a string that matches in both databases for data validation. An important security blanket, if you will!
The problem is that “ODK Key” in Salesforce it cannot be set to unique (which would 100% guarantee the blocking of duplicate records) for objects that receive multiple records per submission.
For example, a new beneficiary linked to an existing clinic with a new visit and two new test results would look like this:
- New Beneficiary, ODK Key = uuid:c35c2cb3–3d05–4660-bc22–5de0d4999fd9
- New Visit, ODK Key = uuid:c35c2cb3–3d05–4660-bc22–5de0d4999fd9
- New Test Result 1, ODK Key = uuid:c35c2cb3–3d05–4660-bc22–5de0d4999fd9
- New Test Result 2, ODK Key = uuid:c35c2cb3–3d05–4660-bc22–5de0d4999fd9
You could make Beneficiary__c.ODK_Key__c and Visit__c.ODK_Key__c unique in Salesforce, butnot Test_Result__c.ODK_Key__c.
Much in the same way that ODK Briefcase generates unique keys by appending the “metainstanceID” for each submission with a string, we now give bridge users the option (through a checkbox next to metainstanceID) to use this modified metainstanceID and have a unique “ODK Key” field on every destination object. I know I shouldn’t always show how the sausage is made, but this is very straightforward and you might be curious.
IF UUID = true, convert meta/instanceID to:
IF (repeat = FALSE, meta/instanceID + “/” + designator, meta/instanceID + “/” designator + “.” + array_position )
You’ll see this now and a new error handling UI in the next couple of days. You’re also always welcome to check out our staging server if you’re bored and curious about what’s next in the pipeline: the-staging-bridge.herokuapp.com