Bring Customers Joy With Automation
Automation these days can be viewed as an attempt for businesses to separate themselves from their customers to save time or money. It is used to send customers down automated tunnels so as to reduce costs and remove the annoyance of having to deal with complaints. The business might not intend to shunt customers aside, but the effect is the same.
At Appian, we think differently. Our approach to automation keeps human needs at the forefront.
Appian is a platform-as-a-service company that empowers customers to create custom applications to deliver amazing user experiences across web and mobile devices. It allows non-developers to quickly make beautiful and sophisticated applications.
Appian’s customers can customize our default Appian mobile application so it fits seamlessly into their own branding, look, feel, and feature set.
As we at Appian exposed more and more configuration options to our customers, our build system had become slow and painful. I want to share how we tackled this problem and succeeded in building an automated, simple, and fast system that allows our customers to build their own custom version of the mobile application in minutes, instead of days (or even weeks).
I will give you more background on this problem and how we incrementally improved the process and system until we finally automated the entire process. Not only did we solve the problem, but we also made it a joy to use.
I challenge you to think about the problems you face as big opportunities to make significant improvements for you and your customers rather than just a pain to resolve. How can you make lemonade out of lemons?
Background
In my previous article, Rescue Your Mobile Builds from Madness Using Fastlane, I mentioned that Appian provides customers with branded versions of the mobile app for their needs.
For the customer to get a custom mobile app, there were several tasks that had to performed:
- Customers had to review the 20 page document that Appian had created in order to consolidate the knowledge required to work with both iOS and Android application creation.
- The customer had to create an Apple Developer account for iOS and a Google Play Store account for Android.
- After that, the customer needed to provide Appian with correctly formatted images as well as code signing¹ and provisioning² files for the iOS apps.
- Customers then had to create an Appian Support Case with the images, code signing, and provisioning files. They had to also ask for the configuration options that they wanted.
- Next, the Support team then had to manually insert the customer’s files into a storage system and then create a Jira³ ticket to request the customer’s branded build.
- Engineering then had to notice the request and self-assign the Jira ticket, run a Jenkins job that inserted the customer’s files and configurations into the source code, build the mobile app, and, for iOS, code sign and provision the app using the customer’s assets.
- Finally, Engineering needed to attach the mobile app to Jira, Support had to attach the mobile app to the Support Case, and the customer had to download their build and then upload it to the App Store or to their Intranet for Enterprise builds.
At each step of this process, there were plenty of opportunities for user error (and we encountered many of them). The farther along in the process that we noticed an error, the more time the customer had to wait for their mobile apps, and the more frustration we all felt.
Turning a challenge into an opportunity
The customer had to perform many manual steps to initiate a branded build request. These steps are complex and do not fit in the customer’s normal day job.
Given that, it was not a surprise to us when we were given assets that were not formatted correctly. It required a lot of time for the customer, for Support, and for the Engineers to diagnose the problem and iterate between each other to get the correct data.
It was a frustrating and time-consuming experience. Very far from ideal. Dealing with branded build issues did not occupy long stretches of our time, but it was a frequent cause of interruptions. We knew that we needed to simplify the customer’s experience and simplify our system while keeping the customer’s assets secured during the whole process.
This is what we proposed:
- Automate most of the generation of the customer’s code signing assets. This makes our customer’s process easier.
- Automate the submission of the build request, the delivery of the built mobile app, and notifications of the status of the build requests. This removes Engineering from the process.
- Provide a self service app on an Appian community and customer support website, that customers could use to upload their assets. This removes Support from the process.
From Fix to Feature
As our team is often interrupted by out-of-band requests or internal infrastructure incidents, we knew we could not just put our heads down and get everything done in one big project. We had to build the system out in increments that would provide value and that would also allow us to be interrupted without too much damage to our process.
The following is a high level overview of how we achieved each increment.
Stage 1: Standardize Asset Generation
We needed to ensure that what the customer sent us was correct and in a known format so that later on, we could build automation around how we handled those assets.
We created a script named the Appian Provisioner that would guide the user to create everything that we needed:
- A locked keychain with the private key and the Apple provided certificate used for code signing.
- An encrypted random password to that keychain so a third party could not unlock it.
- Provisioning profiles that would later be bundled into the mobile application.
We had to also ensure that running this script was as simple as possible. Not all of our customers are technical, and they may delay using something that they perceive as being complex.
Read “Painless Code-Signing Assets” to learn how we automated this and reduced many of the headaches that we and our customers had to suffer through when the wrong data was sent to us.
Stage 2: Automate Securing and Using Customer’s Assets
Now that we had the customer’s signing assets in a known and secure format, we still had to personally run terminal commands to put their keychains and encrypted passwords into a secure location. We had to do this in such a way that would not allow us to view their secrets.
Thankfully, Appian Engineering already uses a Vault server, which is a secured database for managing secrets and other sensitive data. We addressed our issue with a script that extracts the customer’s keychain and encrypted keychain password and stores them in Vault.
When we wanted a custom mobile application for a customer, we had to make sure that only the build system could read and decrypt the customer’s keychain password and keychain from Vault. We did this by generating a one-time time-limited Vault token⁴ that can only be used by the build system to retrieve and decrypt the data. This limited the potential damage if the token were ever to fall into the hands of a bad actor.
Read “Automate Secure Code-Signing Assets” to learn how we automated securing customer credentials in a way that only build systems can read them.
Optimization! Customize and sign vs. rebuild the entire mobile app
While working on this project, we learned a lot about the ins and outs of code signing and realized that we did not need to build the application from scratch for each customer. Instead, we could take apart an already released version of the Appian mobile app, customize it, and then code sign it with the customer’s certificate. This reduced the time to create the customer’s build from 20 minutes to 2 minutes 🎉.
Read “Customizing and Code-signing a Mobile app” to learn how stopped rebuilding our mobile apps and simply customized a pre-built application to save us 18 minutes per request.
Stage 3: Automate Accessing Customer Assets with Vault App Roles
At this point, we had a much more streamlined process, but we still were more involved than we wanted to be. We still had to manually start the job and provide a Vault token to allow the job to access the customer’s code signing assets.
To remove this final manual step, we used Vault’s App Roles⁵ so that the build machines could automatically get the customer’s code signing assets. This took away the need for us to manually create a one-use token in order for the Jenkins job to resign the customized mobile application.
Read “Automate Code-signing with Vault App Roles” to learn how we used Vault and Ruby to code-sign the customized mobile apps for us.
Stage 4: Create a Secure Signing Server
Now that we had automated the resigning of our customer’s mobile apps, we wanted to eliminate the risk of anyone getting access to their code signing assets.
In all of the previous stages, we were using the same Jenkins worker nodes that we use to build our regular app to also to resign our customer’s app. In order to lock down our customer’s code signing assets further, our IT and InfoSec experts worked together to build out a Mac mini on its own network. It has its own Launch Agent to monitor and serve build requests. Only it is allowed to use the Vault App Role to access the customer’s signing assets from Vault.
Stage 5: Automate Delivery of Branded Builds to Customers
At this point, we have a nearly automated system. Here were the remaining manual steps:
- The customer still had to run the Appian Provisioner to generate code-signing and provisioning assets.
- The customer still had to create a Support case to upload the zipped and secured assets as well as the images that they wanted.
- Support still had to create a Jira ticket and attach these files to that ticket.
- Engineering still had to download these files and run the script to put them in the secure location.
- Engineering still had to wait for the Signing Server to provide the customized and resigned application.
- Engineering still had to attach the built app to the Jira ticket and assign it back to Support.
- Support still had to take this app and attach it to the Appian Support case that the customer had created.
- The customer had to download the application from the Appian Support system.
- The customer still had to upload it to the App Store or their Intranet for Enterprise builds.
That was not good enough for us. Fortunately, we work on a platform that is perfect for automating and connecting systems: Appian.
So, we built a beautiful application in Appian that allows the customer to upload their own files, get them validated, initiate a build with the Signing Server, and receive the final application. At each stage, their code signing assets are secure.
From the original set of seven steps that used to take hours or days, we’ve streamlined the process down to four simple tasks that take only minutes:
- The customer creates an Apple Developer account for iOS and a Google Play Store account for Android.
- The customer runs the Appian Provisioner to generate code-signing and provisioning assets.
- The customer uploads their assets to the Mobile Branding application in Appian.
- The customer waits two minutes to get and download the built application.
- The customer uploads the built application to the App Store or to their Intranet for Enterprise builds.
Now, the Support team is free to handle other issues and Engineering can focus on new features to make customer’s work less burdensome and rote.
As a bonus, the customers get their custom mobile application in minutes instead of days!
As one customer told us, “this capability is why I chose Appian. Other companies would have required me to integrate with an SDK or provide them access to my Apple Developer account”.
Conclusion
What started out as a problem turned out to be a huge opportunity. After building up this convenient, simple, and approachable system, we now have customers using our system who, in the past, would have never considered customizing their Appian experience for their mobile users.
My motto is that if you have a process that is manual, requires a lot of human interaction, is something that can be messed up, forgotten, or requires a lot of training, automation should be considered. Next time you face a problem with one of your processes, take a step back and ask yourself: can I automate some or all of it? If not, can I make the discovery of how to use it automated, obvious, and validatable?
Choose to reduce the mental burdens to yourself and to your users. Choose to make the world better, 1% at a time.
Keep your eyes open for future posts where I’ll describe the details of how our DevOps streamlined this process.
Update
The Mobile Branding application has been especially helpful to customers who have iOS build, but who are not able to go to the office and use their Mac computers due to the covid-19 pandemic: they can just request a new build from any computer.
Afterword
While I may have provided the framework for how we were going to solve this problem, the system was made beautiful by the amazing people who I worked with. They are the ones that breathed life and energy into the concept.
If you find this kind of work interesting and want to work with creative, passionate, and smart people, there are plenty of other projects to work on like this at Appian. Apply today!
Appendix
[1]: As summarized by Wikipedia: Code signing is the process of digitally signing executables and scripts to confirm the software author and guarantee that the code has not been altered or corrupted since it was signed.
[2]: Provisioning a iOS app involves embedding Provisioning Profile files that indicate which devices the application can run on, how the apps can be distributed (over the air, via the App Store), and what entitlements are enabled (such as Push Notifications or iCloud connectivity).
[3]: Jira is a bug tracking system from Atlassian.
[4]: A Vault token is something that Vault can use to access a subset of the secrets that it contains instead of using a username and password.
[5] A Vault App Role is like a username/password for an automated system, also known as a Service Account. The way that Appian implemented this App Role allows the password to be rotated on a frequent basis so even if someone got access to the credentials, it would only be useful for a short period of time.
