Givelify’s secret sauce for maintaining 4.9 out of 5 stars on our iOS and Android Apps

Givelify, one of the most loved and trusted mobile giving platforms, is also one of the highest-rated. We are consistently rated at 4.9 stars out of five on both the iOS and Android platforms. This is achieved by never compromising on our promise of a five-star experience to our donors. Givelify’s engineering and quality teams have operationalized this promise with a strict multi-tiered release process for Givelify’s iOS and Android applications. We call this approach the release train and only put our iOS and Android applications through this process.

What is a Release Train?

A release train is a methodology that ensures consistent, regular delivery of features. The critical thing is that it fosters discipline by forcing teams to adhere to deadlines and reducing scope creep.

Our release train consists of two types of teams: cross-functional sprint teams (software engineers, designers, quality analysts, and test automation engineers) responsible for building thoroughly tested high-quality features. The other is the release team of quality analysts (QA) who will be accountable for successfully regression testing entire release builds, verifying new features, and getting them ready for public launch.

Release Train Process

usually there are multiple sprint teams and a single release team

One thing to note is that there will be multiple sprint teams and usually a single release team. Sometimes, you can have two release teams, with one focused on iOS and another focused on Android.

The mission and goals for the QA members in the release team:

The mission of the release team is to ensure that iOS and Android release builds are show-ready and to sign off on the release. They will also ensure that all regression test tickets are flushed out so we can build UI automation around them. Finally, they will ensure that the release candidates will not impact our servers’ performance.

The mission and goals for the QA members of sprint teams:

The mission of the QA members in sprint teams is to advance quality ownership, enable test automation, ensure new features are thoroughly flushed out, clean out old test cases made stale by feature change, and work closely with other members of the product development team (engineers, designers, product managers). They will ensure backlog grooming, thoroughly flush out test cases, and decide when to test manually and when to automate. Most importantly, they should be integral to their sprint teams to ensure features are ready for production.

How does the release train work?

The various sprint teams take a git-flow approach. At the end of every sprint, the following happens:

· At the end of every sprint, the respective iOS and Android Lead engineers will create a release iOS and a release Android candidate (I suggest creating stage/QA and prod builds of the candidates). The release candidates should only include all features (bugs, user stories, etc.) that have been tested and merged.

· The next business day after the end of the sprint, a Hand Off Meeting is held between key release and sprint team members. Once the sprint teams hand over to the release team, the sprint teams start working on the next sprint (release).

· The release team will then take the release candidates (builds), run regression tests, and re-test all new features.

· If the release team identifies any bugs, they will be addressed by the sprint team associated with the bug. If we do not know the team, the engineer leaders will decide who will take on the bug. Addressing bugs caught by the release team will be the highest priority.

· Once the release team approves the build for release, the head of the release team, the iOS team lead, the Android team lead, the engineering leader for the respective department, the customer support liaison, the group product manager, the SRE Engineer for the team, will review the release checklist and sign it. (See below for more details on the release checklist.)

Once approved, we take a phased release approach, incrementally scaling from 1% to 5% to 20% to 100% (some call these canary releases).

What is the release process?

• Once the app store approves them, we release 1% of our donors. Do not release on your busiest time and release early in the day so teams can monitor for issues, support tickets, etc. At Givelify, we release on Monday mornings. The apps should be at 1% through at least one of your busiest activity periods to get some realistic product usage.
• The following Monday, the team will review customer support tickets and other logs per the app scaling checklist. Once the appropriate individuals have signed it, the teams scale apps to 5%. The apps should be at 5% through at least one of your busiest activity periods to get some realistic product usage.
• We repeat this process until we scale to 100% unless we notice an issue that needs to be addressed (see below for common issues to look out for); In that case, we halt the release of this build and address it either through hotfixes on the server or through the next build in the train.

What are some common issues I might encounter during the release process?

• Are you seeing a spike in support calls?
• Is the crash crate above approved limits?
• Is server activity abnormal? For example, is there a spike in API traffic or a massive unexpected change in API activity patterns.
• Is there an increase in server-side API errors or other backend errors that you can attribute to mobile activity?
• Is there a drop in logging? You always want to ensure your products are properly instrumented.

KPIs that measure the success of the process

• QA, engineering, and product teams’ CSAT (Customer Satisfaction Score) scores
• Test automation coverage for unit tests and regression tests
• Ensure that test automation keeps up with new releases
• If the coverage rate drops, then the team needs to address it[JV1]
• The number of kickbacks caught by the release team. A high kickback rate by the release team is not ideal; it can throw the entire cadence off balance. If the team notices many kickbacks, the sprint teams should have a retro and see how to reduce the kickback rate.
• Time to do release testing. Release testing should take about a week. It should take at most two weeks (a standard sprint). If it does, either a high kickback rate, test automation coverage is weak, or the release testing team needs to be staffed better.[JV3]
• The number of 100% releases in a given year: If the process works well, we should release a new version every two weeks.

KPIs to measure the quality of each release:

· The number of production bugs reported

· The 5-star app rating. A key metric of customer satisfaction

In Summary: Givelify ensures a 5-star experience through a robust multistep release train approach to development, testing, and releases. It starts with cross-functional sprint teams building high-quality features; release candidate builds are then passed to the release team that thoroughly tests the release build candidates before there is a joint approval for release. This is followed by a multi-week scaled released process that rises from 1% to 100% over three weeks.

Our final secret ingredient: At the end of the day, our people in these teams bring it all to life week after week. No process on its own will ensure high-quality delivery. The teams’ ownership, camaraderie, and teamwork ensure the processes work and provide continuous, reliable, high-quality delivery of our iOS and Android Apps.