Olympics of Top 5 Cloud iOS Continuous Integration Servers

Continuous Integration a.k.a CI for iOS apps is one of my favourite topics. Continuous Integration is essential for every iOS project now as the number of developers growing in the iOS teams. As of now, I have deeply experienced with self-hosted CI servers like Xcode Server, TeamCity, and Jenkins for iOS projects. However, there are still an array of cloud-based CI services available in the market which are becoming very popular in the recent days. There are a lot of articles and tweets going on for those cloud-based CI services. I am keen to try those services and get some exposure by using them with my favourite personal project XCFit. In order to do this, I have decided to play an Olympics game involving the most of the cloud-based iOS Continuous Integration services candidates.

Purpose

The most of the services taking part in this competition that I haven’t used before apart from TravisCI. It would be the good opportunity for me to make myself familiar with those CI services and compare them using my personal project XCFit. The purpose of this Olympics competition is to share my experiences and first impressions on those cloud-based iOS CI services. Please note that purpose of this competition is not to recommend particular service that you should use for your iOS project.

The whole purpose of this competition is to try and share the first impressions of these cloud-based iOS CI services.

Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of anyone else in this world. This evaluation is only performed against XCFit project and it may vary from project to project.

Olympic Players

The player involved in this Olympics games are the most used cloud-based CI servers which supports iOS and macOS platforms. The current list has these players

XCFit: The Game

This Olympic competition will be played against the project XCFit, the library we are going to use to try out the features of CI services mentioned above. This is one of my favourite pet project XCFit which is the small library that allows us to write a XCUITests in Behaviour Driven Development a.k.a BDD fashion. This project is interesting candidate for CI server test as it consists of

  • Swift Framework distributed with CocoaPods and Carthage
  • Swift Package that can be distributed with Swift Package Manager
  • An Example app using XCFit framework built with Carthage
  • Ruby package distributed using RubyGems to download Xcode templates

As there is a lot involved in this project so it would be interesting to see how all those cloud-based CI servers treat this project to run the builds on respective platform.

Olympic Athletics

The Olympic athletics, we will be using for this competition are

  • Sign Up & Project Setup
  • Default Build Configuration
  • Build Execution
  • Build Artefacts & Test reporting
  • Real Device Testing
  • Build Customisation
  • Parallelisation & Concurrent Builds
  • Build Pipelining
  • Infrastructure As a Code
  • Directory Caching: CocoaPods & Carthage
  • Code Signing & Distribution
  • Third Party Integrations
  • Speed
  • Secret Store
  • Xcode & Apple Developer Tools Upgrades
  • Open-Source Project Support
  • Learning Curve
  • Documentation & Support
  • Clean UI/UX
  • Price

The players will be evaluated on basis of these games. Each of the athletics will earn a medal that would be gold, silver or bronze. At the end, there will be pros and cons section where we will discuss the outcome of this evaluation. This will give the brief idea about the functionality of the cloud-based iOS CI servers and their features related to iOS development.

20 Olympic Medals

Each of the game mentioned above will earn a medal. The medals we are going offer are Gold, Silver and Bronze.

  • Gold Medal

The player will get a gold medal for the criteria if it is strong to satisfy user and met requirements clearly as well as exceptionally.

  • Silver Medal

The player will get the Silver medal for the specific criteria if it is performed alright and satisfy user up to the certain extent.

  • Bronze Medal

The player will get a Bronze medal if it’s met the minimum requirements but doesn’t satisfy user that much. This is the least medal that every athletics will get for the performance.

Each player will get 20 medals and the winner will be decided on the basis of the number of Gold medals. There is no good or bad CI services but the winner will be announced based on how each cloud-based CI servers will perform against XCFit project only. There might be different winners for different projects.

5 Magic Medals: At the end

There are 20 different games in the Olympics and each player will get a medal for each game. This means every player will get 20 medals for sure but there will be additional 5 magic medal that will be offered to the player while the evaluation is going on and announced once all players are evaluated. The 5 medals will be based on following criteria

  • Social Media support during evaluation
  • Correcting my observations during evaluation
  • Accepting facts and respecting competitors
  • Showing courage to improve services
  • User Feedback on Twitter & Public API

BuddyBuild : Player One

In the iOS Continuous Integration Server Olympics competition, the first candidate selected is BuddyBuild. The most heard name on Twitter and other tech blogging site, because of its simplicity and ease of usage. BuddyBuild is designed specifically for the mobile continuous integration which supports iOS and Android platforms. The clients of BuddyBuild include some big brands like The Washington Post, FireFox, Slack, Meetup etc.

Olympic Performance of the BuddyBuild

I have done the critical evaluation of BuddyBuild for each of the 20 criteria and the results are shared in the blog post here

At the end of 20 Games BuddyBuild Scored :

13 GOLD 🥇

6 SILVER 🥈

1 BRONZE 🥉

BuddyBuild— XCFit Build Logs

The build logs executed for the XCFit project are available here. We can navigate to details, logs and test tables to get the more information about the build.

Bitrise : Player Two

In the iOS Continuous Service Olympics competition, the second candidate selected is Bitrise. There are many articles on medium and lot of good words on Twitter about the Bitrise. Like BuddyBuild, the Bitrise is also designed for the mobile continuous integration which supports iOS and Android platforms. Bitrise also got some good clients like ABB, Foursquare, FOX, TNT etc.

Olympic Performance of the Bitrise

I have done the critical evaluation of Bitrise for each of the 20 criteria and the results are shared in the blog post here

At the end of 20 Games Bitrise Scored :

13 GOLD 🥇

7 SILVER 🥈

0 BRONZE 🥉

Bitrise — XCFit Build Logs

The build logs executed for the XCFit project are available here but you need to have an account to access those logs.

TravisCI: Player Three

In the iOS Continuous Integration Server Olympics competition, the third candidate selected is TravisCI, the most used cloud-based continuous integration for open-source and private projects. It’s truly the home of the open-source testing. As mentioned on the TravisCI page there are about 900K open-source projects and 600K users running the continuous integration on TravisCI. It’s deeply integrated with GitHub and setting up an open-source project on TravisCI is a minute away from developers.

Olympic Performance of the TravisCI

I have done the critical evaluation of TravisCI for each of the 20 criteria and the results are shared in the blog post here

At the end of 20 Games TravisCI Scored :

12 GOLD 🥇

5 SILVER 🥈

3 BRONZE 🥉

TravisCI — XCFit Build Logs

The Travis build logs executed for the XCFit project are available here.

Nevercode: Player Four

In the iOS Continuous Integration Server Olympics competition, the fourth candidate selected is Nevercode. It is designed specifically for the mobile continuous integration which supports iOS and Android platforms. The clients of Nevercode include some big brands like Sainsbury’s, Toyota, Eurostar etc.

Olympic Performance of the Nevercode

I have done the critical evaluation of Nevercode for each of the 20 criteria and the results are shared in the blog post here

At the end of 20 Games Nevercode Scored :

12 GOLD 🥇

5 SILVER 🥈

3 BRONZE 🥉

Nevercode— XCFit Build Logs

The build logs executed for the XCFit project are available here but you need to be logged into Nevercode portal.

CircleCI: Player Five

In the iOS Continuous Integration Server Olympics competition, the third candidate selected is CircleCI, the continuous integration service that is being used by the most popular tech firms like Facebook, Spotify, Percolate etc. The CircleCI and TravisCI have most of the things common but some features are unique to the each platform.

Olympic Performance of the CircleCI

I have done the critical evaluation of CircleCI for each of the 20 criteria and the results are shared in the blog post here

At the end of 20 Games CircleCI Scored :

12 GOLD 🥇

7 SILVER 🥈

1 BRONZE 🥉

CircleCI — XCFit Build Logs

The CircleCI build logs executed for the XCFit project are available here.

Olympic Medal Table for 20 Games

So far we have done with 20 games and medal table looks like this:

Magic Medal Stage

As of now, we can see that, there is a tough fight between Bitrise and BuddyBuild at this moment but other players like TravisCI, CircleCI and Nevercode are too close. However, the game hasn’t finished yet. We still have 5 magic medals to be awarded to each player according to the following criteria

  • Social Media support during evaluation
  • Correcting my observations during evaluation
  • Accepting facts and respecting competitors
  • Showing courage to improve services
  • User Feedback on Twitter & Public API

A medal will be awarded for each of the criteria. This means every player will get 25 medals and winners will be the player who won maximum GOLD medals. If there are equal GOLD medal then we will look for the number of SILVER medals.

Magic Medals Awards

Now that, we need to present magic medals awards for each of the criteria mentioned above.

BB = BuddyBuild
BR = Bitrise
TC = TravisCI
NC = Nevercode
CC = CircleCI

Social Media Support(SMS)

There were huge supports on Social Media during the evaluation process, especially on the Twitter. The tweets were liked replied and retweeted during the evaluation. The example Tweet can be found here. BudduBuild and Nevercode were top on Social Media support. CircleCI joined late but there wasn’t any interest shown by Bitrise and TravisCI. The medals for the SMS a

Correcting Observations(CO)

There were some features that I haven’t discovered in the first place but the players corrected me with right features by communicating with me on Twitter, Email and Blog comments. Nevercode, BuddyBuild and Bitrise contacted me and corrected most of my observations. The medals for the correcting observations are

Accepting Facts and Respecting Competitor

There are some conversations with Twitter and emails on accepting the facts and respecting the competitors work. BuddyBuild and Nevercode are good at it. The medals for the accepting facts are

Showing courage to Improve Services

Nevercode and BuddyBuild have shown strong desire to improve their services by email and with the tweet. The medals for showing courage to improve services are

User Feedback on Twitter & Public API

I have collected some user feedback on Twitter and also public APIs like the status badge and public logs for the open source projects. BudduBuild, TravisCI and CircleCI have both good feedback on Twiter and Public logs and APIs. Bitrise and Nevercode lack public logs and some Twitter users reported issues with Xcode 9 upgrade for Bitrise and Nevercode still has failing build for me with Xcode 9. The medals for user feedback and public APIs are

Final Medal Table & Winner

Now that, we have awarded all the magic medals to all players. It’s time to declare the winner of the competition.

The final Medal table:

The Winner is ……………..

Congratulations, BuddyBuild !!

I would like to thank all other players including Bitrise, Nevercode, CircleCI and TravisCI. As you can see from the medal table it was tough competition and wasn’t an easy decision to make.

Our Runner-up is Bitrise

As you can see, CircleCI and Nevercode were very close. Anyways, it was BuddyBuild who selected as Winner of this Olympics game. Let’s Celebrate together for BuddyBuild !!!

This is the end of Cloud iOS CI Olympics Ceremony 2017 !! Be ready for the next year Olympics.

Reference for Series of Blog Posts

Note this is repost of series of posts published on my personal blog ‘XCBlog’. The series of blog post on the cloud-based iOS CI servers Olympics are linked below

  1. Introduction to Cloud iOS CI Olympics
  2. iOS CI Olympics-Player One: BuddyBuild
  3. iOS CI Olympics-Player Two: Bitrise
  4. iOS CI Olympics-Player Three: TravisCI
  5. iOS CI Olympics-Player Four: Nevercode
  6. iOS CI Olympics-Player Five: CircleCI
  7. Announcing Winner of the 2017 Cloud iOS CI Olympics

Hope you like this post. Please let me know if you think, I miss/misunderstood something. I appreciate you feedback.