Olympics of Top 5 Cloud iOS Continuous Integration Servers

Shashikant Jagtap
Sep 28, 2017 · 11 min read
Image for post
Image for post

a.k.a CI for iOS apps is one of my favourite topics. Continuous Integration is essential for every iOS project now as the number of developers growing in the iOS teams. As of now, I have deeply experienced with self-hosted CI servers like Xcode Server, TeamCity, and Jenkins for iOS projects. However, there are still an array of cloud-based CI services available in the market which are becoming very popular in the recent days. There are a lot of articles and tweets going on for those cloud-based CI services. I am keen to try those services and get some exposure by using them with my favourite personal project . In order to do this, I have decided to play an Olympics game involving the most of the cloud-based iOS Continuous Integration services candidates.

Purpose

The most of the services taking part in this competition that I haven’t used before apart from TravisCI. It would be the good opportunity for me to make myself familiar with those CI services and compare them using my personal project . The purpose of this Olympics competition is to share my experiences and first impressions on those cloud-based iOS CI services. Please note that purpose of this competition is not to recommend particular service that you should use for your iOS project.

The whole purpose of this competition is to try and share the first impressions of these cloud-based iOS CI services.

Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of anyone else in this world. This evaluation is only performed against project and it may vary from project to project.

Olympic Players

The player involved in this Olympics games are the most used cloud-based CI servers which supports iOS and macOS platforms. The current list has these players

XCFit: The Game

This Olympic competition will be played against the project , the library we are going to use to try out the features of CI services mentioned above. This is one of my favourite pet project which is the small library that allows us to write a in Behaviour Driven Development a.k.a fashion. This project is interesting candidate for CI server test as it consists of

  • Swift Framework distributed with CocoaPods and Carthage
  • Swift Package that can be distributed with Swift Package Manager
  • An Example app using framework built with Carthage
  • Ruby package distributed using RubyGems to download Xcode templates

As there is a lot involved in this project so it would be interesting to see how all those cloud-based CI servers treat this project to run the builds on respective platform.

Image for post
Image for post

Olympic Athletics

The Olympic athletics, we will be using for this competition are

  • Sign Up & Project Setup
  • Default Build Configuration
  • Build Execution
  • Build Artefacts & Test reporting
  • Real Device Testing
  • Build Customisation
  • Parallelisation & Concurrent Builds
  • Build Pipelining
  • Infrastructure As a Code
  • Directory Caching: CocoaPods & Carthage
  • Code Signing & Distribution
  • Third Party Integrations
  • Speed
  • Secret Store
  • Xcode & Apple Developer Tools Upgrades
  • Open-Source Project Support
  • Learning Curve
  • Documentation & Support
  • Clean UI/UX
  • Price

The players will be evaluated on basis of these games. Each of the athletics will earn a medal that would be gold, silver or bronze. At the end, there will be pros and cons section where we will discuss the outcome of this evaluation. This will give the brief idea about the functionality of the cloud-based iOS CI servers and their features related to iOS development.

20 Olympic Medals

Each of the game mentioned above will earn a medal. The medals we are going offer are Gold, Silver and Bronze.

  • Gold Medal

The player will get a gold medal for the criteria if it is strong to satisfy user and met requirements clearly as well as exceptionally.

Image for post
Image for post
  • Silver Medal

The player will get the Silver medal for the specific criteria if it is performed alright and satisfy user up to the certain extent.

Image for post
Image for post
  • Bronze Medal

The player will get a Bronze medal if it’s met the minimum requirements but doesn’t satisfy user that much. This is the least medal that every athletics will get for the performance.

Image for post
Image for post

Each player will get 20 medals and the winner will be decided on the basis of the number of Gold medals. There is no good or bad CI services but the winner will be announced based on how each cloud-based CI servers will perform against project only. There might be different winners for different projects.

5 Magic Medals: At the end

There are 20 different games in the Olympics and each player will get a medal for each game. This means every player will get 20 medals for sure but there will be additional 5 magic medal that will be offered to the player while the evaluation is going on and announced once all players are evaluated. The 5 medals will be based on following criteria

  • Social Media support during evaluation
  • Correcting my observations during evaluation
  • Accepting facts and respecting competitors
  • Showing courage to improve services
  • User Feedback on Twitter & Public API

BuddyBuild : Player One

In the iOS Continuous Integration Server Olympics competition, the first candidate selected is . The most heard name on Twitter and other tech blogging site, because of its simplicity and ease of usage. is designed specifically for the mobile continuous integration which supports iOS and Android platforms. The clients of include some big brands like The Washington Post, FireFox, Slack, Meetup etc.

Olympic Performance of the BuddyBuild

I have done the critical evaluation of BuddyBuild for each of the 20 criteria and the results are shared in the blog post

At the end of 20 Games BuddyBuild Scored :

13 GOLD 🥇

6 SILVER 🥈

1 BRONZE 🥉

BuddyBuild— XCFit Build Logs

The build logs executed for the project are available . We can navigate to details, logs and test tables to get the more information about the build.

Bitrise : Player Two

In the iOS Continuous Service Olympics competition, the second candidate selected is . There are many articles on medium and lot of good words on Twitter about the Bitrise. Like , the is also designed for the mobile continuous integration which supports iOS and Android platforms. Bitrise also got some good clients like ABB, Foursquare, FOX, TNT etc.

Olympic Performance of the Bitrise

I have done the critical evaluation of Bitrise for each of the 20 criteria and the results are shared in the blog post

At the end of 20 Games Bitrise Scored :

13 GOLD 🥇

7 SILVER 🥈

0 BRONZE 🥉

Bitrise — XCFit Build Logs

The build logs executed for the project are available but you need to have an account to access those logs.

TravisCI: Player Three

In the iOS Continuous Integration Server Olympics competition, the third candidate selected is , the most used cloud-based continuous integration for open-source and private projects. It’s truly the home of the open-source testing. As mentioned on the TravisCI page there are about 900K open-source projects and 600K users running the continuous integration on TravisCI. It’s deeply integrated with and setting up an open-source project on TravisCI is a minute away from developers.

Olympic Performance of the TravisCI

I have done the critical evaluation of TravisCI for each of the 20 criteria and the results are shared in the blog post

At the end of 20 Games TravisCI Scored :

12 GOLD 🥇

5 SILVER 🥈

3 BRONZE 🥉

TravisCI — XCFit Build Logs

The Travis build logs executed for the project are available .

Nevercode: Player Four

In the iOS Continuous Integration Server Olympics competition, the fourth candidate selected is . It is designed specifically for the mobile continuous integration which supports iOS and Android platforms. The clients of include some big brands like Sainsbury’s, Toyota, Eurostar etc.

Olympic Performance of the Nevercode

I have done the critical evaluation of Nevercode for each of the 20 criteria and the results are shared in the blog post

At the end of 20 Games Nevercode Scored :

12 GOLD 🥇

5 SILVER 🥈

3 BRONZE 🥉

Nevercode— XCFit Build Logs

The build logs executed for the project are available but you need to be logged into Nevercode portal.

CircleCI: Player Five

In the iOS Continuous Integration Server Olympics competition, the third candidate selected is , the continuous integration service that is being used by the most popular tech firms like Facebook, Spotify, Percolate etc. The CircleCI and TravisCI have most of the things common but some features are unique to the each platform.

Olympic Performance of the CircleCI

I have done the critical evaluation of CircleCI for each of the 20 criteria and the results are shared in the blog post

At the end of 20 Games CircleCI Scored :

12 GOLD 🥇

7 SILVER 🥈

1 BRONZE 🥉

CircleCI — XCFit Build Logs

The CircleCI build logs executed for the project are available.

Olympic Medal Table for 20 Games

So far we have done with 20 games and medal table looks like this:

Image for post
Image for post

Magic Medal Stage

As of now, we can see that, there is a tough fight between Bitrise and BuddyBuild at this moment but other players like TravisCI, CircleCI and Nevercode are too close. However, the game hasn’t finished yet. We still have 5 magic medals to be awarded to each player according to the following criteria

  • Social Media support during evaluation
  • Correcting my observations during evaluation
  • Accepting facts and respecting competitors
  • Showing courage to improve services
  • User Feedback on Twitter & Public API

A medal will be awarded for each of the criteria. This means every player will get 25 medals and winners will be the player who won maximum GOLD medals. If there are equal GOLD medal then we will look for the number of SILVER medals.

Magic Medals Awards

Now that, we need to present magic medals awards for each of the criteria mentioned above.

BB = BuddyBuild
BR = Bitrise
TC = TravisCI
NC = Nevercode
CC = CircleCI

Social Media Support(SMS)

There were huge supports on Social Media during the evaluation process, especially on the Twitter. The tweets were liked replied and retweeted during the evaluation. The example Tweet can be found . BudduBuild and Nevercode were top on Social Media support. CircleCI joined late but there wasn’t any interest shown by Bitrise and TravisCI. The medals for the SMS a

Image for post
Image for post

Correcting Observations(CO)

There were some features that I haven’t discovered in the first place but the players corrected me with right features by communicating with me on Twitter, Email and Blog comments. Nevercode, BuddyBuild and Bitrise contacted me and corrected most of my observations. The medals for the correcting observations are

Image for post
Image for post

Accepting Facts and Respecting Competitor

There are some conversations with Twitter and emails on accepting the facts and respecting the competitors work. BuddyBuild and Nevercode are good at it. The medals for the accepting facts are

Image for post
Image for post

Showing courage to Improve Services

Nevercode and BuddyBuild have shown strong desire to improve their services by email and with the tweet. The medals for showing courage to improve services are

Image for post
Image for post

User Feedback on Twitter & Public API

I have collected some user feedback on Twitter and also public APIs like the status badge and public logs for the open source projects. BudduBuild, TravisCI and CircleCI have both good feedback on Twiter and Public logs and APIs. Bitrise and Nevercode lack public logs and some Twitter users reported issues with Xcode 9 upgrade for Bitrise and Nevercode still has failing build for me with Xcode 9. The medals for user feedback and public APIs are

Image for post
Image for post

Final Medal Table & Winner

Now that, we have awarded all the magic medals to all players. It’s time to declare the winner of the competition.

Image for post
Image for post

The final Medal table:

Image for post
Image for post

The Winner is ……………..

Image for post
Image for post

Congratulations, BuddyBuild !!

I would like to thank all other players including Bitrise, Nevercode, CircleCI and TravisCI. As you can see from the medal table it was tough competition and wasn’t an easy decision to make.

Our Runner-up is Bitrise

As you can see, CircleCI and Nevercode were very close. Anyways, it was BuddyBuild who selected as Winner of this Olympics game. Let’s Celebrate together for BuddyBuild !!!

Image for post
Image for post

This is the end of Cloud iOS CI Olympics Ceremony 2017 !! Be ready for the next year Olympics.

Reference for Series of Blog Posts

Note this is repost of series of posts published on my personal blog ‘’. The series of blog post on the cloud-based iOS CI servers Olympics are linked below

Hope you like this post. Please let me know if you think, I miss/misunderstood something. I appreciate you feedback.

Like this post from XCBlog By ? You may also like some of our services like guest blogging or Mobile DevOps(CI/CD) or Test Automation. Chekout our s, open source projects on or Follow us on , , , . Download Our iOS App to read the blogs offline.

Image for post
Image for post
XCTEQ Limited: Mobile DevOps, CI/CD and Automation

X is a company specialised in Mobile DevOps, CI/CD, Mobile, AI/ML based Test Automation Checkout XCTEQ products and services at or write to us on info@xcteq.co.uk..

XCBlog

P.S:

Shashikant Jagtap

Written by

All the posts published on this channel before I joined Apple. Thanks for being a reader of XCBlog. Web: shashikantjatap.net, xcteq.co.uk

XCBlog

XCBlog

P.S: All the posts from this publication are published before author Shashikant Jagtap joined Apple. Thank you so much for being a reader of XCBlog. Tech Blog about DevOps, CI/CD for  Platforms. Xcode Server, XCTest, SwiftPM, Server Side Swift, XCUITest etc.

Shashikant Jagtap

Written by

All the posts published on this channel before I joined Apple. Thanks for being a reader of XCBlog. Web: shashikantjatap.net, xcteq.co.uk

XCBlog

XCBlog

P.S: All the posts from this publication are published before author Shashikant Jagtap joined Apple. Thank you so much for being a reader of XCBlog. Tech Blog about DevOps, CI/CD for  Platforms. Xcode Server, XCTest, SwiftPM, Server Side Swift, XCUITest etc.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app