Delivering proprietary libraries via Amazon S3

A step-by-step guide for Android & iOS SDK shipment

When designing complex software, there are plenty of things to think about and issues to tackle. How you are going to provide to the world what you built is often not high on a developer’s priority list. But at some point, you need to make your creation accessible to everyone.

At Streamroot, my team is in charge of our core product integration into different platforms. After perfecting our Android and iOS SDKs, it was time to decide how exactly we intended to deliver them to our customers, who, in turn, integrate them in their player apps, and how we can make it as easy as possible for them to maintain and update to the latest version. We soon realized that delivering SDKs while keeping our code private at a reasonable cost, can prove to be a complex mission. After successfully tackling that challenge, we’ve decided to share our experience: this article provides a solution for anyone providing proprietary libraries to app developers.

The challenge of closed-source projects

As soon as we started looking for hosting platforms for our SDKs, we identified an issue. Those platforms are highly adapted to the open-source world. It is a different story when your code remains private:

• Existing platforms do not provide the flexibility and control we require in our delivery solution.
• Free plans do not allow the use of a proprietary license. Our product is copyrighted. In addition, certain features, such as automatic snapshots based on your GitHub commit history, become useless when proprietary code is concerned.

Basically, whatever your reason could be, you might prefer to do this on your own and keep everything in-house, while making the whole process transparent for the customer.

A bit of context

The product we ship is a library for Android and iOS called Streamroot DNA™. Powering over 20 million video sessions daily, it enables broadcasters to better handle traffic spikes and improve the overall quality of the video for the final viewers by allowing people who are watching the same content at the same time to share portions of the video between them rather than getting them from the CDN.

The diagram below is a good summary of what happens behind the scenes.

Our customers are content owners who provide subscribers with their video player app, into which they integrate our SDK. As we support a wide range of players, platforms and devices, it was important for us to maintain a robust, scalable SDK delivery process that would still allow us to keep our libraries private.

As there is no widely-used cross-platform dependency manager for both Android and iOS, we need to comply with what exists for each platform. For Android - Gradle and for iOS, Carthage and CocoaPods. We were already delivering JavaScript libraries from our Amazon S3 storage, so we decided to continue using it for shipping our SDKs; however, the process described in the article can be applied using other storage solutions.

We started by creating a dedicated bucket to put our mobile libraries in. We preferred to have a special bucket for this as dependency managers need to list the bucket’s content and we wanted to keep only those libraries accessible for listing. Our bucket is split into two directories: Android and iOS. To make the library accessible from sdk.streamroot.io, our final step was to put our bucket on Cloudfront, which also makes distribution faster and more robust.

Shipping an Android library with Amazon S3

There are three things needed to properly deliver an Android library though Amazon S3 and make it compliant with Gradle:

• An Android library of course :) — a .jar, or in our case, the .aar library;
• A POM file which contains your library metadata;
• The right folder architecture.

As there are tons of materials on how to build an Android library available online, we will not discuss this step. I recommend the official Android documentation for a deeper dive on the subject. We will start from the second requirement, the POM file.

Create a POM file

A POM file (Project Object Model) is a file that contains a description of your library, including all the information a dependency manager might require: version, name, dependencies, etc. To start, you can simply add the following snippet in your module’s build.gradle:

The above script will generate a POM file that can be used by Gradle to download your library.

Put everything in the right place

Once we have the Android library and the POM file, we can technically put everything on our Amazon S3 storage to make it accessible. It is important to note that you’ll need to adhere to the following file organization from the root of your bucket:

your-domain-extension
|-- your-company-name
    |-- your-library-name
        |-- your-library-version

All your available versions should be stored under your-library-name. In each of those version folders, you will have to put the two following files:

• your-library-name-x.x.x.aar, or .jar, your compiled library;
• your-library-name-x.x.x.pom, the POM file we generated earlier

Let’s take as an example our main library which is called dna-core with our latest Android version — 2.1.3. The two mandatory files can be found in these paths:

• android / io / streamroot / dna / dna-core / 2.1.3 / dna-core-2.1.3.aar
• android / io / streamroot / dna / dna-core / 2.1.1 / dna-core-2.1.3.pom

For your customer

Your customer will only need to add your URL to their project’s build.gradle:

That is basically it; from now on, your customers can add the library as a dependency using Gradle like they would any other open source library:

implementation 'io.streamroot.dna:dna-core:2.1.3'

Furthermore, when updating your library, your customer will be prompted by Android Studio to get the the latest version of your library. Perfect! That is exactly what we wanted; our custom delivery is transparent for the user. Let’s do the same for our iOS library.

Shipping an iOS library with Amazon S3

While Gradle is the standard dependency manager for Android, this is not that simple for iOS. In order to be compatible with most use-cases, we had to be compliant with the two most popular dependency managers: CocoaPods and Carthage.

CocoaPods

Similarly to what we saw on Android, publishing on CocoaPods requires a file that describes our SDK, this time a podspec file which will contain all your library metadata. You can find examples of podspec files on the CocoaPods website.

As our code is not accessible from GitHub, we needed to change the source in the podspec file to make it point to the right location on our Amazon S3.

s.source = { :http => "https://sdk.streamroot.io/ios/2.2.0/StreamrootSDK.framework.zip" }

Once our podspec file is ready, we needed to push it to CocoaPods. At Streamroot, we use Fastlane, an awesome automation tool, to make our lives easier when working with iOS. Fastlane already provides what is necessary to push to production with CocoaPods.

The following pod_push lane will allow you to publish new versions of your library:

pod_push(path: "YourLibrary.podspec")

With those simple steps, we are able to see our SDK on the CocoaPods website:

For each new release, we call this lane to push our new podspec file, with the right version. Simple right? Now if our customers want to integrate our library into their Xcode project using CocoaPods, they can do it by specifying it in their Podfile:

Carthage

For Carthage, we do not need to synchronize with any platform. All we need is a JSON file to list all of our versions that exist on our Amazon S3. The file is really simple, we chose to host it at the root of our ios folder on our S3. Here’s an example our JSON file entries for our 2.2.x version:

All we need now is to provide a link to the right generated framework for each version. As you can see here, we can make our version 2.2 resolve as the latest 2.2.x (2.2.1 in the example above) which is useful when deploying patches.

Our customers only need to point to this JSON file on their Cartfile:

binary "https://sdk.streamroot.io/ios/StreamrootSDK.json" ~> 2.2.0

Et voilà! We made our SDK available from both CocoaPods and Carthage while keeping our source code private, using our framework hosted on Amazon S3.

Conclusion

As we saw, it is quite easy for everyone to publish libraries while retaining full control over the delivery. Thanks to our work on SDK delivery, our customers easily implement our latest versions, allowing hundreds of thousands of mobile viewers to enjoy the most advanced features of our product.

But good delivery is one that is built upon a robust continuous integration process. In my next article, I’ll share with you my CIDER principles (Continuous Integration Delivery Empowered Release) and how they guide our development workflow. We’ll learn how we keep agile, automate and avoid human error in our integration process here at Streamroot.