Square APIs

Square’s Connect API reads a large number of internal APIs and makes them available. See how we manage our integrations and learn about newly open sourced Pilgrim and Protob.

Written by Daniel Neighman.

At Square we love building great APIs. Internally, we’ve created an extensive SOA with many services. On top of the SOA, we’ve built the Connect platform to expose the power of our internal operations. We’ve open sourced two of our tools today: Pilgrimand Protob. More in “Our Tooling” and “Part Two” sections below.

Part One: Managing our internal APIs

When we decided that we wanted a platform API, we had a few things to consider. The platform must be:

  1. JSON format API
  2. Easily consumable
  3. Consistent
  4. Maintainable
  5. Fit in with our existing infrastructure

These are pretty standard goals, but what is interesting is the way we went about guaranteeing that our system had these properties. Our API is a facade service that accepts requests, translates the requests, and passes them through to our internal services. Lastly, we translate and return the results. Again, the approach is pretty standard, but Square has hundreds of services in many languages that we need to interface with.

In order to develop effectively in this environment, Square describes all of our internal APIs (except some legacy ones) with Protocol Buffers. There are plenty of good reasons for describing your APIs this way, especially if your communications relies on them. There are other tangible benefits though:

  • Strong API contracts with the schema enforced by code
  • Reasonably easy to follow
  • Decoupled API definitions and implementations

Strong API Contracts

Strong API contracts are really important. We have servers and clients written in Ruby, Javascript, Go, Objective C, Python, and Java. With so many APIs being consumed by typed and untyped languages, having strong API contracts that are described means that all of our API providers and consumers are mostly isolated from implementation details. The fact that we use Protobufs means we not only have descriptions of our APIs for documentation, but the code actually executes according to the same definition. No documentation rot!

From day one, our Connect API was defined first in Protocol buffers and then translated into an HTTP JSON API. This might sound a little heavy handed to many people who work with dynamic languages, but it’s helped us implement the consistent API that we were after. It gave us the ability to scrutinize, critique, and enforce convention within the API in isolation from implementation. No code was written at this point, which meant that once we had our definitions, everyone who had to participate could get started. For test driven fans, we could now confidently mock our request/response objects and teams could get on with the job of building implementation in parallel. There are many other ways protocol buffers are valuable to our team when developing our API; however, protocol buffers are most beneficial in reducing the overall cost of the API.

API Costs

All APIs cost something. As providers we know they cost developer time, servers, etc. One part of the cost equation — which is not considered enough — is the consumption cost. When we build an API, we’re providing value (we hope); but we’re also asking everyone who wants to integrate with us to bear some cost as well. We require each consumer to pay this cost for each client. If the value gained from using the API is lower than the cost of integration, it’s not worth consuming. There are many ways that we can offset this cost like providing SDKs, libraries, documentation, etc.

One of the costs of consuming a JSON API — which is potentially underrated by API providers — is type safety. Wait. What? … Yes, that’s right. There are many languages out there that are type safe. Even if you don’t care to use them by providing an API, my guess is that you want to reach a broad audience and you have to care for their sakes. In type safe languages like Go and Java, dealing with JSON APIs can be a real headache. Polymorphic data structures and using dynamic keys make describing the API for consumption more difficult. Imagine trying to define your API using strict typed structs, as in the case of Go. The fact there is the mapping required for these languages is already a cost for your API; if you make that more difficult by getting clever with dynamic and polymorphic data structures then the cost escalates. It is not hard to build an API that is friendly to typed languages, but it is easy to accidentally make it hard for them.

Our Tooling (and introducing Protob and Pilgrim)

So we have all our APIs defined in protobufs, and we potentially talk to over 200 services. We also decided to use Node to act as our server so we needed to find tools to make it all work. Unfortunately at the time, the protocol buffer support in Node was very limited for using protos — which utilized advanced features such as extensions and options. We couldn’t even compile our protos for the most part because these libraries used hand rolled parsers for their raw proto files. One example of a case we needed to handle is having extension names that match, but had different tag ids. It’s a valid proto definition, but we had no way to use these masked extensions.

Unfortunately this meant we had to roll our own, but we were able to borrow ideas from the existing community along with other language implementations. What we ended up with was a new Protocol Buffer library for Node, Protob. Protob uses protoc (the official parser) and outputs JSON. At runtime, it loads this JSON and creates constructors for the messages, services, and enums. This also includes extensions and options with full reflection available.

We also needed a way to keep on top of these APIs. Chasing down the individual proto files and following import statements wasn’t very appealing. So, we decided to extract the documentation from our definitions into an easy to digest form. We developed a single page app called Pilgrim to help us navigate our APIs across the 200 some services. Today I’m excited to announce that we have open sourced it! You can take a look at how Pilgrim describes our API with the free hosted version. Go ahead, click it! I’ll wait.

You can see that Pilgrim opens up our protos in a navigable way, we have our “Services” that define RPC methods with input and output types. Using Pilgrim, we’re able to navigate our large collection of APIs with relative ease. We also use just about every proto trick there is (and having Pilgrim expose extensions is a huge boon). We can even checkout the registered extensions on protos like google.protobuf.FieldOptions. No more wondering what extensions are available.

You can find our raw proto files at https://github.com/square/connect-protos.git.

Part Two: Pilgrim.

The awesome news is, Pilgrim is not just for Square’s APIs. You can use it to view your own APIs defined in protocol buffers. Pilgrimize is a small tool to let you define your dependencies from git or local file systems. It will compile them and run a local server that pilgrim is aware of. As an example, if you wanted to do a Square/Marvel mashup for your own application, just define a protos.json file:

[
{ "git": "https://github.com/hassox/fender.git"},
{ "git": "https://github.com/hassox/public-protos.git"},
{ "git": "https://github.com/hassox/google-protos.git"},
{ "git": "https://github.com/square/connect-protos.git"},
{ "git": "https://github.com/hassox/marvel-protos.git"}
]

Save this into a protos.json file and in the same directory run:

$> npm install -g pilgrimize
$> pilgrimize

Now if you visit Pilgrim you’ll notice that you have both Square and Marvel services. Nothing is uploaded to the Pilgrim server. Pilgrimize runs a localhost server with CORS headers to serve your protos, Pilgrim just happens to know to look there.

Using Pilgrim and Pilgrimize, we’ve been able to define our API, grab it, and browse it, as well as make use of the type checking in Protob for our API — to ensure we’re keeping our end of the contract.

While I was on paternity leave, I started exploring what it would look like to describe an existing JSON API with protos so we could reduce the costs of consumption and potentially make a client that knows how to operate based on it. The result was fender. By itself it’s not very exciting. It’s a collection of extensions that allow you to mark up your proto rpc definitions with information on how to call them via REST. It allows Pilgrim to support exploring your API from inside the Pilgrim interface. Let’s try it. If you haven’t created a Square application, head over to Square Connect and create one.

We wanted to make sure that it worked for more than just Square though. Since we defined our APIs with protos anyway. What would it look like to retrofit to otherexisting APIs? Pretty cool huh!

We’ve found this approach to be very useful. Providing type safety on our service boundaries makes it really easy to consume APIs. A little bit of type safety at the boundaries goes a long way with clients. Hopefully by open sourcing Pilgrim and Protob your team will be able to make use of it to document, implement, and explore your own APIs.