Scaling and securing the Dropbox API
Welcome to GET PUT POST, a newsletter all about APIs. Each edition features an interview with a startup about their API and ideas for developers to build on their platform. Want the latest interviews in your inbox? Subscribe now.
For this edition, I spoke with Leah Culver, Developer Advocate at Dropbox, and Chris Varenhorst, an early author of the Dropbox API. We discussed some of the lessons learned from launching an API in 2010 and transitioning to a second major version last year. Leah co-authored the OAuth specification and shared her thoughts on growing and securing the Dropbox API. Enjoy the interview!
What is Dropbox and what can developers do with your API?
Dropbox is all your digital stuff everywhere. Secure backup of your files that you can access on mobile or desktop, anywhere you go. We also give you the ability to share and collaborate on those files with other people.
We have three products: Dropbox Personal, Dropbox Pro, and Dropbox Business. Pro gives you more space and more features like extended backups. Dropbox Business gives a team unlimited space and better collaboration features. A lot of the new features we’re working on right now are targeted at Dropbox Business.
The Dropbox API lets developers build applications that interact with Dropbox on behalf of their users. The main API (now at v2) lets you work with files in Dropbox, including features like full-text search and sharing. For example, Microsoft Office on mobile lets you save and edit files in your Dropbox. We have a separate Dropbox Business API, which is specifically for admin access features. A lot of apps are built on that for security, monitoring, and being able to look at what your company is doing on Dropbox as a whole.
Finally, we offer pre-built UI widgets for the API. There’s a security advantage (users don’t have to trust the app) and it’s really simple for the app to integrate. This is great if your users are already using Dropbox and you want them to be able to access their files.
How do you promote the APIs?
We go to a lot of hackathons of all different types. Student hackathons, business hackathons, all over the world. We also give talks on the API. We write blog posts, send newsletters, and pretty much anything we can do to get out there. We also have a Meetup group.
How do you track API adoption?
We currently process over 500 billion calls to the Dropbox API per year. That metric gives you a sense of the infrastructure, but it’s more difficult to accurately measure API value from usage.
We talk a lot about metrics and constantly debate what’s most important. Ultimately, we care about the number of Dropbox users and Dropbox Business Teams getting value out of an app. In particular, teams are already paying us and dedicated to the platform, so we want to make sure they get all the functionality they need. People and teams that install more apps tend to be heavier Dropbox users who use more space and share more files.
We currently process over 500 billion calls to the Dropbox API per year.
On an ad hoc basis, we track when we’ve been able to close a sales deal thanks to a particular app or integration. Businesses in certain industries with special requirements like finance need features that 3rd-party developers build on top of the API. In those cases, it’s priceless.
What are some cool apps built on the Dropbox API?
CloudLock is interesting. It gives Dropbox Business admins lots of control over how their organizations use sharing features, especially the security and privacy of the data. For example, you can set policies based on a team member’s role to control what shared links they can create. You can even create rules based on filetype. That’s a feature that Dropbox itself might not want to build, but partners can focus on. These features end up being pretty important to certain big customers.
A lot of apps are variations in monitoring how your employees are using Dropbox. Sometimes you need to enforce sharing policies. Sometimes you need to make sure a file that’s created is never lost for e-discovery. There are a lot of examples on our partners page.
Walk me through how the API was developed and launched.
Version zero launched in 2010. Our mobile apps are entirely built on an API. When the server-side for mobile was being built, it was intentionally done in a very generic way so that 3rd-party developers could use it. The desktop client itself is older and didn’t use this API.
There are some private APIs for things like account management and creation. But, the basics of downloading a file and listing the contents of a folder are the same API calls that the public use. Internal teams, like Dropbox Paper, dogfood new features. We have a standardized way to build API endpoints, so they can test things quickly.
How did it evolve over time?
It’s important to point out that it’s pretty tough to evolve an API at the same pace as the product surface area. In a way, Dropbox didn’t build an API until the core product was well-understood. Uploading files and listing folders were stable features that have not really changed.
If you’re adding lots of features and changing how features work in a product, users will be happy with improvements. In contrast, when developing APIs, those semantic changes can have a negative impact. They start becoming compatibility issues with legacy clients. You need to make sure API consumers are resilient to those changes.
As a result, we’re very careful about adding new API functionality. When we add API functionality, we’re committing ourselves to supporting that for a very long time.
The main idea behind v2 was to address some shortcomings in the initial design that we came up with. We learned a lot about how to structure the API and make it more scalable. Also, we did lots of housekeeping on things like deprecated fields.
We’re often working on compatibility. Anytime we deprecate something, it’s done very thoughtfully. We don’t just do it haphazardly. We need to support our 3rd-party applications and developers in what we consider to be a kind, considerate, and thoughtful way.
How do you approach monetization for the platform?
We have a symbiotic relationship with companies building on the Dropbox API. We don’t provide the API as means to make money directly, but rather to build an ecosystem that improves and extends Dropbox’s functionality. Because of this approach, our API is totally free.
Users get more apps that integrate with the service they’re already using and developers get access to more users through Dropbox. It’s a cycle that encourages more and more people to use the platform and taps into more Dropbox users for 3rd-party apps.
What are your thoughts on REST? Why did you avoid it?
There’s a blog post that goes in-depth on this. HTTP was designed for a high-level document retrieval use case. When your service fits that model, REST conventions let you leverage the wisdom that went into HTTP’s design. But trying to force REST on something that doesn’t fit can result in pain, as we learned with the previous version of our API.
For version 2, we started with the simplest thing possible — JSON in, JSON out — and added individual HTTP features if we could see a benefit. For example, we use ETag-based caching and HTTP range requests. In contrast, we largely ignore HTTP error codes and return HTTP 409 for most errors (with a more specific error identifier in the response body) instead of trying to map error cases to HTTP codes that don’t quite fit.
Externally, the reaction has been mixed. Some people are pretty against breaking from REST. While that feedback is useful, it’s important to remember that the silent majority of developers just want to get stuff done. Having good documentation and developer support is just as important as API design in the end.
This is directed at Leah. What are your thoughts on securing an API?
The biggest thing is follow the standard: use a pre-built, widely-used, open-source library. This will give you support for smart features like scopes for different levels of access. Imagine that all your APIs, even private ones, will be snooped on.
We also worry a lot about rate-limiting. There are attackers who will try to DDoS an API or try arbitrary queries to steal data. Sometimes even non-malicious clients will crawl a service too aggressively or have impolite retrying logic. When a request comes in, we first use NGINX. There’s a lot of Python on the backend and we do use Amazon SQS for queuing a few things like the webhook service.
What are some apps you’d like developers to build on Dropbox?
Any application that edits and saves files makes sense to integrate with Dropbox. Mobile integrations in particular tend to shine because there isn’t always an easily-accessible file system and a lot of file types are proprietary.
Some of the best apps center on making teams more productive. One interesting thing that Dropbox Paper and Slack both do: they call the API to preview links that you copy and paste. I’d like to see smarter previews now that more companies enable document collaboration and chat.
Developing an API (vs. a UI) brings a new set of challenges like maintaining backwards-compatibility with legacy clients. It’s interesting to think about ways to iterate quickly on an API to avoid settling on a bad design. Chris and Leah also brought up some great points about optimizing for the developer experience, not for the sake of a standard like REST.
Want more API interviews in your inbox? Subscribe now.
