Building security into our deployment service system
Nick DeChant | Pinterest engineer, Infrastructure
We use an in-house deployment service system we built called Teletraan to push new code to Pinners around the world. Our goal for Teletraan is to be as dependable and straightforward as possible to remove any deploy-related issues for engineers. In order to achieve this, we’ve put great effort into keeping our deploy process and system secure.
Before, Teletraan didn’t have native authentication and authorization checks when performing operations, so our products and internal tools were at risk for unintended deploy-related changes across teams. As we continue to quickly grow as an organization, it’s important to have a robust permissions control mechanism to prevent possible poor experiences for Pinners and for our productivity as engineers.
Role-based permissions and access control lists
At Pinterest, deploy security permissions is based on the idea of individual team ownership. It’s important that any sort of permissions system minimally impacts the deploy process for each team, but also protects their service environments from unintended changes.
Our deploy system is a traditional 3-tier system: a front-end web UI, a MySQL backend and a RESTful service in the middle (which is also called Teletraan server). Teletraan server authenticates requests from the web UI and command-line based on three different kinds of tokens: short-lived OAuth tokens, personal long-lived OAuth tokens and Teletraan managed script tokens. Authorization is enforced by checking user roles or team roles against the requirements for a resource they’re requesting.
Teletraan is very flexible in terms of configurations for this. Turning on or off any combination of authentication and authorization is as easy as removing or adding lines in a config file.
When users access Teletraan through the web UI, the front-end client will work with our internal OAuth 2.0 server, go through the OAuth v2 workflow and obtain a short-lived token. The web UI will then use this token in HTTP headers for any subsequent calls to Teletraan server. For example, a command-line request to create a deploy with a specific token is described here:
curl -H "Authorization: token <token>" -H "Content-Type: application/json" -X POST https://<yourSite>.com/v1/envs/<environmentName>/<stageName>/deploys/?build_id=12345&description=foo
Additionally, users can obtain long-lived tokens from an OAuth server manually and use them to access Teletraan server through RESTful API calls. Both the short-lived and long-lived tokens are OAuth tokens. Teletraan server will verify these tokens with the OAuth server directly.
Teletraan server also issues and maintains script tokens, which are intended to be used by scripts. Script tokens are not OAuth tokens. Only ADMIN level users in Teletraan can create script tokens for their respective environments.
We plan to open-source Teletraan in the near future, and with that in mind, we made our deploy security model support OpenID Connect. Authenticating using your Google, Amazon or any other OpenID Connect supported login service is easily configurable and works in the same way as querying our internal OAuth servers. All it takes is changing a few endpoint URLs in a config file.
Once authenticated, Authorization checks take place against MySQL database roles tables. From command-line, authorization against request requirements are conducted using the same methods.
We created tiered levels with defined roles, and left them open to change and grow with new roles in the future. Alongside roles we designed a layer of abstraction called Resource which contains information such as an ID and a type. This allows us to apply our permissions design to other applications with different broken-up components such as hosts or groups along with our current use case of Teletraan service environments.
This design provides an effective way to protect our service environments and ensure deploy security. As we continue building our cloud platform, we intend to use this model across all our cloud services.
We’re excited to open-source Teletraan early next year, which will support the access control list model. You can read more about Teletraan as a deployment service system in a previous blog post.
Acknowledgements: Teletraan was built by Jinru He, Linda Lo, Baogang Song and Nick DeChant from the Cloud Engineering team. We’d also like to thank Devin Lundberg from the Cloud Engineering team for his help with security design.