Doorman: A Tool to Offer Secure Access to Live Data for Over 250 Developers
What was the main problem?
Since Pipedrive began as a small startup, it wasn’t until the company started to grow larger that we really started to think more about security. Although developers never had an access to live customer databases, the datacenter network was still accessible from a developer machine and it was possible to execute HTTP requests directly into any live service. This process made it impossible to understand who was doing requests manually and which request were automatically executed. At one point, having (hidden) direct access to live services became unacceptable and we decided to revoke developers’ access to any live data. Nevertheless, developers still had some cases in which they needed to execute requests directly against live services (e.g. problem solving, bug-fixes, etc). To solve this issue, our team decided to start a mission to build a service access tool which would allow our developers to have access to live services, but in a secure way.
Collecting requirements
Without being able to execute HTTP requests to a service, developers still have a tool to monitor the service. Every service is able to send logs (in our case, Graylog) and service metrics are available in Grafana. However, this is not enough since there are cases where developers still need to send direct HTTP requests to the service:
- Execute a private endpoint (for example: to start a migration for the company or perform a consistency check)
- Check container health
- Debug a special case, that can’t be reproduced any other way
The requirements for a new service were the following:
- A need to be able to log every request and provide visibility
- 1 Single tool, which could be used to send requests to any data-center(DC)/Service/Container
- A handy tool with UI similar to Postman
We needed a tool which could serve as a backdoor to access live services with a UI similar to Postman, and we called this tool — ‘Doorman’.
Pipedrive’s multi data-center (DC) architecture
Before we go any further, I would like to give a short introduction to the multi DC architecture at Pipedrive. The architecture consists of several product DCs and a single management DC. Product DCs contain all the required services and data storage to serve our customers. Internally we have a separate management region where our “Backoffice” and other internal tools/services are located. The Backoffice is a (internal) tool, which allows Pipedrive employees to find companies and users, manage promo codes or start cross DC migrations. The management region can send requests to product DCs, but Product DCs will never send requests to the management region.
One of the requirements for the tool is to have a single entry point which will be able to send request to a correct DC. So, in our case, the Management DC is a perfect place to implement the doorman-api which will accept all the requests and send them to product DCs. This solution does pose a few questions, such as:
- How can it access specific containers in other DCs?
- How can we make Doorman secure enough to trust the requests?
Accessing specific containers in other DCs?
Inside a single DC, each service can easily find every other service, but by default there isn’t a way to access services from another region. We can explicitly grant a service access from a management region, but we don’t want to make all services accessible from other regions — instead we wanted to create a backdoor so that services would only be accessible from Doorman. To solve this we created a special doorman-proxy service that is deployed to every region. The doorman-proxy service is accessible from the management region and can access all other services inside the region.
Here is an illustration of our Doorman architecture:
The Doorman tool consists of the doorman-api, doorman-proxy, and doorman-db:
- Doorman-api: a service within the management region. It is responsible for authorization, authentication, logging, and proxying requests to correct regions
- Doorman-proxy: a service inside every product region which is responsible for proxying requests from doorman-api to correct containers inside the DC
- Doorman-db: a MySQL database where all access logs are stored
How can we make Doorman secure enough to trust the requests?
Each Pipedrive employee has their own LDAP credentials and Doorman uses LDAP to do the initial user authentication. Afterwards, Doorman generates a JWT token which is used to make requests to doorman-api which is also checked by the doormany-proxy. This is to make sure that requests to the doorman-proxy are only coming from the doorman-api (to prevent abuse of the doorman-proxy via unauthorized requests).
In order to make sure there is no way to bypass the system and send unauthenticated requests, we have a few restrictions in place:
- Before sending a request to region, we authenticate and authorize user
- We also write a log to doorman-db before sending a request to region
- JWT token validation inside the doorman-proxy makes sure that requests from any other service inside Product DC will not work
- Network to product DC is not directly accessible, so users can’t send requests directly to doorman-proxy
Now that we know it’s only possible to send a request through a single place in management region, we have a perfect place to log requests. To spin up some storage we use a MySQL database.
Doorman as a useful Tool
Doorman, as a tool for our developers, has 2 main features.
1 — A Postman-like UI that can execute most of the requests.
We, however, did not want to restrict our developers. With the JWT token that is generated, our developers are able to execute requests to doorman-api with their tool of preference, such as curl or Postman. Developers can then do even more using their own tools (like uploading files or writing own scripts).
2 — An Audit log where one can see all the requests executed by developers. This page provides visibility for security team and also has a convenient way to retry previous requests if needed.
3 — A tool to send requests in bulk.
One of the use cases with this tool was to execute endpoints and provide company ids, such as migrating a company from one state to another. Instead of doing those requests manually by developers, Doorman is capable of sending requests in bulk, for specific set of companies. This makes things like gradual rollouts much easier.
Conclusion
Having non-secure access to live data is clearly not ideal, yet it’s not a rare phenomena because developers need to have access to live data and it’s usually a difficult and time-consuming project to implement a fully secure solution. In Pipedrive, security is something we can’t sacrifice, so creating a secure method to access live data was paramount and luckily the solution we came up with turned out to be simple, secure and handy.
