Building the Brex Security & Privacy Center

The Product Security Engineering team at Brex is excited to introduce our first iteration of the Security & Privacy Center, launched in May 2023. Our goal is to give Brex customers all of the tools they need to effectively monitor and secure their Brex accounts. This product feature is the first step in our long-term vision to provide more visibility and transparency for Brex customers on the security posture of their Brex account.

This post details the features included in the first iteration of the Brex Security & Privacy Center, along with some interesting technical challenges and unexpected internal benefits that we noticed during development.

What is Product Security at Brex?

We are a team of engineers dedicated to anything and everything related to customer security. We own features like Single Sign-On (SSO), authentication, and authorization. We are uniquely positioned as a team within the security organization that primarily contributes customer-facing code. This gives us the opportunity to positively impact the security of our Brex product while working directly with other engineering teams who are affected by new security initiatives.

The Security & Privacy Center

Inception and First Iteration

The Security & Privacy Center was first conceived during our annual company hackathon in April 2022 and featured a small subset of the features we support now. It garnered much excitement from others at Brex and would go on to win the employees’ choice award. Due to its perceived impact and relevance for customers, we were able to turn this into a full-fledged security feature within the product.

In addition to laying the groundwork for the initial set of features that we support today, we started long-term planning for our vision of what kinds of security features we wanted to support in the Brex dashboard.

We decided that the first iteration of the Security & Privacy Center would support:

• Displaying and revoking mobile and dashboard sessions.
• Displaying third-party access to a user’s account (like Slack and 1Password extensions).
• Migrating the existing 2FA, Browser Cookie, and Password Reset settings to the new view.
• Displaying and revoking email-verified devices.
• Displaying a history of login with location metadata.

We felt that we needed to start by supporting the features that many people have come to expect from their SaaS applications. While these features are (usually) considered standard for services from post-IPO financial companies, younger companies like Brex often don’t invest the engineering resources in security-focused features like this, so we were excited to take this next step to advancing Brex’s security maturity.

Additionally, this new view would remove clutter from the general settings page and make it easy for users to manage their security settings and make all future security features more discoverable.

First Steps and Early Mistakes

Development began on the Security & Privacy Center in January 2023 with planning, backend architecture, and frontend design. During this phase, we partnered with our UX team to finalize the UX and make decisions about how things would be displayed in the frontend. This provided valuable insight into understanding how users would interact with the product and how we could limit challenges and unnecessary learning curves for those using the Security & Privacy Center for the first time.

We spent some time with the UX team to iterate through a few design options:

Before landing on a final design:

Our primary UX goal was to make the first iteration of the Security & Privacy Center UI as approachable as possible. We kept things streamlined while still providing valuable information, and decided to reserve some of the less readable information like user agents and other device data. In future iterations, we will provide power users with the greater depth and density of information needed to get a more granular view of account security for the users they manage.

For the backend, we started designing the internal and external APIs that would serve the new content. We designed GraphQL APIs to serve the data to the frontend and added new gRPC endpoints to our internal microservices to provide the data. While most of the backend design work was fairly straightforward, we ended up underestimating the work required to display user sessions, which was our first mistake.

Brex’s authorization backend is partially stateless using bearer JWTs, with a proxy layer that allows us to revoke tokens at-will if needed. Previously, session revocation was a task that only internal administrators had access to, and we could only revoke sessions across the whole product for a given user, not individually per user device. Adding this feature to the Security & Privacy Center would thus prove challenging once development began.

During the design phase, we stuck to only designing the API schemas to expose session data and recorded a high-level idea of how we would solve the session issue. Looking back, we should have recognized this complexity earlier and spent the time to do a more thorough design before continuing on to implementation. Instead, for the sake of time, we decided that these challenges would be solved when it was time to start developing.

The rest of the backend design went smoothly, and the level of complexity was well understood. We went on to finalize the backend GraphQL and gRPC APIs, reviewed and approved the design, and were ready to proceed with implementation.

Development

Two-factor authentication (2FA) management, browser cookie settings, and password management already existed in the application, so we migrated these to their new home in the frontend to match the new design with minimal backend changes.

From here, we had to build some new systems to collect and present user security data, which posed some interesting challenges and gave us the opportunity to deep-dive into how we could improve our existing systems.

Much of this functionality either did not exist yet, or existed in a state that only internal staff could make sense of the data. Not only did we need to build the systems to access and display this sensitive information securely, we needed to ensure a high degree of data integrity and accuracy before we could make it externally visible. This was a unique challenge and required us to rethink many long-standing processes.

Trusted Device Management

Brex requires that all customers who have not enabled SSO verify logins from new devices with their registered email address. Once you verify a device, it is stored for future use as a “Trusted Device”. We also wanted to enable customers to revoke this trust at any time — if, for example, a laptop is stolen or otherwise compromised, customers should be able to revoke this trust and view a complete list of their trusted devices. To do this, we simply needed to develop new backend endpoints to pull this existing data securely and deploy the associated GraphQL resolvers. It was important to us that customers be able to do this from the unified Security & Privacy Center for ease and velocity.

Third-party Connection Management

We used our existing OAuth 2.0 infrastructure to support manually revoking some third-party connections from the Brex application. While we already have a dedicated page for third-party integrations, we wanted to give users additional access to any third-party application that we support OAuth connections with. This was fairly straightforward to implement as we simply needed to revoke the third party’s OAuth grant.

We ran into some interesting UX challenges here. If an integration is configured through the dedicated integrations page or affected more than just a single user (such as an ERP or HRIS integration), we did not want to support revoking these without going through the integration’s specialized disable flow. After some deliberation, we decided to simply hide these unique integrations from this page and only show those where there was no existing user-facing option to revoke.

Session Management

The idea of presenting users with a list of their current login sessions and allowing them to revoke sessions individually seems simple on the surface, and would be trivial to implement if your application uses traditional HTTP sessions. However, with stateless bearer tokens, this problem becomes much more complex.

Specifically, Brex uses JWTs issued by an Okta OAuth 2.0 server to authorize requests from external users. This means that there is no implicit storage of what tokens were issued to whom, when, and on what device. In the past, if a user needed to have a session revoked, we would simply revoke all tokens issued to that user for all OAuth clients.

When a user authenticates with Okta, they are issued the following:

• An Access Token: These are cryptographically signed and are used by the application to authorize requests, these are used as the Bearer token in our GraphQL calls.
• A Refresh Token: These are used to receive new access tokens after the old one expires. Okta-granted refresh tokens are unique in that they have an ID that persists between token rotations in implementations where you have RTR (refresh token rotation) enabled.
• A Session ID: If your application is using JWTs for authorization, these have no value aside from being able to retrieve device details from the event logs for the corresponding user.session.start event. This is a critical piece if you’re trying to display device details to the user for a given login.

Together, these paint a picture of who logged in from where using what device, but they aren’t implicitly related.

Theoretically, we could paint this picture ourselves using API calls to Okta, but Okta also presents us with the following limitations:

• No API to list Access tokens, meaning that we cannot get a snapshot of what tokens are currently being used to authorize requests.
• No API to list sessions, so sessions can only be captured from the user’s local browser, or from event logs in Okta.

In order to fill these gaps, we upgraded our authentication flows to gather more context from these different systems at the same time and generate a complete picture of what tokens and session metadata are associated.

We needed to make some significant changes to a few backend components. First, we updated our token issuing service to provide additional identifying information, which we use to better track sessions. We also added PostgreSQL tables to track each component individually, as well as a new table to track session-backed devices. Lastly, we created new internal endpoints to write and update session metadata in response to user activities. This ended up taking multiple weeks to complete, and an entirely new design document needed to be written to fully capture all of the details and edge cases.

With these changes, we now treat JWTs more like session tokens. We do this by using a proxy service to inspect the bearer token from each request against our Okta-backed cache. This proxy service also allows us to complete the full picture of the sessions by collecting additional details that we may have missed during provisioning.

Due to the sensitive nature of login sessions, we have multiple places where we can asynchronously complete any session record if one of the synchronous processes fails. We wanted to ensure that users are getting the full picture, and our infrastructure allows us to deploy this extra code without impacting performance.

This additional complexity forced us to push back some of our timelines, but in the end we ended up with much more robust session and device tracking that has already been useful to some ongoing internal work.

Account Activity History

In our current version of SPC, we support viewing login events for mobile and desktop applications, and we can display some geolocation and device information. Our long-term goal is to provide an extensive view of account activities to Brex users and account administrators, and we are actively working on big changes to our account activity logging systems to get them ready to serve requests from Brex customers.

Lessons Learned

Developing the Security & Privacy Center was a great learning opportunity for us in the Brex security organization, as it gave us hands-on experience with parts of the product lifecycle that we typically don’t have. It also gave us excellent insight into how our application security controls impact the development of a new user-facing feature.

It is important for security teams to collaborate with their internal customers and to see how the requirements and controls they deploy affect the development of new features. Additionally, we believe that building transparency and trust into your product is crucial to delivering a great experience to your customers.

We are excited to iterate on the Security & Privacy Center and deliver more security-focused features to our customers.