Enabling Agent Collaboration

Compass for Teams provides customers the transparency and organization to collaborate and boost productivity

Authors: Jim Ballew, Toktam Rezaei, Kenneth Bergquist, Shay Artzi

Empowering the agent to collaborate with best-in-class team functionality, within their direct team (and more aspirationally through their indirect team), within the same platform as their CRM, from lead to contact to close, passing off responsibilities across different roles within the team throughout the transaction… that will be the company that transforms real estate.

And that’s going to be Compass.

— Robert Reffkin, Founder & CEO

The customers of the Compass platform are real estate agents, and the platform succeeds by empowering this community to grow their business rapidly. Managing a real estate business requires a variety of activities to convert leads, serve clients, and nurture relationships. As agents grow their businesses, they build a team of additional agents, assistants, and other support staff to collaborate and divide work to serve their clients better. This capacity growth is critical to their productivity growth.

As collaboration increases, transparency and communication are vital to coordinate efforts effectively. Agent teams have historically relied on various collaboration tools, from physical whiteboards and paper printouts to a wide selection of disparate technology solutions to support different functions. The need to provide a consolidated, integrated suite of collaboration tools for agents existed long before the COVID pandemic and has become critical with the shift from in-office to largely remote work.

The Compass platform provides agent teams with a broad array of products and features across dozens of service domains: clients, transactions, tasks, communications, documents, and many more. To allow agents to collaborate, we had to create a seamless experience that enabled them to use multiple services without understanding — or even knowing the existence of — the permission models within each of those services. Sharing had to be simple, intuitive, and consistent across the platform.

Here’s an example to illustrate.

In the example, Agent Erin is working with Client Bob to sell his property. Erin wants to schedule an Open House and brings in her teammate, Martyna, to collaborate. To do so, Erin shares the Business Tracker Folder with Martyna, granting her access to the workspace. Additionally, Erin creates a task and assigns it to Martyna to follow up on the Open House with Bob. From one surface in the Compass platform, Erin takes actions that seamlessly grant access to 3 resources — Folder, Client, and Task — across different service domains. In the past, Erin would have had to share these resources from three different tools, and her teammates would have had to know to access them from those tools. This brings everything to one place, saving the team valuable time.

With more than 25,000 agents in the Compass universe, we learned that agent teams work in different collaboration models. Smaller teams often follow a “share everything” approach, while larger teams may have subsets of agents working in different states of collaboration. Our solution has to have the flexibility to support varying approaches to collaboration while always maintaining an intuitive experience for users. As one of our Principal Engineers put it, “it’s easy to make complex and it’s complex to make easy.”

We followed some key guiding principles in our solutions:

• Agents’ customer data is their own — Agents are free to take their client data with them if they leave Compass.
• Customer agency — Sharing is a result of direct action taken by an agent to provide explicit or implicit access.
• Sharing is simple and consistent — We provide an intuitive experience that is consistent across the Compass platform.

Technical Challenges

Engineering teams at Compass have historically implemented their own permissions for resources in the services they own. Feature squads used different methods of permissioning, including embedded ACLs or tracking userIds on each object, resulting in ad-hoc permission models that work in different (and often conflicting) ways across the platform. As a result, any approach to cross-service sharing required bespoke implementation, which was likely to be inefficient and brittle given the ongoing evolution of Compass services. To enable collaboration across service-distributed domains, we needed a long-term, sustainable solution that provided a standard model for resource permissions and observability into permission changes across the entire platform.

• Consistency: A standard model for representing permissions within Compass, for development teams and end-users alike.
• Guaranteed Ordering: Permission changes must be consumed by services in the order that they were issued by the user to ensure data integrity.
• Scalability: Support 100s of millions of events per month with the expectation to grow along with Compass services.
• Performance: Permission changes need to propagate across distributed services quickly and reliably.

The need for scalability and performance was especially crucial. You can assume that every single API request starts by checking that the person requesting access to a certain resource does in fact have permission to access it. These checks need to be quick so that we do not slow down all of our upstream services. At the same time, if the check fails, permission cannot be assumed which would essentially cause the upstream service to error out or return an incomplete response. Availability is key. Our authorization services need to have the fastest and most reliable read responses on our platform. That is not easy!

Technical Solution

After synthesizing the requirements and tenets on both the product and the engineering sides, we began building and iterating on our vision. What we came up with is a developer-friendly set of Authorization (AuthZ) APIs designed to address a broad scope of resourcing and permissioning use-cases needed to facilitate collaboration on our platform — and we empowered our product and feature teams to #movefast to add them. These AuthZ services centralize the Compass platform’s permissions at the resource, team, and application level. They aggregate responses and provide a variety of options for downstream services to consume them. In short, they provide a small handful of simple API calls anyone can use to answer the question, “What capabilities does User X have on Resource Y?”

Each service addresses a distinct collaboration pattern:

• Resource: Manage the capabilities that subjects (users, teams) have on specific resources.
• Wildcard: Assign permissions to all resources of a particular type scoped to a user or team.
• Inheritance: Enable parent/child attachments between resources, and manage inherited permission for the child resource based on parent capabilities.
• Application: Empower team leaders to enable features and application settings for their team members.

Back-end services can query permissions through direct gRPC requests or via a set of SDKs developed for our most commonly-used languages.

On the front end, a universal set of sharing components allows drop-in compatibility across apps at Compass and ensures a consistent sharing and permission management experience across the platform. This means users will go through the same exact flow when sharing a contact in the CRM as they would when sharing a Listing, Collection, or any other resource in another app. (Think about the flow when you click “Share” in Google Docs, Sheets, or Slides. It’s always the same.) This will reduce cognitive load, increase productivity, and build confidence and trust in our platform.

Our most challenging hurdle came when we started to get pushback from other engineering teams. The main response was, “We’re not going to hit your services every single time we need to check the permissions for one of the millions of resources in our database. It’ll kill our app.”

They’re not wrong. We realized that we needed to let teams localize permissions so their checks can be as simple as a join in a SQL query. But the data had to be up to date within a matter of seconds. We needed a way to let services consume permission changes in a scalable manner across our distributed architecture. We chose to take an event-driven approach built on Apache Kafka due to its proven ability to scale, high reliability, and ordering guarantees. Any time a permission is updated, a corresponding Kafka event is published. Consuming services can use these events as the source of truth to drive permissioning in their applications.

In the first implementation of event publishing, a collaboration service would update its permission state and publish to Kafka synchronously in a database transaction. This provided a level of guarantee that any update to the permission database would result in a corresponding Kafka event. This initial approach allowed for fast development and enabled the first consuming services to begin onboarding to our new AuthZ services. The tradeoff of moving fast to deliver this solution was additional client latency due to the need to guarantee the database commit and Kafka event publishing in a transaction. We’re talking about 100ms added onto any WRITE. Additionally, a service disruption in Kafka would prevent permission changes in the database table, reducing availability of the authorization services and putting into question the accuracy of any permission in the database. These are deal-breakers!

Our first improvement aimed to reduce client latency and increase service availability while still guaranteeing that the event state would always reflect the internal permission state. We chose to implement an “Outbox” pattern for event publishing. The Outbox pattern allows for both atomic consistency and lower latency when requesting updates. It involves two parts:

1. A database transaction that will commit both the desired data change and insert a new row in a separate table with the event that must be sent to Kafka.
2. A separate Kafka Connector process that will asynchronously read contents from this dedicated outbox table and produce the events into Kafka.

This pattern allows for data consistency between the two parallel sources of truth and reduced latency on the producer side. After implementing this pattern, we saw our write latencies reduced by 90% (100ms to 10ms) while maintaining strong transactional data consistency across our Kafka events and databases.

The technical solution in place today provides a scalable, standardized toolkit for sharing and collaboration on the Compass platform that can evolve to meet the needs of a wide variety of collaboration patterns and product use cases we haven’t even thought of yet. Engineering teams are more efficiently delivering collaboration capabilities to agents. We’ve seen the development timeline condense since the initial rollout due to having established patterns that are applicable across all services. Most importantly, we’ve received great feedback from agents.

Customer Testimonials

“Checklists are simply excellent when as a team, we often have two people working on getting everything done, whether getting a listing life or a transaction through escrow. This way, things do not fall through the cracks thinking that my team partner scheduled the photoshoot, sign placement/removal, or anything else vital. It also lets us know if we have already reached out to our client with the weekly listing/escrow/buyer update. I LOVE the checklists!”

— a Compass Agent Team Lead in Bellevue, Washington

“Compass for Teams makes our work so much easier and organized. Now I can collaborate with my teammates without having to switch accounts anymore — all the collections, listings, and our communication history, they are all just there.”

— a Compass Agent in the Bay area

“The changes your team has made have been a game-changer for [our team]. It’s made an immediate impact on our business.”

“We are so excited to be able to upload our teams’ checklists into Business Tracker. The team component that allows us to assign tasks to individual team members is amazing and a fantastic communication tool.”

— a Compass Agent Team Lead in Lafayette, California

Upcoming Improvements

We made significant progress in 2021 to enable agent collaboration across the Compass platform, and we’re continually looking for ways to improve our customers’ experience further.

Earlier, on the technology side, we discussed the 10x improvements to producer write latencies through the adoption of the Outbox pattern. We want to do the same for consumer read latencies.

One approach we’ve adopted is to utilize the concept of materialized views with Kafka. Materialized views are internal representations of the Kafka event log that are customized for each consuming service. Each materialized view is independent of the others and can project data in totally different ways. Materialized views are flexible, with the choice of data store up to the consuming service, such as an in-memory hash table, a PostgreSQL database, or more complicated data storages like an ElasticSearch cluster or a graph database. Each consuming service can optimize for only relevant data and scale independently of the Kafka event stream.

We’ve already begun early experiments using Materialized Views in some of our consuming services. In one extreme example involving nested permissions, we have been able to reduce read latencies by 50x (1000ms to 20ms). While we don’t expect that same magnitude reduction across the board, it represents value in continuing to evolve our implementation. Similar to the benefits seen for producers and write latency upon adoption of the Outbox pattern, we expect to see significant benefits for read latency as we adopt Materialized Views more broadly.

We are also looking at adding robust features to provide flexibility to agent teams in acknowledgement of the myriad of models they use to operate. What we have today is granular in nature, enabling the sharing of specific resources with specific people. As we move forward, our agents will have the power to define the sharing policies that govern what collaboration means for their team. We’re evolving our experience to provide modular and more flexible integrations across a broad range of additional use cases, including non-agent and 3rd party collaboration. Our vision is to empower every agent to collaborate with any person, on any resource or job, from any part of our platform. The time and energy they save via collaboration can be spent to serve their clients and grow their business.

The work done to enable agent team collaboration represents a foundational shift in the Compass product and engineering approach to delivering software for our customers. Our internal teams work more collaboratively, thanks to the common framework. We’re rapidly adopting an event-driven approach and realizing better performance with improved reliability.

Acknowledgements

This blog is written from the perspective of owners of some of the domain services that rely on the core AuthZ services to provide agent collaboration. The Compass Core Experiences team, including the Identity-Authorization team in particular, set the vision for collaboration at Compass and performed most of the heavy lifting to deliver the set of services upon which all agent collaboration features are built. They are excellent partners at the center of what we’re building at Compass.