What’s Interesting and Challenging in Compass Engineering
Compass’s mission is to help everyone find their place in the world. Our strategy is to recruit the best agents in the business, give them unparalleled support, and help them grow their business.
We are not looking to replace agents. We believe that real estate decisions are very complex and in today’s world can best be made through High Tech and High Touch. It’s about combination of agent’s deep knowledge of real estate, their ability to negotiate and software — our product platform; technology is a key enabler, but only part of the story. We build tools to help agents to collaborate with their clients, to help them market themselves and their seller’s properties, analytics tools to help them understand how to improve their business efficiency, and products to help them understand the current market conditions, and much more.
This post describes the interesting aspects of engineering and product creation happening at Compass R&D, currently numbering around over 100 people across Engineering, Product, Design, QA and Product Marketing. Summary:
- Complex Domain & Workflow
- Platform Orientation & Integration
- User Experience Focus & Beautiful Design
- Large Dataset, Heterogeneous Sources
- Data-Driven Insights & Machine Learning
- Modular, Cloud Native Architecture
- Strong Engineering Practices
Complex Domain & Workflow
Real Estate agents run a small (and some not so small) business, with everything that entails — managing finances and cash flow, hiring people (many agents work in teams), lead generation, marketing, sales operations, keeping up with the current market trends, and many other things. There are many different business styles, from an agent that sells five $20M properties a year, to a team that transacts hundreds of properties. An agent can represent either buyer, seller, or renter, and each of those entails specialized workflows. Agents’ business necessitates being mobile, so ensuring that many of our tools work seamlessly across web and our iOS app is important.
Additionally, since Compass works in multiple geographies, each region has different working styles, data that’s important, and regulations that we need to understand and solve for. To add to the fun, because the small businesses are connected to the brokerage, our solutions need to not only work for disconnected businesses, but also need to connect into a larger whole.
Lastly, we need to present our listings, search and other tools to both agents and consumers.
Platform Orientation & Integration
In my previous job, I was responsible for third-party developer platform, ensuring that we made our software easy for external developers to integrate with. I thought that Compass would be a step down in terms of engineering challenges and complexity.
To be honest, that was true at the time I joined in September of ‘16, when we were more focused on building point products (although even then we had a few platform-like applications). As we mature and grow, we are increasingly moving thinking in terms of reusable products and platforms that help us solve whole classes of use cases vs just one or two specific problems.
Here are some parts of the platform we are building:
- Search & List of Listings — this is arguably the first part we built — our search is easy to integrate with other products. For example, we need our search to support ability to select a few listing, and take them to another product (say you want to email it to a client, or save them to collection).
- Marketing Center is a systematic approach to marketing that’s both a destination product, and a capability we want to draw upon any time it’s needed across the platform.
- Insights — new product (Business Intelligence dashboard for agents) in 2017, we focused the initial release on marketing insights, but designed it in a way to be reusable, so we can easily display metrics to agents.
- Customer Relationship Management (CRM) — this is a major effort for 2018 to expand on some efforts we started last year. Clients are at the center of what agents do. CRM will connect many workflows together and enable agents to serve their clients better.
An important source of value is that the above platforms are integrated with each other. This gives the user a very powerful experience and ability to easily switch from one task to another. As we build out more products and systems, this will increasingly be an advantage.
Large Dataset, Heterogeneous Sources
Our data engineering team ingests real estate listings data from many data sources at scale (tens of millions of listings) and with minimum latency. In addition to the listing data we get from MLS, we integrate listings with additional data sources, such as school data, public records, and many others. This is where we use some big data technologies such as Spark. After we ingest the data, we need to normalize it (the 150+ listing data fields are different in each region), index it quickly and make available in our search, which has unique interesting technical challenges.
Once the Listings data is imported, it’s used in most of our products.
User Experience Focus & Beautiful Design
Taming the complex domain and workflow, especially with users of different technological sophistication, requires great user experience. As a luxury brand, visual presentation is hugely important both to us, our agents, and ultimately, our clients. We have a world-class user research, branding, and product design teams that help inform our products and ensure that our suite of products is easy to use and intuitive.
One of the key advantages is that we have direct access to our users, and many agents joined because they knew we are interested in their experience. We have many channels for understanding our clients and gathering feedback, from direct meetings and focus groups, to surveys, to in-product feedback, and more. Being platform-oriented also helps improve user experience, as we reuse patterns throughout the suite.
All of the above combines to get us industry-leading adoption of our technology among agents, but as we expand our product offering and complexity, we need to continue paying attention to our customers.
Data-Driven Insights & Machine Learning
We are already fairly data-driven in our decision making around products, but we want to take that to the next level. A big part of our vision is to be a data company, one that has access to something that nobody else in the industry really has, which is near-to-real time transaction data. We feel this will be increasingly a differentiator which will allow our agents to serve their clients better and grow their business.
We are not there yet, but we’ve made huge progress over the last year and a half in our data warehouse and BI infrastructure and are significantly increasing our investment in data analysis and machine learning analysis going forward. For example, as part of our CRM work, we expect to be able to do interesting things with predictive analytics in helping identify past clients that may be ready to move again.
Our longer-term vision involves creating an even larger data platform that will have real time market data and allow to make more intelligent decisions, further differentiating Compass agents, and helping grow their business. Compass platform will understand the agent’s business type and style, and help him / her grow their business on a daily basis.
Modular, Cloud Native Architecture
Our architecture, both on web frontend and backend is designed to be modular. On backend, we have a microservices architecture with three first-class languages — Java, Python and Go. We are cloud-native, hosted on AWS, and take full advantage of its ecosystem of services (we use 25+!) — that allows us to build on top of a reliable platform and focus on differentiated business logic.
On frontend, we spend a lot of time thinking about modularization of our applications, which helps teams work independently and allows supporting the platform approach mentioned above. In 2018 we anticipate launching a new version of our frontend stack and deployment tooling, based on Kubernetes, which will enable our engineers to experiment with new ideas and frameworks much easier.
Our mobile team has achieved high velocity through a combination of modularization and use of clear patterns throughout their apps (see all mobile posts).
Strong Engineering Practices
The above is not possible, or at least much less fun, if not underpinned by strong engineering practices and processes. We have a number of practices to ensure high quality code and products — from strong code review process and technical design processes, to initiatives to improve developer experience for different type of work and diverse developer workflows.
While we have a number of strong processes as the foundation, there is more to do. We do daily deploys, which is behind the state of the art, but have improvement projects to improve this. We have some remaining single points of failures that we are also unwrapping. On the QA side, we are moving towards the test pyramid, having introduced an API testing framework that allows us to do a lot of testing on API level, while our Selenium / Cucumber tests are doing a lot of functional verification.
Conclusion
We plan to double our engineering team from about 60 to 120 this year, and keeping the best parts of our culture as we evolve is definitely a key focus. Our colleagues are talented and collaborative and solution-driven. Many new people remark how they feel people are so supportive and happy to help each other out, and that overall culture is very welcoming. We’d like to welcome you — please see if there is a position that matches your skills!
