Maintaining Reusability and Observability When Integrating 3rd-Party Data.

In this article, we will discuss how we developed a generic system for synchronizing data between various Property Management Software (PMS) systems and Venn.

Data integration flow

The main ideas behind the solution were to enable better observability of the flow’s lifecycle, centralize all sync logic into a single place, and allow for easy adoption of new system integration into Venn.
In this article, we will only focus on one-way sync: pulling data to Venn.
Although the primary concept is within the realm of PMS, it can be applied to any type of data integration with various data systems.

What is PMS:

Property Management Software is an information system that enables real estate property managers to manage and track buildings and residential units they own. It also allows them to keep track of tenants in the building, their leases, and everything that revolves around them, such as work orders, receiving packages, rental payments, and more.

Motivation — Who we are? and why the hell do we want this data?

At Venn, we believe that everyone deserves to feel a sense of belonging in their homes and neighborhoods.

Our mission is to create vibrant and inclusive communities where residents can thrive and connect with one another on a deeper level.

As the world’s leading resident experience platform, we empower multi-family owners&operators to transform how they interact with their residents.

Through innovative tools, data, and experiences, we help them create a seamless end-to-end resident journey that maximizes engagement and increases their net operating income.

In addition, we have an app where residents can easily communicate with each other, participate in various events, talk to the property manager, reserve space in shared work and leisure areas, and even open smart locks in their surroundings.

Every one of our customers gets a “Hood” — a group of buildings, their units, and their residents.

So, as you can understand, the data that interests us at Venn is everything that is in the PMS. However, we will focus on four specific entities that are at the core of our business which are:

• Buildings
• Units
• Users (Residents)
• Lease Contracts

Now that we understand why we need this data, let’s move on to the synchronization process.

Sync Process:

We will start with a simple presentation of the architecture.

PMS data sync flow architecture

Sync Trigger:

A scheduled job that runs every X minutes (configurable), pulling the data from the configured Property Management Systems (PMS) for each of the neighborhoods that need to be synchronized.

Client adapter:

Why do we need an adapter? In integration, the dynamic part is the most important. Every PMS has a different API, and a different way to pull data. For example, in one system, to pull data we would use a REST API, in another, SOAP, and in a third, we might use an SQL client. Therefore, inside the adapter, a very simple abstraction is implemented. We define an interface for synchronization. For building the adapter, we need to implement a function called the “initializer” that will run at the beginning of each sync on each client. We also need to implement the interface for each module that supports the PMS that the adapter implements

PMSClient interface:

export interface PMSClient {
  Initializer: Initializer;
  CommonSync: ICommonSync;
  ServiceRequestSync?: IServiceRequestSync;
  BillingSync?: IBillingSync;
}

Implementation:

export const demoClient: PMSClient = {
  Initializer: { init: async () => {}, close: async () => {} },
  CommonSync: commonSyncImplementation,
  ServiceRequestSync: supportSyncImplementation
};

In the code above, we see an empty implementation of the “init” function — in other cases, we will open a connection to the PMS or define credentials to fetch from their API.

We also see three additional interfaces:

• CommonSync — This is a mandatory interface that contains Buildings, Units, Users, and Leases.
• SupportSync — This is an optional interface to implement, as not all PMS systems support it.
• BillingSync — This is also optional, and as we can see in the example above, we haven’t implemented it yet on the demo client. We can define a module for each module we want. The work on them is very similar, as we can see in the following lines

let’s dig into the CommonSync Interface:

export interface ICommonSync {
  Building: BuildingSync;
  Unit: UnitSync;
  User: UserSync;
  LeaseContract: LeaseContractSync;
}

Implementation:

export const commonSyncImplementation: ICommonSync = {
  Building: building,
  Unit: unit,
  User: user,
  LeaseContract: leaseContract,
};

For each entity, we have defined three essential components:

• Fetch Function: A function that retrieves data from a specific Property Management System (PMS) and returns it as an array of entities.
• Mappings: An object that specifies how to map the PMS fields to the corresponding Venn fields. We use the Morphism NPM library to facilitate this process.
• Filters: An optional array of JavaScript filter functions that allow us to exclude specific data that we don’t want to synchronize. These filters can be applied after fetching the data from the PMS.

For instance, let’s consider the example of a Building entity.

export interface BuildingSync {
  fetch: FetcherType<BuildingTypes>;
  mappings: BuildingMappings<BuildingTypes>;
  filters?: FilterType<BuildingTypes>[];
}

export const building: BuildingSync = {
  fetch,
  mappings: mappingFields,
  filters
};

Once we wrap all the entities in all the different PMS, we will get one generic interface for fetching and mapping data, regardless of which PMS it came from.

To understand the harvest stage, we will present the...

IntegrationNodes & SyncOperations entities

We want to follow the event life cycle once we receive an event (Create/Update/Delete) on an entity (Building, Unit, User…) until it gets created/updated/deleted.

We Created 2 entities that help us manage this:

1. IntegrationNode

type IntegrationNode {
  id:                  ID!                
  externalId:          String!               // the id on the pms        
  externalEntityName:  String!                 
  vennEntityId:        String                // the id on our side        
  vennEntityName:      SyncedVennEntityNames   
  hoodName:            String!                 
  source:              CreationSource!       // to support bi-directional sync        
  entityData:          Json                  // the data as it fetched from the pms      
  status:              IntegrationNodeStatus!// IN_PROGRESS | COMPLETED | FAILD
  parent:              IntegrationNode       // Building -parent-> Unit for example
  children:            [IntegrationNode]       
  operations:          [SyncOperation]       // list of operation per record fetched  
  partnerName:         String!                 
}

2. SyncOperation

type SyncOperation  {
  id:                     ID!                     
  integrationNode:        IntegrationNode!      // operation's iNode         
  type:                   OperationType!        // CREATE | UPDATE | DELETE     
  status:                 IntegrationNodeStatus!// IN_PROGRESS | COMPLETED | FAILD 
  changes:                Json                  // part of the entity data that get changed  
  metadata:               Json                  // store any metadata while the sync running 
  trigger:                OperationTrigger!     // which flow triggered the sync could be Manual or Schedule   
  userId:                 String                // if manual which user triggered it 
}

Let’s look at the example below:

Some notes before:

1. To understand how we got events please see the harvest part
2. all the integration nodes and sync operations will be created with IN_PROGRESS status by default at the beginning of the sync process

Simple scenario:

1. We got a Building Create event -> We create Building IntegrationNode with one SyncOperation with Create type.
2. We got a Building Update event -> We create one UPDATE SyncOperation and connect it to our existing IntegrationNode (the same will happen if we got other updates or deletions).
3. We got a Unit Create event -> We create Unit IntegrationNode with one Create SyncOperation -> then we set his parent to be the building IntegrationNode
4. The same will happen for user & lease create
5. Then on the ingest part, we will fetch those syncOperations and we will persist them on our DB
6. If the Ingest finishes without any errors it changes the status both on the integration node and the sync operation to COMPLETED
7. Otherwise, it changes their’s status to FAILED and stores the error message on the sync operation metadata

This data structure will allow us to understand the event cycle, map our errors, and audit the operations in the order they occurred.

Harvest:

During the harvest stage, after calling the initializer function, we perform several actions for each Entity:

1. Fetch PMS data — We utilize the fetch function implemented by each entity to retrieve data from the PMS.
2. Fetch iNodes data — We retrieve iNodes data by using the external ids obtained from step 1.
3. Diff check — We compare data obtained from steps 1 and 2 and check if they already exist on our side. If not, we create an iNode with a CREATE type syncOperation. If the iNode already exists, we check the entity data for any changes. If there are any changes, we create an UPDATE type syncOperation.

• All iNodes and syncOperations created during the harvest stage will have an IN_PROGRESS status.

Ingest:

The Ingest component serves as the final step in the integration process. It receives events from other processes (such as Common Sync and Service Request Sync) and converts them into GQL queries that ultimately become a row in our database.

Similar to Harvest, the Ingest part involves multiple steps:

1. For each entity:
2. We fetch all syncOperations with an IN_PROGRESS status.
3. For each syncOperation:
4. We map the entity data (or changes if it’s an UPDATE event) to Venn’s expected fields using a library called Morphism.
5. We validate all fields, such as verifying that the user has a valid email and that the building address is real.
6. We persist the data in our database.
7. If all of the above steps are completed without any errors or validation issues, we change the INode and syncOperation status from IN_PROGRESS to COMPLETED.
8. Otherwise, we change the status to FAILED and add the error or validation message to the syncOperation metadata for better understanding later on.

Conclusions:

Integrating with any new 3rd party system can be a time-consuming process.

With the solution we implemented, we have a clearly managed sync process, monitoring and fail recovery,
but the biggest outcome is that now, the work time of adding a new integration has significantly decreased and has been simplified.

stay tuned for more insights to come

At Venn, we believe that everyone deserves to feel a sense of belonging in their homes and neighborhoods.
Our mission is to create vibrant and inclusive communities where residents can thrive and connect with one another on a deeper level.

As the world’s leading resident experience platform, we empower multi-family owners&operators to transform how they interact with their residents.
Through innovative tools, data, and experiences, we help them create a seamless end-to-end resident journey that maximizes engagement and increases their net operating income.

With Venn, it’s not just about improving the bottom line — it’s about creating meaningful and impactful connections between residents and their communities.
We’re committed to making a positive difference in people’s lives, and we believe that starts with fostering a strong sense of belonging in the places we call home. Join us on our mission to build a better world, one community at a time.