Platform architecture of repliers.io

repliers.io’s primary function is as a lead nurturing system for real estate agents and teams. However, it may also be used to access MLS data from various real estate boards in an extremely easy way. API documentation can be found here.

About six months ago, fellow repliers.io co-founder Pat Arlia posted this article https://medium.com/@patrick.arlia/navigating-the-toronto-real-estate-boards-treb-s-data-exchange-522b728b4009 detailing the difficulties and complications when accessing data from Real Estate Boards and how @repliers_io helps solve these issues.

I’ll outline and explain, in a high-level, our current architectural approach to overcome some of these problems (numbered below). Over the past year, we have iterated over our architecture a few times. Transitioning from platforms like Digital Ocean to Heroku to AWS and moving from full monolithic apps to a combination/hybrid of monolith and microservices. We also believe this layout permits repliers.io to quickly integrate additional and/or remove 3rd-party services with little impact to the overall system.

1. We decided to go the AWS route mainly for three reasons: Cost/Affordability, Functionality, and Usage Experience.
2. MLS data is imported/uploaded into our Aurora MySQL-compatible cluster at certain intervals. Due to the fact that each board might have different designs and/or models, we transform the incoming information to fit our schema, so the details returned from repliers.io will be uniform.
3. MLS images are uploaded in an S3 bucket. The majority of the images that are uploaded from the real estate boards are baseline images which are not performant for sites and apps. To alleviate this problem, once the image is placed in the bucket, a lambda is run to convert the image to a progressive JPEG. On average, this conversion process takes under a second. In general, progressive JPEG images can be up to 10% smaller than baseline JPEGs, thus using them will provide a better visual experience and improve the overall loading and performance of a website/app¹.
4. When certain data is inserted/updated/deleted, triggers in the Aurora MySQL-compatible databases are executed to place messages in queues (SQS) which then call specific lambdas to process such messages. This permits repliers.io to provide realtime/instant information when it becomes available. We feel that triggers provide the better benefit over polling for changes in the tables. We just made sure that the triggers are lightweight and quick as to not bog down the database. In addition, the use of queues allows to us to retry failed messages in cases of failed network connectivity to other services. Examples of such messages are: sold/leased notifications, new listing that matches a client’s search criteria, incoming sms/email, etc.
5. The MLS data that is uploaded from some of the real estate boards are missing geocodes (lat/lng coordinates). These data points are necessary for one main reason: repliers.io allows searches based on maps and takes advantage of Aurora MySQL-compatible database’s GEOSpatial query support.
6. As repliers.io is a lead nurturing service, the ability to handle outgoing sms, email, and whatsapp messages is also of vital importance. Massaging outgoing data to map to specific message templates and formats for numerous messaging services is provided by dedicated lambdas.
7. Being able to execute outbound messages also implies that repliers.io needs to handle incoming sms, email, voice, and whatsapp messages. Using AWS API Gateway permits us to secure endpoints into repliers.io from various sources and allows the execution of lambdas to validate, handle, parse, and store each type of message.
8. By leveraging the AWS Aurora MySQL-compatible cluster, repliers.io’s nodeJS-based API takes advantage by forwarding GET only calls/queries to the read replicas, thus taking some load off the primary instance since it is continuously hit from many sources for inserts/updates/deletes.
9. All of repliers.io data is stored in an AWS Aurora MySQL-compatible cluster. Ease of setup, security, automatic redundancy and failover, low-latency read replicas, auto-scaling, continuous backup, the ability to call lambdas directly, and the soon to be available, multi-master configuration were keys factors in our decision. All at a fraction of the cost of some other commercial databases we looked at.

*As is with any other API-based SaaS play, we are constantly re-engineering the architecture to improve performance/efficiency while trying to maintain costs as low as possible, so what we have now will most likely change in the future.

If you have any questions or would like to know more details regarding either repliers.io itself or the underlying technology, please feel free to contact myself or fellow co-founder, Pat Arlia.

[1]: Gediminas B. (February 4, 2019). Improving Website Performance — Using Progressive JPEG images https://www.hostinger.com/tutorials/website/improving-website-performance-using-progressive-jpeg-images