Guaranteed Message Delivery

Grainite enables reliable and lossless communication between enterprise applications.

Guaranteed message delivery refers to the assurance that a message will be delivered to its intended recipient successfully and without loss or error. Most Enterprise applications communicate with other internal or external applications and services (downstream systems). Information (system of record) from one application becomes the source of information for other downstream systems. This is important in many applications, such as financial transactions, critical system monitoring, and real-time communication systems, where the correct and timely delivery of messages is essential. So, the source application must guarantee that the message will be delivered to the downstream application.

Examples where guaranteed message delivery is critical:

• A user changed their profile information. This must be notified to the customer via systems like SendGrid or Twilio, to ensure these changes were not fraudulent.
• A banking customer opened a new line of credit. Notification must be sent to all three credit bureaus.
• The Finance team provided a discount to a customer in Quickbooks software. This information must be recorded in SalesForce for audit purposes.

Challenges faced by Enterprises

Issues faced by enterprises include message delivery failures, duplicate messages, or out-of-order message delivery. In most cases, ensuring that the downstream system receives the notification exactly- once is a burden on the source application. This results in information being out of sync. The downstream system cannot take any action as its not aware of messages being originated from the source application.

Suboptimal approaches that are taken by organizations today

To overcome these challenges, organizations

• Store the change notification in a separate database or a queue and have downstream systems pull the message from that database
• Another option is to leverage an intermediate component that pulls the data (instead of making the source application push) and ensures the downstream system receives the message

The complexity of current architectures

There are multiple challenges with these approaches

• Developers end up building separate intermediate components for each downstream system. This leads to increased system complexity and significant engineering effort.
• Each downstream system will have different SLAs, configuration parameters, message types, and security mechanisms that need special handling.
• If any part of the system were to crash, all pending messages will be resent to the downstream system. This makes message delivery at-least-once, which may not be acceptable in many use cases.
• There is no way to determine if the downstream system processed the message.
• It’s difficult to search for messages within the system and determine if they have to be reprocessed.
• It’s difficult to avoid sending duplicate messages to downstream systems in the event of failures.

Guaranteed Message Delivery with Grainite

Grainite is an integrated event processing and data platform that provides an out-of-box guaranteed message delivery capability. Applications that need to send a message to a downstream system simply route the message through Grainite via an HTTP call. A plugin framework ensures information about the downstream system can be specified by customers to support a wide variety of applications (e.g. Snowflake, SendGrid, Salesforce, and CDP such as Rudder Stack).

How does Grainite Guarantee message delivery?

• Applications append messages to endpoint-specific topics. Topics in Grainite are similar to any other messaging system but have additional advantages.
• Grainite atomically records the consumption of the message from the endpoint-specific topic and the delivery/acknowledgment of the message from the downstream system, thereby maintaining the requirements for exactly-once processing
• Grainite allows developers to define target-specific code to check the status of the message on the external service before a submission is retried.
• In case of temporary or permanent outages, Grainite can be configured for ordered processing of the messages by endpoint, perform appropriate backoff and retries, and optionally set up dead letter behavior, including alerts as needed.
• IT teams perform operations such as searching messages, reprocessing messages, and managing downstream (related systems) outages using the management interface.

Benefits of Grainite

Development simplicity: Developers can just focus on the business logic. Grainite takes care of the resiliency & scaling for your needs.

No message loss: All events coming in from Salesforce are processed exactly once, and full audit logs can be generated.

Automatic handling of retry and failures: Grainite’s action handlers and tasks are resilient to temporary or longer-term failures in any of these systems. Grainite has built-in exponential backoffs to handle any rate limits and network outages.