Harvesting Sales Data from B2B Channels

A use case with Salesforce, EDIFACT B2B messages, and Ballerina for real-time sales data extraction from B2B EDI messages

Businesses often engage in B2B communications that include the exchange of diverse documents like invoices, purchase orders, insurance claims, and shipment handling orders. Predominantly, these communications use Electronic Data Interchange (EDI) messages, with EDIFACT and X12 being the most common standards. It’s crucial for organizations to integrate these EDI-based B2B channels with their internal IT systems to ensure a seamless, end-to-end digital experience for customers and partners.

Just as B2B messaging is vital for engaging with business partners, Customer Relationship Management (CRM) systems are essential elements in an IT landscape for managing sales and customer data. Integrating CRM with all customer touchpoints is critical for up-to-date information capture. Given that B2B is a principal method of business communication, companies dealing with enterprise customers can reap significant benefits by effectively integrating their CRM systems with B2B channels.

Ballerina’s EDI package offers a straightforward yet robust way to manage EDI messages, with built-in support for EDIFACT and X12 standards. Furthermore, Ballerina has connectors for widely used systems such as Salesforce, Oracle NetSuite, and Microsoft Dynamics 365. In this context, this article explores how Ballerina can facilitate seamless integration between CRM systems and B2B channels, focusing on Salesforce and EDIFACT B2B messages.

Architecture overview

It is important to note that in addition to CRM systems like Salesforce, there are many other systems, including Enterprise Resource Planning (ERP), auditing, and warehouse management systems, that require access to B2B messages. In such scenarios, messaging systems like Kafka can be used to build an event-driven architecture, enabling multiple consumers to process each incoming and outgoing message. The following diagram outlines an architecture that integrates B2B channels with Salesforce and other IT systems using Kafka and Ballerina, highlighting an example scenario involving quote requests and responses.

In the above architecture, an FTP server is used for EDI message exchange. However, alternative mechanisms like HTTP, Amazon S3, Azure Blob Storage, or SFTP can also be used. Ballerina components serve as the integration layer between source systems, target systems, and the message broker.

Ballerina components handle protocol bridging, message transformations, and other processing tasks applicable on EDI data. Alongside the demonstrated Ballerina components, any number of additional software components can consume B2B messages by subscribing to relevant topics in the message broker.

A Ballerina component and a Kafka topic are used per each EDI message type. Therefore, the capability for handling new message types can added to this architecture, without affecting existing components.

A Glimpse into EDIFACT

Before diving into a specific example scenario involving this architecture, it’s important to briefly discuss the EDIFACT standard that we’ll be using. EDIFACT is one of the most prevalent EDI standards, especially popular in Europe and countries outside North America. It defines over 230 different messages, or transaction sets, that span various industries like retail, manufacturing, logistics, and finance.

For example, QUOTES message captures a comprehensive set of details for quotations, including customer identification, quotation dates, payment terms, and details of quoted products. The specification dives into granular aspects as well; under payment instructions, more than 60 methods are standardized with unique number codes — like cash (10), cheque (20), and credit transfer (30) — enabling clear and efficient transactions.

Example scenario

Now let’s focus on an example scenario of processing quote requests and quote responses within an organization named TechPulse.

• Customers who need quotations for certain products place quote request messages (EDIFACT REQOTE) in a given FTP location.
• These EDI quote request messages are transformed into an internal JSON format and placed in a Kafka topic.
• JSON quote request messages are used to create opportunities in Salesforce.
• Once TechPulse issues quotations, which are placed in another Kafka topic as JSON messages.
• Relevant Salesforce opportunities are updated using data from these quote response JSON messages.
• JSON-based quote response messages are converted to EDI messages (EDIFACT QUOTES) and placed in the output FTP location.

The following sections go through the Ballerina components performing the above tasks. Note that each Ballerina component is written to be executed periodically (E.g. using a cron job)

Reading EDIFACT REQOTE messages and publishing transformed messages to Kafka

The edi-to-quote-request component reads from the given FTP location and translates incoming EDIFACT REQOTE messages into an internal format. These are then published to a Kafka topic, allowing any interested system to consume these messages. The source code of this component is shown below. Note that one of Ballerina’s built-in EDIFACT packages (ballerinax/edifact.d03a.retail) is used for processing the REQOTE message.

Ballerina can visualize code as sequence diagrams that illustrate connections to external systems and control-flow constructs. Visualization of the above code is shown below:

EDI standards like EDIFACT and X12 come in various versions, and even when using the same version, organizations often send messages with slight variations. Therefore, converting these external EDI messages to an internal format is typically beneficial before further processing.

Creating a Salesforce opportunities for quote requests

The component quote-request-to-salesforce reads these internal messages, creates Salesforce opportunities, and adds relevant products under these opportunities. Prices are initially populated from Salesforce’s price book.

Updating Salesforce with quote response data

In order to keep Salesforce in sync with the quotation process, the opportunity created for the corresponding quote request needs to be updated with the details of the quote response. This is done by the quote-response-to-salesforce component, which will change the opportunity stage to Proposal/Quote and update the prices of all ordered items with the prices given in the quote.

Sending transformed EDIFACT QUOTES messages to an FTP location

Finally, the quote-response-to-edi component converts internal quote-response messages to EDIFACT QUOTES messages. These messages are then saved in an FTP location for consumption by the relevant partners.

The complete source code of this example solution can be found in GitHub.

How it works with a sample quotation request…

Once the solution is deployed, quote request EDIFACT messages can be placed into the FTP input location. A sample EDIFACT REQOTE message is shown below:

After executing the edi-to-quote-request component, the EDI quote request will be transformed to the internal JSON format and will be published to the quote-requests topic. JSON message generated by transforming the above request is shown below:

{
    "oppName": "TSP_015",
    "accountName": "TechShop",
    "itemData": [
        {
            "itemId": "01t6C000003idHmQAI",
            "quantity": 38
        },
        {
            "itemId": "01t6C000003idHrQAI",
            "quantity": 56
        }
    ]
}

Now once the quote-request-to-salesforce component is executed, a Salesforce opportunity will be created for the above message as follows:

Products associated with the newly created Salesforce opportunity are shown below:

Let’s assume that TechPulse decides to give a 5% discount for the first product (VisionPro TV). Let’s also assume that TechPulse can only provide 50 units of the second product (NexaView TV). This results in a quote response JSON message, which will be placed in the quote-responses topic (Sample quote response can be generated by executing the quote-processor component).

{
    "accountName": "TechShop",
    "oppName": "TSP_015",
    "itemPrices": [
        {
            "item": "01t6C000003idHmQAI",
            "unitPrice": 880.0,
            "quantity": 38,
            "discount": 5.0,
            "totalPrice": 31768.0
        },
        {
            "item": "01t6C000003idHrQAI",
            "unitPrice": 640.0,
            "quantity": 50,
            "discount": 0.0,
            "totalPrice": 32000.0
        }
    ]
}

Now, once the quote-response-to-salesforce component is executed, it picks this quote response and updates the stage of the opportunity created for the corresponding quote-request as follows:

It also updates the product list associated with the opportunity by updating quantities and discounts as shown below:

Finally, once the quote-response-to-edi component is executed, it transforms the quote-response message to the following EDIFACT QUOTES message and places it in the output FTP location.

Further considerations on granularity and deployment

In this architecture, we employ four separate Ballerina components, each handling a specific task, in line with microservices architecture principles. However, the granularity of these components can vary based on several factors like load, extensibility requirements, infrastructure costs, and team structure. For instance, under low load and tight budgets, it is possible to consolidate all tasks into a single Ballerina component. Conversely, if the system must handle multiple transport types such as HTTP, Amazon S3, and FTP, a dedicated component might be needed for each, while isolating EDIFACT-to-internal format transformations into another component.

On the deployment front, each Ballerina component can be encapsulated as a Kubernetes pod, compatible with various container orchestration systems like plain Kubernetes, OpenShift, Amazon EKS, or Azure AKS. Alternatively, this architecture can be built on an internal developer platform like Choreo. This accelerates the go-live timeframe by offering pre-configured CI/CD pipelines, multiple deployment environments (E.g. dev, test, staging, prod), security features, and analytics.