How we built a Serverless ETL data integration from NetSuite to Salesforce

Here is an a-ha moment in building serverless, cost-effective NetSuite and Salesforce data integration. We will tell you how we built a solution and saved more than $6,000 annually for one of our customers.

Serverless computing is becoming a thing these days, and many companies have already adopted this approach to build their products and tools. You can see how we created serverless integration to integrate data for one of our customers. Serverless is no longer slow, as many people think. However, serverless execution for cloud-delivered applications must be applied thoughtfully.

Our goals and requirements

We were about to start building a long-term low total cost of ownership solution for our customer. Some of the requirements were:

1. Reliable and robust data integration tool for currency exchange rates from NetSuite to Salesforce.
2. Be able to configure integration from Salesforce without writing the code in Apex.
3. Safely store all sensitive information.
4. Build a library so that our customer IT department can extend an integration in the future if needed.
5. Close to zero monthly associated expenses and burden on the IT department during adoption and product lifetime.

We chose to use AWS Serverless Application Model (SAM) over Serverless Framework. We made this logical decision for a few reasons:

1. A lot of new infrastructures that our customer built was already in AWS Cloud.
2. They were migrating old infrastructure into the AWS.
3. AWS provides a combination of the tools and building blocks that we needed without significant changes to the off-the-shelf.

Serverless Framework suites better if you need to have multi-cloud support. For instance, if you need to deploy your serverless code to Microsoft Azure and AWS at the same time. We suggest that you may want to use AWS SAM if you are building AWS specific integrations since it designed explicitly for AWS. You may think it sounds like a vendor-lock, but it is not: both frameworks are similar, which allows you to migrate your code from AWS SAM to Serverless Framework and vice versa.

AWS SAM provides excellent support for local development out of the box without installing additional plugins.

The decision on the language came naturally. We are experts in writing serverless Python code (or Python Serverless if you will), so this came well into play with our customer needs, as most of their engineers are Python experts.

One-time quick set-up

Here is a diagram of the user interaction with the system. On this diagram, we show the NetSuite to Salesforce data connector which does the heavy lifting in the integration.

It only requires NetSuite and Salesforce administrators to set credentials once.

We built serverless web applications for both NetSuite and Salesforce to securely encrypt and store login credentials.

Action

Once the system was deployed and configured, it started to synchronize data from the NetSuite to Salesforce, based on Salesforce custom object time field. Here is a detailed diagram.

In step one, we decided to pull schedule changes from Salesforce instead of triggering an action on custom object field change, adding a trigger in Apex would couple this integration with Salesforce implementation more tightly, which is not what we wanted. It would also expose a Lambda function endpoint to the world, which would add a vector for a cyber attack. Instead, we used the Salesforce REST API to ask for record change in our Scheduler Lambda function.

A quick demo of the serverless integration we built is shown. With a single command, you can quickly deploy the entire serverless stack.

Before this integration, someone from the company, once a week has to create or update dated exchange rates in Salesforce manually. That person did not even have access to the NetSuite, so they have to use some other source for currency exchange rates for each day of the week. By eliminating this routine, repetitive and error prone work and avoiding monthly fees on expensive connectors that exist, we ware able to save for our customer thousands of dollars per year.

As of this writing, monthly cloud computing fees, for this integration, are around three dollars, expenses associated with the burden on IT are close to zero.

Conclusion

In this article, we have shown one example of how to implement a serverless approach for building reliable data synchronization tools. In this case, it is a data synchronization, or data connector tool. It is a cost-effective, efficient and secure way for data integration solutions.

At Konstankino, we can solve our customers’ computing needs using effective serverless integrations. Email us at info[at]konstankino.com if you need help with your serverless computing.