Chemical & Consumer Goods Company

Real-Time Operation Analytics SCM Platform

RUBICON Develops a Custom Real-Time Operation Analytics Supply Chain Management (SCM) Platform for a Global Client

“Their level of maturity is outstanding. They have a strict methodology for running projects, and it works absolutely smoothly. We haven’t had any hiccups. It’s one of the best projects I’ve ever been involved in in my 20-year career. Overall, it’s incredible how well-organized they are as a company.”

(Team Lead Analytics Use Case)

Introduction

Effective supply chain management is an integral part of manufacturing and retail businesses. A successful supply chain management process means that a company optimizes its production and quickly and efficiently delivers products to its customers. The best way to ensure that a supply chain management process is successful is by integrating digital technology into business operations and investing in a robust Supply Chain Management (SCM) application.

SCM applications are used for a number of processes:

• Demand planning
• Supply network planning
• Production planning
• Precise scheduling
• Sourcing and supplier management
• Analytics and logistics

Our partner, a Chemical & Consumer Goods Company, decided to enhance their SCM process by replacing their already existing software with a new custom SCM data platform.

Background

With an aim to transition from the already existing off-the-shelf software solution to a custom SCM data platform that would meet their needs, The Chemical & Consumer Goods company partnered with RUBICON and developed a high-performing and cost-effective solution.

After onboarding our software development team to research and successfully prototype a proof of concept (POC) and thus demonstrating the feasibility of the proposed solution, we were able to start with the project itself and develop the data platform.

Our goal was to successfully implement the proof of concept and deliver a software solution in the designated time frame.

The Client

The Chemical & Consumer is a global company that operates worldwide with its headquarters based in Western Europe. Positioned in a number of markets in the world, ranging from beauty care to laundry and home care, the company has been around for over 100 years.

The Project

After delivering a cost-effective, productive and efficient POC way ahead of the deadline, which exceeded the client’s expectations, we started developing the very first version of the SCM data platform. Our team developed a technology solution that was the most suitable option for the company’s supply chain management process.

The Problem

The very problem was the client’s already-existing costly web platform, which didn’t provide them with unified and performant access to data.

The Solution

The POC entailed building a technical proof of concept that would meet all of the client’s requirements. These included building a full-stack solution enabling near real-time querying of supply chain network datasets (40 million rows of data), capable of filtering and displaying network data on the UI map. The goal was to create a fast, efficient, and affordable data-driven platform that quickly (near real-time) queries, analyzes and aggregates supply chain network data. The solution needed to provide better performance, flexibility, modularity, and give the client full ownership of the product. After a successful and feasible POC, the final solution was to develop the product itself, ie. the first version of the SCM data platform.

Project Objectives

• The general goal is to reduce cost within the selection processes by automatizing specific aspects of the cost calculation
• Enable cost simulations of reallocations of production sights (plants) based on technology, material, geographical location within the supply chain
• Enable a user-friendly way to filter the supply chain with more than 40 million rows of data in the main dataset
• Enforce dimension based user access to data in the supply chain by implementing a custom Authorization System
• Manage projects and scenarios with scoped and restricted access to data
• Build an ETL Data Pipeline, to integrate and use provided Company core data sets in the Data lake
• Implement cloud-native solution on Azure
• Secure access to relevant data
• Enable user-friendly and performant UI visualization of the supply chain data on the world map
• Have fully automated DevOps processes

The Challenges

While developing the SCM application, our team overcame the following challenges:

• Replacing off the shelf solution that has performance and usability issues with a customizable full ownership solution
• Making proof of technology choices, feasibility and performance
• Choosing the best-fit Serving and Interactive Querying Storage for a supply chain network big dataset
• Creating a full-stack solution that enables near real-time querying of a supply chain dataset (40 million rows of data)
• Filtering and visualizing data on the UI map
• Building ETL pipelines for extracting, cleaning, transforming, and loading datasets into the Serving storage
• Creating a Custom Authorization System that integrates with other services and enables secure dimension based access to data

The Software Development Process

After meeting the client’s expectations and optimizing the SCM process resulting in a successful POC, RUBICON’s dedicated team continued working on the custom data platform itself. The project officially started right at the end of POC, in August 2020. In the beginning, we organized a workshop with the client where we finally agreed on the scope of the project and the features they wanted to implement. By the end of 2020, we developed the V1.0 of the platform. The development of V1.1 started at the end of 2020 and lasted until June 2021.

ETL data pipelines

ETL pipelines enabled a seamless transformation of datasets in the platform. This automation significantly accelerated the processes performed on the platform.

In short, ETL stands for “extract, transform, load”. These are the three interdependent processes of data integration where the data is moved from one stage to another. Once loaded into the Serving and Querying database, this data can be used for analysis, reporting and obtaining business insights.

RUBICON’s Dedicated Team

The team continued working in the same arrangement as in POC, and it consisted of:

• Product Owner
• Solution Architect
• DevOps Engineer
• UX/UI Designer
• Front-End Developers
• Back-End Developers

Working in Agile

The very first thing that the team created during the POC development was a product roadmap that they would be following throughout the project development phase. Practicing the Agile methodology and working in Scrum, the team organized the project into two-week sprints. Azure DevOps was used as a key tool that helped the team keep organized and follow the process of their work.

Thanks to RUBICON’s agile approach, we successfully and on time implemented both V1.0 and V1.1. The clear and constant communication and involvement of the client as well as the swiftness of developers and their ideas, contributed to the overall successfulness of the project.

Results

In June 2021, we delivered the cost-effective and high-performing final version of the product.

Our team successfully managed to:

• Implement an architecture and cloud-native solution using Azure managed services and choose the most suitable service for near real-time operational analytic scenarios
• Improve performance by 10X and more in some cases compared to the original software
• Build ETL pipelines for extracting, cleaning, transforming, and loading datasets into the Serving and Querying database

Technology Stack

Applications and Data

• Microsoft stack: C#, ASP. NET Core
  - Implementing the backend API and serverless functions
• Distributed Services Architecture
• Azure Data Factory
  - Automating ETL data processing of supply chain data
• Azure SQL
  - Enabling near real-time operational analytics scenarios: using column store indexes, memory optimized tables, dynamic stored procedures
• Databricks
  - PySpark application for scenario costs calculation and simulation
  - Spark, PySpark, DevOps
  - Azure Function to orchestrate job execution

Front-End Technologies

Angular

• Modular design paired with lazy loading for fast load time
• Implementation of custom component library using core Angular features like reactive forms, content projection and Angular templates with focus on performance, reusability and developer experience.
  - All components apply On push change detection strategy for maximum performance
  - By focusing on reusability we were able to streamline the development of client’s other projects
  - We took inspiration for component structure design from Angular Material library to enhance developers experience
• Async pipe subscription pattern to minimize risk of memory leaks
• Sass style preprocessor with BEM naming for scope control

Mapbox

• Integration with wrapper Angular library using directives and components
• Mapbox Geocoding API for forward and reverse geocoding

AG-Grid

• Displaying and editing data directly from tables
• Custom field types and restrictions
• Exporting data from the tables

DevOps

• Git
  - Version control and a multi-repository project structure were used. Each component of the MFO system had its own repository and CI/CD pipelines.
• Azure DevOps
  - The one-stop-shop for all the MFO agile processes. We used it to plan, collaborate and deploy our solution.
• Docker
  - Our solution was containerized with Docker and Docker Compose so that developers could have the whole microservice architecture up locally and with one command.
• IaaC Terraform, Terragrunt (GitOps)
  - Provisioning the infrastructure on Azure with a combination of terraform and terragrunt. Resources were defined in terraform modules and configured with terragrunt.

Microsoft Azure Cloud

• Azure SQL Database
  - Storing application related data
• Azure CosmosDB
  - Storing authorization system data
• Azure Queue Storage
  - Storing messages that are used for updating the authorization system
• Azure Blob Storage
  - Used by Azure Functions, for storing terraform state files and table storage
• Azure Functions
  - Hosting the custom authorization system and orchestrating the pyspark application job executions
• Azure Active Directory, Microsoft identity platform
  - Implementing single sign on authentication
• Azure App Service
  - Hosting the backend APIs
• Azure Container Registry
  - Storing container images for deploying the backend APIs
• Application Insights, Log Analytics
  - Monitoring the application

Project Screenshots

Information

Company: Global Chemical & Consumer Goods company

Region: Western Europe

Industry: Chemicals & Consumer Goods

Project Duration: August, 2020 — June, 2021

Project: Real-Time Operation Analytics SCM Platform

Originally published at https://www.rubicon-world.com.