Coupang SCM Workflow: Building a low-code, no-code platform
Introducing our efficient and scalable data building and service engineering tool for engineers and business users alike
By Ryan Donghyun Jin (@atinjin)
This post is also available in Korean.
Supply Chain Management (SCM) is the complex network of suppliers, delivery companies, products, and more. At Coupang, our next-day rocket delivery and other delivery services are met by the numerous Coupang fulfillment centers around the nation. The SCM team is responsible for predicting product demand and architecting the pipeline for sellers to deal with customer orders. Our team develops algorithms and models that improve the efficiency of processing customer orders and product distribution among fulfillment centers. We work with various types of customer order and fulfillment center data.
At the Coupang SCM team, engineers, business analysts (BA), data scientists, and other operational staff work together to create a holistic SCM system. It’s important that everyone on the team has access to data and can contribute to the data system.
As an engineer working with such a diverse group of team members, certain issues frequently came up.
“How can we make data creation and editing more accessible through the web, and how can this data be integrated to other systems in a hands-free manner?”
“How can we scale a domain without developing a new system?”
“If a business requirement changes, how can we alter the data pipeline without writing new code?”
“How can we develop a low-code, no-code system?”
Faced with the issues above, the largest insight came from the low-code and no-code trend. An engineer can focus on algorithm development, and business users can create complex applications without writing a single line of code. With this in mind, we developed the SCM Workflow platform, which houses functions such as no-code data building and low-code service building.
Our motivation was not only to improve and simplify existing workflow tools, but also to go the extra mile and make sure anyone could launch a simple service through an easy interface.
Table of contents
· No-code data builder
· Low-code service builder
· Other features
· Coupang Workflow web interface
· Still we’re moving forward
No-code data builder
Coupang is a data-driven company. Massive amounts of data are divided by domain in our microservice architecture and managed by each domain team. The domain teams provide the data to other teams through ODS, EDW, API, and other services.
However, as our business grew and diversified, the data requirements were also many and varied. As engineers, we needed to quickly reflect the changing data sources and queries to production. A simple data job required accessing several tools, making it difficult for anyone else but the engineer in charge to make the change.
For example, when the BA and data scientist wanted to make a new model, they had to do so through the engineer in charge, who engineered a new data pipeline for the business requirement. It was a standard job, but one that required access to multiple tools and coding expertise. With every new business requirement, the business user had to make a request to the engineer. This communication process ate up large amounts of time for both engineers and business users.
To solve this issue, we developed a no-code data builder function on the SCM Workflow. Through this data builder, users can easily access data sources and queries, export data, and integrate data to their systems. As a data processor, SCM Workflow can access the following data sources: Hive, Presto, MySQL, S3 and other sources.
Using the data builder is easy. For example, let’s say we wanted to determine how much and when we should transfer products between fulfillment centers to fill the most customer orders. The business users can build a product transfer workflow by following the steps below.
- Create nodes that extract data from the various types of data sources to get inventory and customer order data.
- Enter a query for each node to extract the needed data.
- After data extraction, create a node that uses the data as an input and calculates the number of products, and then connect it to the existing data nodes.
Through these three simple steps, the user has created a workflow that determines the quantity of products that need to be transferred.
In addition, a user can break up a complex query into multiple separate queries and process them in parallel. This process is faster than processing them one by one. The results of the query can be shared with other engineers or other teams through the workflow ID, URL, or file path, eliminating the need for offline files. The callback function also makes it easier for it to be integrated to other systems.
The created workflow can be scheduled to produce the data results at a certain hour every day. If the workflow needs to be repeated, the scheduler can automate the data processing process on an hourly, daily, or weekly basis.
Low-code service builder
At Coupang, multiple SCM systems are working together to make sure the Coupang fulfillment centers run smoothly. To manage each system, the team in charge builds servers, batch systems, and systems settings.
However, as the number of business domains continued to grow, each system had similar purposes but required their own maintenance and updates. Especially systems of domains such as customer demand, order fulfillment distribution, orders, and product transfers all used data from one another to operate, resulting in similar code across all systems. With the exception of the main algorithm, the data extraction, processing, and sharing stages were repetitive. To standardize and automate these repetitive processes, the SCM Workflow provides a low-code service builder, through which engineers focus simply on algorithm development and data processing. The low-code service builder improves engineer efficiency and system stability.
Below are some main functions of the low-code service builder.
Development platform
The development platform of the service builder provides essential tools for development such as versioning, configuration, and error logs. The engineer can debug through the web without using external tools. Also, the platform supports notifications and a real-time monitoring dashboard for each workflow.
Shared service components
The service builder includes components that are necessary for building a service. Anyone can easily build a service using these shared components, which include instant messenger notifications, REST callback, retry and more. In addition, users can implement file upload, download, copy, queue message send and receive between components, and other AWS infrastructure functions through service blocks. Using feedback from our users, we are also continually adding new components to our system.
Custom service component usage
To ensure scalability and flexibility of the SCM Workflow, users can register and use custom service components. The domain engineer can develop the key algorithm as a custom service component and register it to SCM Workflow. Then, others can use this component as a service block and integrate it to their workflows as necessary. Through this custom service definition, non-engineers can build complex services such as machine learning models. Custom service components are also easily adjusted and reusable, facilitating easier system upgrades and domain scalability.
Graphical workflow modeling
One large advantage the SCM workflow has over other commercial workflows is that it can integrate the input/output data between the canvas editor and the service components (or blocks). The decision to use service blocks and canvas was influenced largely by low-code, no-code development. In the SCM Workflow canvas editor, users can drag and drop service components and easily connect or combine them. In addition, during runtime, all the service blocks in the workflow coordinate with the input/output data of the other service blocks and can use this data as input parameters. This is how the workflow can work as a single program. Other programming components such as conditional statements, iterations, parallelization can be applied to workflows to produce a more complex internal application.
Other features
In addition to the no-code data builder and low-code service builder, the SCM Workflow has the following features.
Scalability
The architecture of the SCM Workflow is scalable and safe. It can not only access and query various data sources safely, but it can also be integrated to other system jobs that the user is working on. However, to avoid becoming a single point of failure, the SCM Workflow is divided into three layers. Each layer is a collection of clusters that can be scaled out as necessary.
Secure and independent jobs
The SCM Workflow engine creates and manages models and operates each instance that the model is running on. A running workflow instance has its own unique status, and the engine uses the user defined configuration to process the job chronologically, in parallel, or in repetition. The scheduler inside the cluster can repeat the job as many times as the user needs. A task that may take a large amount of time or resources runs independently of other tasks according to asynchronous requests. The workflow engine monitors the job’s status and acts according to the user configuration. To process numerous workflows running concurrently in a secure manner, the workflow engine and the workers were designed as independent entities.
Shared data processing service
The SCM Workflow processor has its own data processing service. It can extract data from various resources that the team manages and provide that data as a shared service component to the engine. Users can work with data sources without additional steps. Furthermore, the processor has data processing utility functions such as file upload, download, merge and more. These practical functions help the user create a full service.
Coupang Workflow web interface
Users can access the SCM Workflow through a web interface to create workflow models, schedule them, access execution histories, and more. Using the Workflow web canvas, users can intuitively configure service components and connect them to specify the order execution. Through the simple and easy-to-use interface, even business users unfamiliar with coding can create a workflow on their own.
Workflow modeler
After execution, users can check the success or failure of the job. The interface also shows you specific details about which process took the longest time. If the execution has failed, users can check the error log for debugging.
Workflow manager
Each team or project can create a group folder through the workflow manager. The group folder has permissions settings that limit model creation, editing, and deletion by user. We are also planning to develop a personal workspace where users can experiment with various production workflows.
Workflow scheduler
For execution automation, the scheduler allows the user to set a time for the job to be run. For business users unfamiliar with code, the scheduler also provides an interface that uses cron expressions to create scheduled jobs.
Workflow dashboard
Through the dashboard, users can monitor the status of the workflow engine. Although users can receive instant messenger notifications about the status of each individual instance, the dashboard allows them to check the status of all services currently run by the team in real-time.
Still we’re moving forward
The SCM Workflow is perpetually being updated with new components and systems, which in turn are supported by new platform functions.
The vision for the SCM Workflow is for it to be able to migrate small team projects into individual services outside of the workflow. Furthermore, we want to strengthen data processing to facilitate simple Excel level jobs to large-scale data processing. Lastly, we want to ensure that business users can share and integrate SCM Workflow services with other users and systems.
We want to develop the SCM Workflow platform into a tool that can fulfill the needs of engineers and business users alike.
If the SCM Workflow sounds like an interesting system that you can contribute to improving, check out our open positions.
