Streamlining Logistics With Automated Dispatching & Transporter Ranking (Part I)

Published in

Kargo Technologies

11 min readMay 2, 2023

Kargo Tech is an Indonesian logistics startup that uses technology to connect shippers and truckers, aiming to reduce logistics costs and improve efficiency in the trucking industry. The company offers a range of solutions for both shippers and truckers, including instant quotes, online payments, real-time tracking, and access to quality loads. Despite these benefits, manual management of contracts and dispatching can be a challenging and time-consuming process that leads to inefficiencies and inaccuracies.

The matching engine’s output is displayed by the user interface.

To address this issue, Kargo Tech is proposing a solution that involves using a matching engine to generate recommendations for dispatching based on transporter rankings. Additionally, the company plans to introduce more flexibility and customization into the tiered dispatching system and simplify contract and dispatch management by automating manual tasks. By implementing these changes, Kargo Tech aims to improve dispatching efficiency and accuracy, leading to better outcomes for both shippers and transporters. Overall, Kargo Tech’s solutions are designed to improve the logistics industry in Indonesia and make the process of shipping goods more efficient and cost-effective.

Article’s Objectives

This article discusses the importance of matching engines in the logistics industry, the challenges faced in finding the right match, and how to build a matching engine using Elastic Search. It also covers the benefits of using Elastic Search, a step-by-step guide to building a matching engine, real-world examples of how they’ve improved logistics operations, and concludes by emphasizing their importance. The article targets software engineers, logistics professionals, entrepreneurs in logistics, and students studying software engineering, computer science, and logistics.

Article Goals

To provide an overview of the importance of matching engines in the logistics industry
To discuss the challenges faced by shippers and transporters when finding the right match
To explain how Elastic Search can be used to develop a matching engine
To highlight the benefits of using Elastic Search for developing a matching engine
To provide a step-by-step guide on how to develop a matching engine using Elastic Search
To showcase real-world examples of how matching engines have improved logistics operations
To conclude by emphasizing the importance of matching engines and the benefits they bring to the logistics industry.

Intended Audience

Software engineers are interested in the technical aspects of building a matching engine using Elastic Search.
Logistics professionals in the transportation industry are interested in how matching engines can streamline operations.
Entrepreneurs in the logistics industry seek to optimize transportation processes through new technologies.
Students in software engineering, computer science, and logistics looking to learn about technology in the transportation industry.

Problem and Proposed Solution

In the world of transportation logistics, manual maintenance of tiered dispatching can be a challenging and time-consuming process for operations teams. This involves creating contracts for specific shippers, routes, and tonnages, which requires a significant amount of manual effort. Additionally, managing these contracts can be difficult, including manual input of transporter prices and flexible tonnage and shipper information. Contract expiration dates and updates must also be manually maintained, which can be overwhelming. To address these issues, a proposed solution is to introduce a new approach to setting tiered dispatch that involves dynamically setting the transporter tier based on the ranking in a matching engine.

The proposed solution also includes using a matching engine to calculate rankings based on criteria such as transporter reliability, cost, and capacity. This ranking system can be used to automate manual tasks and generate recommendations for dispatching based on the most suitable transporters. Furthermore, there should be more flexibility and customization in setting dispatch tiers, allowing for tailored solutions for specific shippers and transporters. Additionally, the solution should simplify the process of maintaining tiered dispatching and optimize it for faster and more accurate dispatching of goods.

Automating contract maintenance tasks such as inputting transporter prices and updating expiration dates, implementing a user-friendly interface for managing contracts and dispatching, and using the matching engine to generate recommendations for dispatching based on the most suitable transporters and their tier are all key components of this proposed solution. Ultimately, implementing these changes will improve the efficiency and accuracy of dispatching, providing better outcomes for shippers and transporters. Kargo Tech can benefit from this solution, improving its logistics operations and providing better services to its customers.

Problem:

Manual maintenance of tiered dispatching can be a challenging and time-consuming process for operations teams.
Contracts have to be created for specific shippers, routes, and tonnages, which requires a significant amount of manual effort.
The process of managing contracts can be difficult, including manual input of transporter prices and flexible tonnage and shipper information.
Contract expiration dates and updates must be manually maintained, which can be overwhelming.

Proposed Solution:

Introduce a new tiered dispatch approach using a ranking system from a matching engine based on criteria such as reliability, cost, and capacity.
Automate manual tasks and generate recommendations for dispatching using the ranking system.
Customize dispatch tiers and rules for tailored solutions and fine-grained control.
Simplify maintain and optimize dispatching for faster and more accurate delivery.
Automate contract maintenance and provide a user-friendly interface.
Create a tiered dispatch system considering multiple factors and use the matching engine to generate recommendations.
Improve transportation logistics with a streamlined and efficient process for managing contracts and dispatching goods.

New Approach to Setting Tiered Dispatch with a Matching Engine and Automated Recommendations

Overall, the proposed solution will automate manual tasks, generate recommendations based on a matching engine, introduce more flexibility and customization into the tiered dispatching system, and simplify the management of contracts and dispatching. These changes will improve the efficiency and accuracy of dispatching, ultimately providing better outcomes for shippers and transporters. Kargo Tech can benefit from implementing this solution to improve its logistics operations and provide better services to its customers.

Overview of the System Requirements and Constraints

This system will leverage a matching engine to generate recommendations based on transporter rankings, enabling the company to optimize its dispatching process. The system should be able to handle a large volume of data, including mid-mile transporter prices and shipping requirements. The matching engine will compute scores based on requirements to generate accurate recommendations, leading to improved efficiency and accuracy in dispatching. Overall, the proposed system is designed to improve the company’s operations, ultimately resulting in better outcomes for shippers and transporters.

Product Requirements and Constraints

The given list represents the following product requirements and constraints:

Automation of manual tasks: The system should automate manual tasks to simplify the management of contracts and dispatching.
Matching engine with recommendation generation: The matching engine should generate recommendations for dispatching based on transporter rankings.
Tiered dispatching system: The tiered dispatching system should offer more flexibility and customization, allowing transporters to be dynamically ranked based on their performance.
Ability to handle a large volume of data: The system should be able to handle a large volume of data, including mid-mile transporter prices and shipping requirements.
Accurate recommendation generation: The matching engine should compute scores based on requirements to generate accurate recommendations.

Technical Requirements and Constraints

These constraints include:

Short-term execution effort: The BTMM squad plans to implement a part of the roadmap in Q1 2023, so the technical strategy should focus on delivering results within a relatively short timeframe.
Long-term plan: While the short-term effort is important, the technical strategy should also align with the long-term plan for the automatic transporter dispatch system.
Order matching complexity: Matching orders with transporters can be a complicated process, so the technical strategy should include advanced algorithms and machine learning techniques to analyze and match the shipper and transporter data accurately.
Scalability and flexibility: The technical strategy should be designed to accommodate future changes and enhancements to the automatic transporter dispatch system, using a modular and component-based design that allows for easy integration of new features and functions.
Real-time data: The automatic transporter dispatch system should take into account real-time traffic and weather data, driver availability, and vehicle routing optimization to ensure efficient and timely delivery of goods.
Decision-making algorithms: The dispatch system should include decision-making algorithms that can analyze and prioritize delivery orders based on various factors, such as distance, time, and cost.

The given database schema represents the following requirements:

The table midmile_transporter_pricings is used to store pricing information for transporters based on various parameters like the origin and destination city/district, vehicle type, minimum and maximum quantity, amount, and status.
The table global_requirement_categories is used to store the categories of requirements that can be used for shipping, such as weight, volume, or special handling requirements.
The table global_requirements is used to store specific requirements within each category, such as weight range or type of special handling required.
The table shipping_requirements is used to store the actual shipping requirements for a given company, along with the value and priority level assigned to each requirement. These requirements can be based on the categories and specific requirements defined in the global_requirement_categories and global_requirements tables.
The table transporter_matching_logs is used to store information about the matching engine, including the matching score, metadata, and transporter price for a specific job.

The database schema for shipping requirements and pricing information.

These database schemas represent the requirements and constraints necessary for implementing a matching engine and managing contracts and dispatching in a logistics company. The tables are designed to store pricing and requirement information in a structured manner, allowing for easier filtering and searching of data. With this information, the matching engine can generate recommendations based on transporter rankings, and the system can introduce more flexibility and customization into the tiered dispatching system. Additionally, automating manual tasks and simplifying the management of contracts and dispatching can improve efficiency and accuracy, ultimately providing better outcomes for shippers and transporters.

Profiling Shipper and Transporter Data

The system is designed to manage pricing and requirements for shipping jobs between shippers and transporters. The data is stored in several tables, including midmile_transporter_pricings, global_requirement_categories, global_requirements, shipping_requirements, and transporter_matching_logs. These tables store information about pricing, shipping categories, specific requirements, and matching logs.

Shipper Requirements

The shipper requirements are stored in the shipping_requirements table, which includes information about the specific requirements needed for a particular job, such as weight, volume, or special handling requirements. The shipper can assign a value and priority level to each requirement, which helps the matching engine to find the most suitable transporter for the job. The global_requirement_categories and global_requirements tables define the categories and specific requirements that can be used by shippers for their jobs.

Transporter Requirements

The transporter requirements are stored in the midmile_transporter_pricings table, which includes pricing information for transporters based on various parameters, such as the origin and destination city/district, vehicle type, minimum and maximum quantity, amount, and status. The matching engine uses this information to find the most suitable transporter for a particular job. The transporter_matching_logs table stores information about the matching engine, including the matching score, metadata, and transporter price for a specific job.

Matching Engine Development

To develop an effective matching engine with elastic search integration, it is necessary to establish a set of specific criteria and schema. Testing the performance of the engine can be done by using an example data set. To retrieve relevant information from elastic search, a well-defined query must be executed.

Define Criteria

The Elasticsearch document was created with a focus on transportation services, and the criteria were carefully designed for this purpose. To enable identification of the transportation service provider, a “transporter_ksuid” field was included in the document. For specifying transportation requirements, a “requirements” field was utilized, which contains key-value pairs. The keys serve as unique identifiers for the specific requirements, while the values indicate the quantity needed. The “pricings” field was also part of my criteria and contain an array of pricing information for the transportation service. Each pricing object in the array includes important information such as the service fee, origin and destination cities, minimum and maximum quantity of goods that can be transported, and identifiers for the origin and destination districts. Some of the pricing objects also include an identifier for a shipper project, which may indicate that the transportation service is being provided for a specific project.

Define The Schema

Elastic Search Index Schema for Transporter Pricing and Requirements

The schema of the elastic search document includes a transporter_ksuid object that contains important fields for representing transporters with their pricings and requirements information.

transporter_ksuid: A string field that represents the unique identifier of the transporter.
requirements: An object field that contains key-value pairs where the key is a string field representing a unique identifier for a requirement, and the value is an integer field representing the quantity required.
pricings: A nested object field that contains an array of pricing objects, each object representing pricing for a certain quantity range.
amount: An integer field representing the amount of the pricing.
origin_city_id: A string field representing the unique identifier of the origin city.
destination_city_id: A string field representing the unique identifier of the destination city.
shipper_project_ksuid: A string field representing the unique identifier of the shipper project (optional).
minimum_quantity: An integer field representing the minimum quantity for the pricing range.
maximum_quantity: An integer field representing the maximum quantity for the pricing range.
origin_district_id: A string field representing the unique identifier of the origin district (optional).
destination_district_id: A string field representing the unique identifier of the destination district (optional).
ksuid: A string field representing the unique identifier of the pricing.

Define The Query

The following information is an outline of an Elasticsearch query for retrieving data from an index. This query uses a function_score query type with two subqueries, filters on specific fields in the index, and includes script score clauses and script fields.

A diagram that presents a visual representation of the query.

The first subquery is a bool query that filters on a specific field in the requirements object, and should clauses containing two bool queries with multiple range filters on different fields in the requirements object. This subquery is designed to retrieve data from the requirements object that meets specific conditions and filters. The scores from this subquery are multiplied by a constant that has been defined using a script_score clause.

The second subquery is a match_all query with a script_score clause that calculates a score based on a custom script that considers various fields and conditions in the pricings object. This subquery retrieves data from the pricings object that meets specific conditions and calculates a score based on a custom script.

The results also include two script fields, transporter_price, and transporter_price_ksuid, which use custom scripts to extract specific values from the pricings object based on the query parameters. These script fields are designed to retrieve specific values from the pricings object that meet specific conditions and filters.

Result of Executed Query

Elasticsearch Query Results and Hit Metadata

For each hit (document), the “_index”, “_type”, “_id”, and “_score” fields provide information on the index name, type, ID, and relevance score, respectively. The original document content is stored in the “_source” field, which includes the “requirements”, “pricings”, and “transporter_ksuid” fields.

The “hits” object has three key fields:

“total” shows the total number of hits found.
“max_score” displays the highest relevance score among all hits.
“hits” contains an array of individual hits (documents) with their corresponding “_index”, “_type”, “_id”, “_score”, and “_source” fields. Additional fields returned by the query for each document, such as “transporter_price” and “transporter_price_ksuid”, are stored in the “fields” field.