DolphinScheduler 3.3.0 Version Updates Overview

Apache DolphinScheduler is about to release version 3.3.0, which will bring a series of important updates and improvements. In the recent community online user-sharing session in May, PMC member Wenzhen Ruan introduced the main updates and improvements that the 3.3.0 version will bring and explained how to participate in the community.

Features of DolphinScheduler

DolphinScheduler is an open-source project mainly used for workflow orchestration, execution, and management. It has four key features:

• Workflow as the basic unit: All functionalities are designed around workflows.
• Low-code platform: Workflows can be managed with little or no coding.
• Scalability without data migration: Server nodes can be scaled in or out without migrating data.
• Plugin architecture.

Workflow as the Basic Unit

Workflows can include tasks and sub-workflows, and each task must belong to a workflow. Workflows can depend on other workflows and are the smallest execution units. Execution of workflows can be triggered manually or automatically.

Low/No-Code Operations

You can define and manage workflows through a drag-and-drop interface on the UI portal or use OpenAPI or PyDolphinScheduler for related operations.

Scalability Without Data Migration

• Metadata is stored in a shared relational database, allowing system expansion without data migration.
• Services automatically register to the cluster when launched and transfer running tasks when taken offline without manual intervention.

Plugin Architecture

Components other than the workflow engine are implemented as plugins:

• Alert plugins: Email, HTTP, WeChat, Slack, etc.
• Task plugins: Shell, SQL, Spark, K8s, etc.
• Data source plugins: MySQL, PG, Oracle, etc.
• Support for Zookeeper, JDBC, ETCD as the registry center.
• Support for HDFS, S3, OSS storage systems.

Now, let’s look at the new changes in version 3.3.0.

Overview of 3.3.0 Version Updates

Architecture Updates

The overall architecture and deployment method of version 3.3.0 have not changed significantly. It still consists of three core services: API (for metadata management and authorization), Master (for workflow orchestration and execution), and Worker (for task execution and management). Most core components are loaded as plugins.

Major Changes in the New Version Include:

New Task Plugin Interface

Previous versions had issues with the task plugin definitions:

• Methods in task plugins did not match lifecycle events, missing methods like pause and fault tolerance controls.
• The task plugin interface was overly complex, containing methods from many sub-plugins, making it bulky.

Version 3.3.0 will introduce a new task plugin definition interface (V2) to manage the lifecycle of task plugin instances. This includes states like run, pause, kill, fault tolerance, success, and failure.

The new interface will be clearer and more streamlined.

• ITaskApplicationContext: Represent the third part task instance info, e.g. processId, applicationId
• ITaskListenerManager: Can register listener used to listen to the task info changes

Improved fault tolerance

MasterFailover:

• Failover the workflows which hold by the crashed master
• Insert workflow failover command
• Workflow failover will take over the running tasks from work server

WorkerFailover:

• Failover of the tasks hold by the crashed worker
• Dispatch a failover task to a new worker
• Failover operation might be different in different task plugin
• Failover support exactly-once、at-least-once、at-most-once

Parameter priority

Previous versions had multiple parameter types with inconsistent priorities. The 3.3.0 version optimizes parameter priorities:

• Unified parameter prioritization, following the nearest principle, e.g., startup parameters > task definition parameters > workflow definition parameters.
• Startup parameters have the highest priority and can override others, solving inconsistencies and runtime modification issues.

Decoupled Workflow Triggers

Previous trigger methods had limitations:

• Triggers were strictly bound to the database.
• No schema for triggers, making expansion difficult.

The new version decouples workflow triggers, allowing triggering through messaging middleware:

• Provides a standard Trigger Schema for easy trigger expansion.
• Introduces a CommandEngine component to detect commands to be processed.
• Command repositories are no longer limited to relational databases, supporting consumption from streaming systems like Kafka.

Event Thread Model Improvements

Previous versions had two event thread pools, causing coordination difficulties and potential event order loss.

The new version optimizes this by:

• Merging the two event thread pools; all events in a workflow are processed in FIFO order.
• Offering more event metrics and query interfaces for better monitoring and management.

Support for Multiple Hadoop/S3 Clusters

Previously, configuring HDFS/S3 required config files, making it impossible to add clusters at runtime.

The new version changes this:

• Allows configuration in metadata storage.
• Supports multiple clusters.
• Allows adding new clusters at runtime.
• Different tasks can use different clusters.

AlertServer Improvements

AlertServe achieves high availability through HA, using a single thread to send alerts. This results in the need to use additional machines to deploy AlertServer.

The new version allows embedding AlertServer into the API server, adopting a multi-threaded model for sending alerts, increasing efficiency.

For more details on the new features and optimizations in Apache DolphinScheduler 3.3.0, please stay tuned to the official update and GitHub release.

How to Participate in the Community

Email: The preferred way to discuss issues, including user and developer mailing lists for discussing usage issues, design proposals, new ideas, and voting for new releases.

• User Group Email: users@dolphinscheduler.apache.org
• Developer Group Email: dev@dolphinscheduler.apache.org

GitHub: The easiest way to participate in the project, including multiple GitHub repositories for DolphinScheduler.

• Introducing new features through DSIP is the only channel.
• Reporting and fixing bugs are equally important.
• Keep PRs as simple as possible.
• GitHub Repositories:
• DolphinScheduler
• DolphinScheduler Website
• DolphinScheduler Operator
• DolphinScheduler SDK Python

Notes:

• Introducing new features through DSIP is the only channel.
• Reporting and fixing bugs are equally important.
• Keep PRs as simple as possible.

Conclusion

Apache DolphinScheduler 3.3.0 brings numerous significant updates, enhancing system stability, scalability, and usability. We encourage community members to actively participate and contribute to the project.