Apache DolphinScheduler

About Apache DolphinScheduler

Apache DolphinScheduler is an open-source, distributed workflow orchestration platform designed to manage complex data pipelines and task dependencies. The project was initiated in 2017, originally named EasyScheduler, by the data services company Analysys to address challenges in big data ETL processes. It was later open-sourced and donated to the Apache Software Foundation (ASF) in August 2019, where it entered the ASF Incubator. The project graduated as an Apache Top-Level Project in April 2021, a journey spearheaded by individuals including Lidong Dai and Wei Guo, the co-founder and CTO of Analysys respectively.

The platform operates as a distributed, cloud-native system, serving enterprises that require robust data workflow scheduling. Its client base includes companies like IBM, Cisco, and Walmart, which utilize it for scenarios ranging from ETL processes and machine learning workflows to data backup and AI orchestration. As an open-source Apache project, it is freely available and sustained by a global community of contributors, with commercial support and enterprise features provided by companies like WhaleOps, which was founded by several of the project's core members.

DolphinScheduler is distinguished by its powerful visual interface, which allows users to create, manage, and monitor workflows through a drag-and-drop Directed Acyclic Graph (DAG) editor. This low-code approach is designed to be accessible to non-developers, such as data analysts. The system's architecture is decentralized, featuring multi-master and multi-worker nodes, which ensures high availability and horizontal scalability, capable of processing tens of millions of tasks daily. The platform supports a wide array of task types, including Shell, Python, SQL, Spark, Flink, and integrations with machine learning frameworks like MLflow and SageMaker. It offers multi-tenancy for resource isolation, version control for workflows, and flexible deployment options, including Standalone, Cluster, Docker, and Kubernetes. The system also includes native cluster monitoring, alerting functions, and role-based access control to ensure secure and stable operations.