Behind the Scenes of an Apache Top-Level Project: How SeaTunnel Became the First Open-Source Data Integration Tool Led by Chinese Developers | BlueRun Ventures Headlines

Earlier this month, we shared the good news that SeaTunnel, under WhaleOps, had become a top-level Apache project. Five years, nearly 250,000 lines of code, and over 200 contributors — how exactly was SeaTunnel forged? Today's article comes from Apache incubator mentor William Guo and WhaleOps co-founder Lidong Dai, who together reveal how SeaTunnel went from nothing to open source to the world stage.

On June 1, 2023, Children's Day, Apache SeaTunnel — the first open-source data integration tool led by Chinese developers — officially graduated from the Apache Software Foundation incubator to become a top-level project. After 18 months of incubation, the project had finally ripened. But much like a newborn's first cry, Apache SeaTunnel's new journey was only just beginning.

From the earliest Waterdrop to today's Apache SeaTunnel;

From a real-time data processing system to a new-generation, one-stop, high-performance, distributed, massive-scale data integration solution;

From the first line of code in January 2018 to today's 245,000 lines;

From fewer than 10 contributors to 200+;

From struggling to find the first user to thousands of enterprises running it in production;

From searching for mentors to smoothly becoming an Apache top-level project.

...

Core members of the Apache SeaTunnel community will recount these dramatic ups and downs, presenting the story behind its open-source journey along a chronological thread.

01

The Birth of Apache SeaTunnel

We've long faced numerous challenges in data processing, one of which is the need to support seamless integration and high-speed synchronization across numerous data sources. When we surveyed existing data integration tools on the market, we found most supported very limited data sources — often you'd find a connector for an upstream source but not for the downstream one. And when dealing with large-scale data volumes, performance was typically poor, operations were complex, and flexibility was lacking. So we conceived the idea of building an open-source data integration tool!

After the core team refined it, Apache SeaTunnel was born. It supports not only hundreds of data sources (Database/Cloud/SaaS), but also real-time CDC and batch synchronization of massive data, stably and efficiently syncing trillions of rows.

Beyond basic data reading and writing, Apache SeaTunnel distinguishes itself from ordinary data integration tools with:

• Decoupling from Spark and Flink engines, with its own engine Zeta designed specifically for data integration scenarios — faster, more stable, and more resource-efficient. This means Apache SeaTunnel simultaneously supports three execution engines: Spark, Flink, and its self-developed Zeta Engine;
• A more intuitive, easy-to-operate web interface;
• Support for 100+ connectors, with rich data processing types to meet production needs;
• Unique Checkpoint functionality design, enhancing data storage capabilities, among other features.

This enables Apache SeaTunnel to achieve:

• Support for hundreds of data sources, faster transmission speeds, and high accuracy;
• Reduced complexity — API-based connectors are compatible with offline synchronization, real-time synchronization, full synchronization, incremental synchronization, CDC real-time synchronization, and other scenarios;
• Drag-and-drop and SQL-like language interfaces that save developers more time, providing visual job management, scheduling, running, and monitoring capabilities, accelerating integration with low-code and no-code tools;
• Simple maintenance, supporting standalone and cluster deployment. If choosing Zeta engine deployment, no dependency on big data components like Spark or Flink is required.

Though SeaTunnel now has so many capabilities, going back two years, when it was still called Waterdrop, its positioning was simply to make Flink and Spark easier to use. So the entire architecture was built on top of Spark and Flink — which led to the community's first major debate: connectors must be made independent of any specific engine.

02

Why Must Connectors Be Engine-Independent?

First, let's look at what connectors do. Connectors are responsible for bridging specific upstream and downstream data sources, and are a critical component of data integration. Waterdrop's architecture at the time essentially imported Spark and Flink connectors, using their native APIs. This required developing separate codebases, and since batch and streaming APIs were still different in early versions, the same data source needed two sets of code to achieve both batch and stream synchronization. And considering Spark and Flink's major version compatibility issues, the development and maintenance costs were simply too high.

So in early 2022, the community initiated discussions to refactor connectors, aiming to define SeaTunnel's own connector API decoupled from specific engines, independent of any engine's API, truly achieving batch-stream unification. The same data source would need only one set of code to run on both Spark and Flink engines.

In the early stages of discussion, many opposed the idea, feeling that Flink and Spark were mature engines and strong dependence on them wasn't problematic. Some contributors felt we should abandon Spark and fully rely on Flink, building out functionality on that foundation. And refactoring the connector API meant starting from scratch on the previous 50+ connectors.

But after extensive exchange and discussion with numerous industry experts, the community soon confirmed that engine-independent connectors were a must. "You can't dance in shackles." The new API would make connector development simpler, and those older connectors could quickly be supported under the new architecture.

It turned out that once this goal was set, the community spent one month designing the new connector API. Many enthusiastic contributors joined in, and we achieved support for 100+ connectors in just four months — an unimaginable speed. And the new API truly delivered multi-engine support.

After achieving this connector-engine independence, "Why not just build a new engine focused specifically on data integration, once and for all?" SeaTunnel PMC Chair Jun Gao casually remarked — a comment that ignited an unstoppable wave of enthusiasm among community contributors.

03

Why Build a Self-Developed Engine?

"What, build our own engine?" Hearing this news about developing a proprietary integration engine, the community exploded, immediately launching an unprecedentedly heated debate over whether building an engine from scratch was truly necessary.

The main points of contention were several:

• From the simplicity and usability perspective: both Spark and Flink require enterprises to have big data platforms, which is a significant technical burden for SMEs. People needed a simpler, lower-cost engine to reduce SeaTunnel's barrier to entry.
• From the performance perspective: Spark and Flink were born for computation. They mainly address the "T" in ETL architecture, while data integration primarily solves the "EL" in ELT. Features like Join, Aggregation, and windowing calculations aren't the focus of data integration. A data integration engine should focus on integration, not computation — all code optimization and architectural design should aim to improve job performance and stability. So we needed an engine specifically designed for integration scenarios, with excellent performance, extreme stability, and lower resource consumption. Especially when syncing many tables, could we complete real-time synchronization of these tables with minimal resources (say, 1 CPU core)?
• From business scenarios: Flink/Spark themselves cannot satisfy CDC multi-table synchronization, whole-database synchronization, DDL change synchronization during CDC, and other features. Supporting these would require modifying Flink/Spark source code. We couldn't be certain whether these features would be accepted by the Spark/Flink communities, as they diverge from those projects' main direction (the "T" in ELT, focused on data warehouse computation). If not accepted, we'd need to maintain our own Spark/Flink version — nearly impossible. From this perspective, SeaTunnel building its own integration engine was also a necessity.

Many community contributors participated in the discussion at the time. Some felt this was reinventing the wheel. Of course, the community ultimately reached consensus and decided to begin designing and developing a specialized integration engine. I remember one contributor even declared, "I'm building this wheel, period."

Thus, the community gritted its teeth and built Zeta, an engine focused on solving synchronization domain problems. Last October, we successfully released Zeta's official version. At the time it was called SeaTunnel Engine, but people felt it needed a catchy name fitting its positioning.

So the community began brainstorming. After about two weeks of discussion, we chose Zeta from numerous candidates. Zeta is currently the fastest observable planet in the universe, and many users affectionately call it "Ultraman Zeta" — the strongest Ultraman power in the universe, guarding the faith of light together! We hope the "Ultraman Zeta" engine makes integration simpler, more efficient, more stable, and more resource-efficient.

04

Starting Incubation: Why Join the Apache Software Foundation?

Actually, before changing its name from Waterdrop, Apache SeaTunnel already had plans to join the world's largest open-source organization — the Apache Software Foundation. William Guo (SeaTunnel Mentor) said when SeaTunnel joined the Apache incubator:

Now that Apache Sqoop has retired, there's no particularly outstanding open-source project solving the problem of data connectivity between sources. With today's dizzying variety of data sources, if each enterprise only solved connectivity for the few sources they used, it could never address the broader need of more people using more sources. And when new sources appeared, they'd need to be rewritten from scratch. Open source is a model of "gathering sand to build a tower, embracing all rivers" — it allows every enterprise and individual to conveniently use open-source data source connectors, while also contributing connectors for sources they use back to the project. This way, a project connecting various data sources can snowball, growing ever larger, making it easier for more users to connect diverse data sources, achieving a "flywheel effect" in data integration.

Another important point: before this, some of Apache SeaTunnel's core contributors and mentors already had successful incubation experience with the open-source project DolphinScheduler, so everyone was confident and optimistic about SeaTunnel entering the incubator. While the process of entering the Apache incubator wasn't entirely smooth sailing, this prior experience meant the team wasn't starting from scratch, but proceeding methodically.

Specifically, core contributors of SeaTunnel's predecessor Waterdrop established close ties with the DolphinScheduler community in 2018. DolphinScheduler's partners had been closely watching Waterdrop, which from both code quality and future potential in data integration was a "rising star." So when Waterdrop discussed with us whether to continue together, we didn't hesitate long before investing manpower and energy into Apache SeaTunnel's R&D, and soon after drove its entry into the Apache incubator. With an open mindset, we hoped to let SeaTunnel, through open source's power, efficiently, accurately, and rapidly perform cross-source synchronization and data transformation, enabling people in multi-source scenarios to quickly and simply achieve their goals. We believed that under the guidance of the "Apache Way," Apache SeaTunnel would gain more support and accelerate the project's growth.

Entering the Apache Foundation, finding mentors is often the first and crucial step. But unlike projects that need to feel their way across the river, Apache SeaTunnel attracted global Apache incubator mentors' attention during its incubator discussion phase. The number of "mentors" far exceeded ordinary incubation projects, so much so that Apache incubator lead Justin emailed a reminder that "there can't be too many mentors." Some mentors also expressed regret on the global Apache incubator discussion mailing list — Apache incubator projects suffer from "some starving, some drowning": some projects scour everywhere for mentors to enter incubation, while others have mentors competing to join.

Soon, with the help and guidance of seven mentors — Jean-Baptiste Onofré, Kevin Ratnasekera, Willem Ning Jiang, Ted Liu, Lidong Dai, Guo William, and Zhenxu Ke — Apache SeaTunnel quickly got on track in the Apache incubator.

Jiang Ning is an open-source "veteran" who ultimately became our Champion. He is one of China's most senior Apache Members, and in 2023 was re-elected to the Apache Software Foundation Board of Directors, becoming the first Chinese person to serve consecutive terms as an Apache Foundation director.

Lidong Dai is Chair of the Apache DolphinScheduler project, with rich experience in open source. He also has deep ties with Apache SeaTunnel — many of its features were organized and designed with his help, and he helped build the Apache SeaTunnel community together. During his more than a year participating in Apache SeaTunnel's development, he went on to serve as mentor for multiple Apache incubation projects, and in 2022 was elected as an ASF Member.

Apache Members Jean Baptiste Onofré and Kevin Ratnasekera were also acquaintances from Apache DolphinScheduler's incubation period, both with rich project incubation experience.

Later, William Guo, Ted Liu, and Zhenxu Ke also joined the mentor ranks, making Apache SeaTunnel's incubation path even smoother.

To formally enter the Apache incubator, we also referenced mature projects in establishing overall code standards for Apache SeaTunnel. To connect with international norms, we also performed extensive English translation and proofreading of project documentation, and fully anglicized the Apache project website. These efforts made the Apache SeaTunnel project more standardized and "international."

Additionally, after joining the incubator, we made relatively major functional changes to the project, the most important being the R&D and release of the dedicated data synchronization engine Zeta mentioned above. This engine, which provides users with high-throughput, low-latency, strongly consistent synchronization job guarantees, was officially released in version 2.3.0 as Apache SeaTunnel's default engine. It achieved decoupling from Flink and Spark engines, allowing users to run independently without depending on Flink and Spark. Its self-governing cluster (decentralized), data caching, speed control, shared connection pools, breakpoint resumption, finer-grained fault tolerance design, dynamic shared threads, and other unique features made Apache SeaTunnel's Zeta engine unprecedentedly simple to use, more resource-efficient, more stable, and faster — capable of full-scenario data synchronization support.

05 Exploring the Apache Way

Just as joining a new company requires understanding its culture, participating in Apache open-source projects requires understanding ASF's culture — the Apache Way.

Digging deeper into open source reveals that it's not simply about open-sourcing code. Open source also concerns community management, activity, communication, culture, and more — requiring us to have a deeper understanding of the Apache Way.

Given prior experience, Apache SeaTunnel had a profound understanding of the Apache Way's importance from its early days in the Apache incubator. For example, for open-source communities, the Community Over Code philosophy must be rooted in our hearts, requiring the community to prepare and strive to lower the barrier for anyone interested in participating — even to zero barrier. This meant establishing community incentive programs, creating beginner guides, curating Good First Issues, tracking important feature progress, obtaining feedback and optimization suggestions through regular user interviews, and regularly answering community questions about the project and community.

Community contribution is not limited to code; non-code contributions can sometimes play even more valuable roles than code — such as using one's influence to raise attention for the project, writing technical and non-technical articles about the project, participating in various community-organized activities, and "representing" Apache SeaTunnel on various occasions and at various events, recommending it to more users — all are channels for community participation.

Meanwhile, Community Over Code also emphasizes openness, communication, and collaboration. Apache SeaTunnel upholds these principles, maintaining communication and mutual learning with domestic and international communities, and establishing communication with the Apache community. All discussions happen in mailing lists and Issues, with major project and community progress and plans announced through community self-media channels, keeping the community open and transparent.

From entering incubation to present, Apache SeaTunnel has held over 20 online and offline Meetups with multiple domestic and international open-source projects, including Apache Shenyu, Apache InLong, and Apache Linkis — which graduated from the ASF incubator before Apache SeaTunnel — as well as mature open-source projects like Apache Doris, IoTDB, StarRocks, TDengine, and overseas events in the United States, India, and other regions jointly held with Trino, APISIX, Shopee, and ALC Indore.

Inter-community cooperation and exchange drive open-source technology development and application. Apache SeaTunnel's collaboration with other open-source projects to jointly solve technical problems helps elevate the overall open-source ecosystem level and expand its boundaries.

With accumulated time, the community has undergone qualitative change. From mailing list discussions and GitHub data, you'll find the Apache SeaTunnel community has truly become active and diverse. The table below shows Apache SeaTunnel's community data changes during its year-plus in the Apache incubator.

As you can see, the community and contributors coexist like "fish" and "water" — more and more contributors participating in the community brings fresh life-giving water to this "fish," allowing the Apache SeaTunnel community to thrive; and the water flows faster and farther because of this big fish's constant leaping. Fish and water coexisting — that's how life continues generation after generation.

06 Graduating from the Incubator

After 18 months of incubation, the community carefully assessed Apache SeaTunnel against the Apache maturity evaluation model across seven dimensions: code, license and copyright, releases, quality, community, consensus, and independence. Finding that Apache SeaTunnel's graduation timing was relatively mature, we began preparations for graduating from the ASF incubator.

Apache Project Maturity Evaluation Model

For example, in code maturity: the community went through multiple version upgrades and new features, improving Apache SeaTunnel's performance and functionality, and further enhancing efficient synchronization and transformation capabilities between data sources. In community building: as described above, through numerous domestic and international online and offline Meetup events, the Apache SeaTunnel community provided platforms for exchange and sharing, promoting communication and collaboration among developers and expanding the open-source project's influence. Additionally, Apache SeaTunnel strengthened integration with upstream and downstream ecosystem projects such as Flink, Spark, TiDB, OceanBase, and IoTDB, promoting collaborative development between different projects and enhancing the entire open-source ecosystem's interoperability and overall performance.

Under Apache Members' guidance, Apache SeaTunnel initiated graduation discussions in the community in April, and according to ASF incubator guidance, improved shortcomings and made continuous corrections. Ultimately, Apache SeaTunnel passed its graduation vote, and on May 17, 2023, was approved by the ASF Board of Directors to join the TLP ranks as wished!

07 The Road Ahead: How Chinese Open Source Goes Global

Apache SeaTunnel's goal is "connecting ten thousand sources, synchronizing like flying" — striving to become a world-class data integration tool. In the future, it will also integrate with more upstream and downstream ecosystem projects. At the same time, it will continue to undertake the mission of promoting open-source culture, facilitating exchange and collaboration among developers, providing more platforms for open-source community development, and inspiring more people to participate in and contribute to open-source projects.

At this important moment, we call on more people to join Apache SeaTunnel's contributors!

Finally, for Apache SeaTunnel, the road to graduating from the ASF was not smooth sailing. Based on our limited experience groping in the open-source world, we'd like to express some views and suggestions on Chinese open-source ecosystem development:

Strengthen Open-Source Culture Building

In China, the dissemination and popularization of open-source culture still needs further strengthening. More developers and enterprises need to be encouraged to participate in open-source projects, promoting knowledge sharing and collaboration. At the same time, open-source awareness and understanding need to be improved, pushing open-source's widespread application in education, enterprise, and government sectors.

Improve Open-Source Project Quality and Influence

Chinese open-source projects have accumulated somewhat in quantity, but still have room for improvement in quality and influence. Attention needs to be paid to project technological innovation and practicality, encouraging more high-quality projects to emerge. At the same time, active participation in international open-source communities, cooperation and exchange with international projects, is needed to improve project visibility and influence.

Strengthen Open-Source Community Building and Governance

Open-source communities are key to open-source project success. Sound community governance mechanisms need to be established, promoting community member participation and contribution. Good communication and collaboration platforms need to be provided, encouraging exchange and collaboration among developers. Additionally, training and support for community members needs to be strengthened, improving their technical and management capabilities.

Strengthen Open-Source and Industry Integration

Open-source technology plays an important role in driving industrial innovation and development. Strengthening the integration of open-source technology with various industries is needed, promoting open-source technology application in enterprises and public services. At the same time, actively cultivating open-source technology ecosystems, promoting collaborative development of open-source projects and industry chains.

In summary, Chinese open-source has achieved some results, with many domestic open-source projects gaining broad international recognition and use, but much work remains to be done. Through strengthening open-source culture building, improving project quality and influence, strengthening community building and governance, and strengthening open-source and industry integration, Chinese open-source ecosystem development can be further promoted, driving technological innovation and industrial upgrading.

▶ More Exciting Content ◀

Originating in Silicon Valley, BlueRun Ventures was established in 2005 as a venture capital firm focused on early-stage startups.

Currently, BlueRun Ventures manages multiple USD and RMB dual-currency funds in China, with assets under management exceeding RMB 15 billion, making it one of the largest early-stage funds domestically. Its investment stage focuses on Pre-A and Series A, covering hard tech and innovative interaction, enterprise technology, new consumption, and healthcare. It has cumulatively invested in over 150 startups, including Li Auto, Waterdrop, QingCloud, Guazi.com, Qudian, Songguo Mobility, Ganji.com, Energy Monster, Yuntu Semiconductor, Machenike, Cloud Saint Intelligence, Anxin Wangdun, and BioMap.

BlueRun Ventures has been ranked #1 in Zero2IPO's "China Early-Stage Investment Institutions Top 30" and ChinaVenture's "China Best Early-Stage Venture Capital Institutions Top 30," and was named among Preqin's Top 10 global venture capital fund managers with consistently high returns.

Additionally, BlueRun Ventures has for consecutive years received honors from Forbes China, 36Kr, Cyzone, Caixin Media, CBNweekly, Jiemian, and other media institutions as "China's Best Early-Stage Firm of the Year," "China's Top Venture Capital Institution," "Early-Stage Investment Institution Most Popular Among Entrepreneurs of the Year," and "Most Influential Early-Stage Investment Institution of the Year."