DEVOURED
dbt Core v2 is here: still open source, now rebuilt for what's next
dbt Core v2.0 open-sources the Rust-based Fusion engine runtime under Apache 2.0, promising faster parsing, Parquet artifacts, and a unified foundation with Fusion.
What: dbt Core v2.0 alpha release is now built on the same Rust-based runtime as dbt Fusion, with much of Fusion's code open-sourced under Apache 2.0. This brings significant parse time improvements, new Parquet artifacts, revamped local documentation, ADBC-powered adapter building, and simpler installations, while Fusion remains the recommended binary for most users.
Why it matters: This move signifies dbt Labs' commitment to a more performant and unified dbt framework, leveraging Rust and modern data technologies, and clarifying the roles of dbt Core (fully open-source) and dbt Fusion (enhanced binary) in the ecosystem.
Takeaway: If you use dbt, consider migrating to dbt Core v1.12 first and running `dbt parse --use-v2-parser` to prepare for the v2.0 upgrade, or install dbt Fusion for the best free CLI experience.
Deep dive
- Unified Foundation: dbt Core v2.0 merges with the dbt Fusion engine's Rust-based runtime, placing much of the previously ELv2-licensed Fusion code under the Apache 2.0 license.
- Key Features: Includes significant parse time improvements for large projects, a tighter language specification, new high-performance Parquet artifacts for data, a revamped local documentation experience, streamlined adapter building via ADBC and Arrow, and simpler installations by reducing Python virtual environment complexities.
- Distributions: dbt Fusion will continue as the recommended precompiled binary for most users, offering additional free and premium features, while dbt Core v2.0 serves teams requiring fully open-source code or custom builds.
- Strategic Shift: This change aims to consolidate development efforts on a single, more performant Rust-based engine, ensuring that all users benefit from ongoing innovations.
- Backward Compatibility: The existing Python-based dbt Core v1.x will still be available, but future capabilities will increasingly require upgrading to v2.x.
- Migration Path: Users are advised to upgrade to dbt v1.12 first, which includes the Fusion-powered parser for compatibility checks, before moving to v2.0.
Decoder
- dbt (data build tool): An open-source framework that helps data analysts and engineers transform data in their warehouse by writing SQL SELECT statements.
- dbt Core: The open-source command-line interface for dbt.
- dbt Fusion: An enhanced version of the dbt CLI, previously under ELv2 license, that includes performance improvements and additional features, now sharing its Rust runtime with dbt Core v2.0.
- Apache 2.0 license: A permissive free software license by the Apache Software Foundation.
- ELv2 license: The Elastic License v2, a non-open source license often used for software where commercial offerings exist.
- Parquet artifacts: Data stored in the Apache Parquet column-oriented data file format, known for efficient data compression and encoding schemes.
- ADBC (Arrow Database Connectivity): A specification for a set of high-performance APIs for database connectivity using Apache Arrow for data transfer.
Original article
Full article content is not available for inline reading.
Explore all 55 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi