DuckDB hf extension — read and write Hugging Face assets with hf://

A native C++ DuckDB extension to read/write to hf:// buckets, datasets, etc.

• Read: using the Hub's /resolve/ endpoints (range requests + a shared read-ahead cache);
• Write: using the Hub's commit API (same as huggingface_hub) and Xet for efficient uploads via deduplication.
• Targets DuckDB 2.0.0 (v2.0.0, Cyanoptera).
• Published at hf.co/duckdb-extensions.

INSTALL hf FROM hf_repo; LOAD hf FROM hf_repo;   -- one-time setup: see Installation

SELECT * FROM 'hf://datasets/lhoestq/demo1/*.parquet';
SELECT * FROM hf_ls('hf://datasets/lhoestq/demo1');
COPY (SELECT * FROM range(10) t(i)) TO 'hf://datasets/me/demo/out.parquet';
COPY (SELECT * FROM read_parquet('hf://datasets/me/demo/*.parquet'))
  TO 'hf://datasets/me/demo/partitioned' (FORMAT PARQUET, PARTITION_BY (kind));

Installation

Make sure you are using DuckDB 2.0:

SELECT version();   -- v2.0.0        <- the identifier, and the directory name a repository serves;
                                     --   a pre-2.0 client answers v2.0.0-alpha47600 instead, and wants
                                     --   a different binary: one directory per identifier is the mapping
PRAGMA platform;    -- linux_arm64         <- one of linux_amd64, linux_arm64, osx_amd64, osx_arm64,
                                            --   windows_amd64, windows_amd64_mingw,
                                            --   linux_amd64_musl, linux_arm64_musl

Install the extension:

• in DuckDB CLI:

SET allow_extension_repositories = 'allowed';   -- the gate for adding *and* installing from a repository
CREATE OR REPLACE EXTENSION REPOSITORY hf_repo
  WITH PREFIX 'https://hf.co/buckets/duckdb-extensions/builds/resolve';
INSTALL hf FROM hf_repo;

• in Python:

import duckdb
con = duckdb.connect(config={"allow_extension_repositories": "allowed"})
con.execute("CREATE OR REPLACE EXTENSION REPOSITORY hf_repo WITH PREFIX "
            "'https://hf.co/buckets/duckdb-extensions/builds/resolve'")
con.execute("INSTALL hf FROM hf_repo");

Usage

Load and use

LOAD hf FROM hf_repo;
SELECT * FROM hf_ls('hf://datasets/lhoestq/demo1');

Reads and writes, by repo type

Reads work for all five types, with or without @revision:

SELECT * FROM read_parquet('hf://datasets/wikimedia/wikipedia/20231101.ab/*.parquet');
SELECT content FROM read_text('hf://models/google-bert/bert-base-uncased/config.json');
SELECT * FROM glob('hf://models/nvidia/**/*.safetensors');
SELECT * FROM hf_ls('hf://spaces/dalle-mini/dalle-mini');
SELECT content FROM read_text('hf://kernels/kernels-community/flash-attn3/README.md');
SELECT * FROM read_parquet('hf://buckets/lhoestq/playground/duckdb/*.parquet');

A bucket file is Xet content — no pointer, no branch, no commit — so a bucket write chunks the file, and uploads it:

COPY (SELECT * FROM large_table) TO 'hf://buckets/me/playground/export.parquet' (FORMAT PARQUET);
COPY (SELECT * FROM events) TO 'hf://buckets/me/playground/events/' (FORMAT PARQUET, PARTITION_BY (day));

hf_bucket_copy enables free file copies since files are pointers in Xet.

Authentication

• with the hf CLI:

hf auth login

• manually with a token:

CREATE SECRET (TYPE HUGGINGFACE, TOKEN 'hf_...');

Token resolution has the same order as huggingface_hub: a matching CREATE SECRET (TYPE HUGGINGFACE, …) → SET hf_token → HF_TOKEN → HF_TOKEN_PATH → $HF_HOME/token → ~/.cache/huggingface/token.

Settings

Table functions

Limitations

• On a released 2.0.0 client, hf_status().duckdb_version answers v2.0.0-alpha49307.
• Writes buffer to a local temp file first and upload + commit happen on Close().
• Dedup is strong on content that repeats and weak on content that is merely similar.
• hf_write_max_parallel_uploads is a fixed client-side policy.
• Small files go as base64 in the commit; larger ones use git-LFS basic multipart.
• A repository cannot be deleted from SQL.
• The read cache is in-memory only and retained outside DuckDB's buffer manager.
• Sequential whole-file reads (read_blob of a big file) are behind httpfs.
• Every request goes through DuckDB's HTTPUtil and the transport must speak https.
• hf_xet_write's fallback cannot be made to fail from SQL.