Back to Devoured - September 10, 2026
DEVOURED
Filament (GitHub Repo)

Filament (GitHub Repo)

Data infrastructurego GitHub
Filament provides a pluggable data replication engine that enforces integrity by verifying batches before advancing checkpoints.
What: Filament moves data between sources and sinks using full, incremental, or Change Data Capture (CDC) replication. It uses PostgreSQL for state storage and NATS for event transport, and it can be run as a service, embedded in Go applications, or deployed via Kubernetes.
Why it matters: This approach to replication treats data movement as an event-driven system where checkpoint advancement is strictly gated by batch verification, increasing reliability for distributed data pipelines.
Takeaway: Use `brew install galaxy-io/tap/filament` or the provided curl command to install the engine locally for testing.
Decoder
  • CDC (Change Data Capture): A pattern where database changes are captured in real-time as a stream of events rather than performing periodic bulk extracts.
  • Bounded Batch: A data processing unit with a defined start and end point, allowing for easier retries and progress tracking.
Original article

Filament

Filament is pluggable data replication with checkpointing, batching, and integrity events.

It moves data from sources to sinks using full, incremental, or change data capture replication. Filament keeps progress durable as data moves, verifies batches on both sides of a write, and safely pauses or resumes runs from their checkpoints.

Sources, sinks, state storage, and event transport are replaceable interfaces. Run Filament as a service with its API and web UI, deploy it to your own infrastructure, or embed the engine in a Go application.

Install

curl -fsSL https://getgalaxy.io/filament/install | sh

On macOS via Homebrew

brew install galaxy-io/tap/filament

Prebuilt binaries for Linux and macOS are on the releases page.

Getting started

To run Filament locally, install Go, Docker, just, Node.js, and pnpm. On macOS, the repository's Brewfile installs the toolchain:

brew bundle

For Podman users, set CONTAINER_ENGINE=podman to use Podman with the commands below.

Start PostgreSQL and NATS, then run Filament:

just infra
just dev

Open http://localhost:5173 to use the web UI.

When you are finished, stop the local infrastructure with:

just infra down

Deploying Filament

Kubernetes with Helm is the recommended way to run Filament in production. See the Kubernetes deployment guide for installation instructions.

For a simpler hosted deployment without Kubernetes, deploy with 1 click:

Read the Railway, Render, and DigitalOcean guides for the deployed resources, runtime behavior, and production checklist.

Warning

The DigitalOcean deployment uses filament/standalone, an all-in-one image that includes the control-plane, server, and SQLite datastore.

How it works

A Filament pipeline connects a source to a sink and defines how selected resources should be replicated. During a run, Filament extracts records into bounded batches, writes and verifies each batch, and advances durable checkpoints only after successful work.

The runtime is event-driven: the server accepts pipeline and run requests, the control plane schedules and dispatches work, and workers execute individual runs. PostgreSQL stores durable state and NATS JetStream carries lifecycle events between components.

Developing Filament

Filament is written in Go, with a React and TypeScript web UI. Contributions to the engine, connectors, UI, deployment tooling, tests, and documentation are welcome.

Community

Explore all 54 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi