Skipper: Building Airbnb's embedded workflow engine (12 minute read)
Airbnb built Skipper, an embedded workflow engine that handles long-running processes by storing state in the service's own database instead of using external orchestration tools.
What: Skipper is Airbnb's lightweight workflow engine that embeds directly into services to manage durable, long-running business processes like insurance claims and payments using an annotation-based approach to persist state and deterministic replay for reliability.
Why it matters: This represents an alternative architectural pattern to traditional workflow orchestration that reduces operational complexity by eliminating dependencies on external systems like message queues or dedicated orchestration platforms while still providing durability guarantees.
Decoder
- Workflow engine: A system that manages the execution of multi-step business processes, coordinating tasks and handling failures across time
- Deterministic replay: A technique where processes can be reliably restarted by replaying events in the same order to reconstruct state without data loss
- Durable execution: Guaranteed process completion even across failures or restarts by persisting state to storage
Original article
Skipper is a lightweight, embedded workflow engine designed to provide durable and reliable execution for long-running business processes (like insurance claims and payments). Instead of relying on external orchestration tools or queues, Skipper uses a simple annotation-based approach to persist state in the service's existing database and achieves durability through deterministic replay.