DEVOURED
Data Projects: Managing Data Assets at Netflix Scale
Netflix launched "Data Projects" to replace its legacy ACL system with durable team-owned app identities and scoped roles, streamlining data asset management and reducing permission churn.
What: Netflix introduced Data Projects to manage its millions of tables and thousands of data jobs, replacing brittle ACLs (Access Control Lists) and human-owned workflow identities. This system groups tables, workflows, secrets, and assets under durable team-owned application identities with scoped roles and tokens to reduce permission churn.
Why it matters: This illustrates how large-scale data operations require custom solutions to manage access control and identity, moving away from granular, brittle manual permissions to a more organized, programmatic, and team-centric model to improve security and efficiency.
Deep dive
- Netflix developed "Data Projects" to address challenges in managing data asset access at scale, specifically replacing complex ACLs and human-owned workflow identities.
- The legacy system led to "permission churn," where roles and access needed constant updates due to changing team members or responsibilities, and made auditing difficult.
- Data Projects group related data assets (tables, workflows, secrets) under a single, durable "team-owned app identity."
- This app identity uses scoped roles and tokens, providing a more stable and auditable access mechanism.
- The system aims to simplify onboarding new employees and teams by providing pre-defined access patterns linked to projects.
- It also improves security posture by centralizing control and making access explicit to project boundaries.
- The architecture likely integrates with existing identity management systems at Netflix but introduces a layer of abstraction specific to data assets.
Decoder
- ACL (Access Control List): A list of permissions associated with an object, specifying which users or system processes are granted access to objects, and what operations are allowed on given objects.
- Permission churn: The constant creation, modification, or revocation of access permissions due to changing personnel, projects, or requirements, leading to complexity and potential security gaps.
- Workflow identities: Digital identities used by automated processes or scripts to access resources, which traditionally might be tied to individual human accounts, creating issues when those individuals leave or change roles.
Original article
Netflix introduced Data Projects to replace brittle ACLs and human-owned workflow identities across millions of tables and thousands of jobs. Projects group tables, workflows, secrets, and assets under durable team-owned app identities, with scoped roles and tokens to reduce permission churn.
Explore all 40 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi