Create DLT pipeline in CI/CD with role segregation
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-03-2025 11:37 PM
In the documentation, most examples use the CREATE OR REFRESH STREAMING TABLE command.
Meanwhile, from a role segregation perspective, create and refresh operations should happen in a separate context.
That is, we want to create these objects (which essentially affect only the catalog) during a deployment run while refreshing tables during pipeline runs. Each of these would be their own role essentially, a deployment role and a pipeline execution role.
But as far as I can tell:
- It's not possible to create a streaming table without running it (synchronously).
- While there is a REFRESH command, it's not supported.
What's the ideal way to set up DLT pipelines in a way that follows best practices of role segregation?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-01-2025 06:41 AM
Hi @Malthe, refreshing is automatically handled during pipeline runs in here. To implement effective role segregation, you should define separate DLT pipelines for deployment and execution, each with its own set of roles and permissions. This approach ensures a controlled and secure management of streaming tables.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-01-2025 11:44 PM
@Renu_this doesn't really address the specific points I'm making – (1) and (2). What I'm looking for here is a firm answer on how to accomplish this, taking into account the current semantics of the CREATE OR REFRESH.
