cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Create DLT pipeline in CI/CD with role segregation

Malthe
New Contributor II

In the documentation, most examples use the CREATE OR REFRESH STREAMING TABLE command.

Meanwhile, from a role segregation perspective, create and refresh operations should happen in a separate context. 

That is, we want to create these objects (which essentially affect only the catalog) during a deployment run while refreshing tables during pipeline runs. Each of these would be their own role essentially, a deployment role and a pipeline execution role.

But as far as I can tell:

  1. It's not possible to create a streaming table without running it (synchronously).
  2. While there is a REFRESH command, it's not supported.

What's the ideal way to set up DLT pipelines in a way that follows best practices of role segregation?

2 REPLIES 2

Renu_
Contributor

Hi @Malthe, refreshing is automatically handled during pipeline runs in here. To implement effective role segregation, you should define separate DLT pipelines for deployment and execution, each with its own set of roles and permissions. This approach ensures a controlled and secure management of streaming tables.

Malthe
New Contributor II

@Renu_this doesn't really address the specific points I'm making – (1) and (2). What I'm looking for here is a firm answer on how to accomplish this, taking into account the current semantics of the CREATE OR REFRESH.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now