topic Re: Create DLT pipeline in CI/CD with role segregation in Get Started Discussions

Create DLT pipeline in CI/CD with role segregation

Malthe — Tue, 04 Mar 2025 07:37:31 GMT

In the documentation, most examples use the CREATE OR REFRESH STREAMING TABLE command.

Meanwhile, from a role segregation perspective, create and refresh operations should happen in a separate context.

That is, we want to create these objects (which essentially affect only the catalog) during a deployment run while refreshing tables during pipeline runs. Each of these would be their own role essentially, a deployment role and a pipeline execution role.

But as far as I can tell:

It's not possible to create a streaming table without running it (synchronously).
While there is a REFRESH command, it's not supported.

What's the ideal way to set up DLT pipelines in a way that follows best practices of role segregation?

Re: Create DLT pipeline in CI/CD with role segregation

Renu_ — Tue, 01 Apr 2025 13:41:50 GMT

Hi @Malthe, refreshing is automatically handled during pipeline runs in here. To implement effective role segregation, you should define separate DLT pipelines for deployment and execution, each with its own set of roles and permissions. This approach ensures a controlled and secure management of streaming tables.

Re: Create DLT pipeline in CI/CD with role segregation

Malthe — Wed, 02 Apr 2025 06:44:19 GMT

@Renu_this doesn't really address the specific points I'm making – (1) and (2). What I'm looking for here is a firm answer on how to accomplish this, taking into account the current semantics of the CREATE OR REFRESH.