In the documentation, most examples use the CREATE OR REFRESH STREAMING TABLE command.
Meanwhile, from a role segregation perspective, create and refresh operations should happen in a separate context.
That is, we want to create these objects (which essentially affect only the catalog) during a deployment run while refreshing tables during pipeline runs. Each of these would be their own role essentially, a deployment role and a pipeline execution role.
But as far as I can tell:
- It's not possible to create a streaming table without running it (synchronously).
- While there is a REFRESH command, it's not supported.
What's the ideal way to set up DLT pipelines in a way that follows best practices of role segregation?