Databricks

knutasm · ‎08-11-2023

How to run a delta live tables pipeline in production? It uses the owner's (creator's) permissions for writing to tables, and I can't change the owner of a UC-enabled pipeline after creation. I don't want regular users to have write access to prod tables, but run as a service principal instead.

I can find no example in the documentation, but it seems like such a glaring shortcoming that I am sure I must have missed something.

Kaniz · ‎08-11-2023

Hi @knutasm, To run a Delta Live Tables pipeline in production with a service principal, ensure the service principal is the pipeline's owner. As per the provided information, the pipeline owner is the one whose permissions are used for writing to tables, and this owner can only be changed by a workspace admin.

Here are the steps to change the owner of a Delta Live Tables pipeline:

1. Go to the details page for a pipeline.
2. Click the **Permissions** button in the **Pipeline Details** panel.
3. In the pop-up dialogue box, assign the **Is Owner** permission to the service principal by clicking the drop-down menu beside the service principal's name.
4. Click **Save**.Remember, the service principal must have the necessary permissions for the operations the pipeline is supposed to perform. Please note that once the pipeline owner is set, it cannot be changed for a Unity Catalog (UC)-enabled pipeline. If you need to change the owner for a UC-enabled pipeline, you must create a new pipeline with the desired owner.

Sources:
- [Docs: dlt-acl](https://docs.databricks.com/security/auth-authz/access-control/dlt-acl.html)
- [Docs: unity-catalog](https://docs.databricks.com/delta-live-tables/unity-catalog.html)

knutasm · ‎08-11-2023

Thank you.

However, like you say the owner of the pipeline cannot be changed once set for a UC pipeline. Yet I am unable to choose pipeline owner when creating the pipeline in the first place, which is where I would have to set the service principal. I am also unable to create a pipeline as a service principal.

ilarsen · ‎11-14-2023

@knutasm wrote:
Thank you.
However, like you say the owner of the pipeline cannot be changed once set for a UC pipeline. Yet I am unable to choose pipeline owner when creating the pipeline in the first place, which is where I would have to set the service principal. I am also unable to create a pipeline as a service principal.

Hmm. I wonder if creating the pipeline could it be done with the API? I've not seen anything about that in my browsing around. Even if it that's an option, I wouldn't say that's an ideal option.

I'm going to assume that DLT with UC is simply not finished baking yet and that this will change sometime... In the meantime, I might not jump in to DLT with both feet just yet.