topic Re: Run Delta Live Tables as service principal in Data Engineering

Run Delta Live Tables as service principal

knutasm — Fri, 11 Aug 2023 08:12:30 GMT

How to run a delta live tables pipeline in production? It uses the owner's (creator's) permissions for writing to tables, and I can't change the owner of a UC-enabled pipeline after creation. I don't want regular users to have write access to prod tables, but run as a service principal instead.

I can find no example in the documentation, but it seems like such a glaring shortcoming that I am sure I must have missed something.

Re: Run Delta Live Tables as service principal

knutasm — Fri, 11 Aug 2023 11:21:44 GMT

Thank you.

However, like you say the owner of the pipeline cannot be changed once set for a UC pipeline. Yet I am unable to choose pipeline owner when creating the pipeline in the first place, which is where I would have to set the service principal. I am also unable to create a pipeline as a service principal.

Re: Run Delta Live Tables as service principal

Oliver_Angelil — Wed, 06 Sep 2023 15:13:36 GMT

Have the same issue.

Re: Run Delta Live Tables as service principal

serelk — Tue, 12 Sep 2023 09:58:06 GMT

Same here

Re: Run Delta Live Tables as service principal

js54123875 — Wed, 01 Nov 2023 15:26:28 GMT

same!

Re: Run Delta Live Tables as service principal

ilarsen — Tue, 14 Nov 2023 23:41:04 GMT

@knutasm wrote:
Thank you.
However, like you say the owner of the pipeline cannot be changed once set for a UC pipeline. Yet I am unable to choose pipeline owner when creating the pipeline in the first place, which is where I would have to set the service principal. I am also unable to create a pipeline as a service principal.

Hmm. I wonder if creating the pipeline could it be done with the API? I've not seen anything about that in my browsing around. Even if it that's an option, I wouldn't say that's an ideal option.

I'm going to assume that DLT with UC is simply not finished baking yet and that this will change sometime... In the meantime, I might not jump in to DLT with both feet just yet.

Re: Run Delta Live Tables as service principal

AmanSehgal — Thu, 13 Feb 2025 03:00:32 GMT

Configure pipeline permissions

You must have the CAN MANAGE or IS OWNER permission on the pipeline to manage permissions. Pipelines use access control lists (ACLs) to control permissions. For a complete list of permissions and their abilities.

In the sidebar, click Delta Live Tables.
Select the name of a pipeline.
Click Share. The Permissions Settings dialog appears.
Click Select User, Group or Service Principal… and select a user, group, or service principal.
Select a permission from the permission drop-down menu.
Click Add.
Click Save.

Re: Run Delta Live Tables as service principal

ashwini0723 — Tue, 04 Mar 2025 12:32:13 GMT

@knutasmI have build the solution for it. The way to create DLT pipeline using SPN is to write a code wherein via databricks API a new DLT pipeline will be created and you mentioned owner as a service principal in the API code as shown below. Below method worked for me 🙂

import requests import requests from azure.identity import ClientSecretCredential databricks_instance = "" client_id = dbutils.secrets.get(scope="", key="") client_secret = dbutils.secrets.get(scope="", key="") tenant_id = "" #pipeline_id = "" # Obtain token token_response = requests.post( f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token", data={ 'grant_type': 'client_credentials', 'client_id': client_id, 'client_secret': client_secret, 'scope': '2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default' } ) access_token = token_response.json().get("access_token") if not access_token: print(f"Error obtaining token: {token_response.text}") else: headers = { 'Authorization': f'Bearer {access_token}', 'Content-Type': 'application/json' } # Define the pipeline creation payload using Unity Catalog create_pipeline_payload = { "name": "", "catalog": "", # Specify the Unity Catalog name "target": "", # Specify the schema within the Unity Catalog "clusters": [ { "label": "default", "node_type_id": "Standard_DS3_v2", "autoscale": { "min_workers": 1, "max_workers": 2, "mode": "ENHANCED" } } ], "libraries": [ { "notebook": { "path": "" } } ], "permissions": [ { "service_principal_name": '', "permission_level": "IS_OWNER" # Set the desired permission level } ], "development": True # Set to False for production mode } # Make the API request to create the pipeline response = requests.post( f'{databricks_instance}/api/2.0/pipelines', headers=headers, json=create_pipeline_payload ) # Check the response if response.status_code == 200: pipeline_id = response.json().get("pipeline_id") print(f"Pipeline created successfully with ID: {pipeline_id}") else: print(f"Failed to create pipeline: {response.status_code}, {response.text}")