Databricks Apps - Deployment process - Need help !!

[Sanjeeb2024]

Hi all,

Do we have any guidelines or a CI/CD approach for deploying a Databricks app from one environment (e.g., DEV) to PROD?We are currently facing the following challenge:We created a Databricks app in the DEV environment through the UI. At runtime, the app creates a service principal, to which we granted the required permissions to access Databricks tables. After deploying the code, everything works as expected in DEV.How can we fully automate the deployment process from DEV to PROD? In particular, how can we automate assigning the required permissions to the service principal that is created at runtime in the target environment?Any guidance or examples would be appreciated.

Regards

[balajij8]

@Sanjeeb2024 DABs is the right approach for managing and promoting Databricks Apps across environments (DEV to PROD). When you deploy an app via DABs, Databricks automatically provisions a dedicated service principal for that app instance. This service principal persists across all subsequent code updates and is deleted only when the app itself is deleted. Each target environment maintains its own isolated app instance with its own dedicated service principal, keeping DEV and PROD credentials and identities completely decoupled. More details here

To automate access in the target environment, you can declare grants resource directly in the bundle YAML rather than configuring permissions manually in Unity Catalog. DABs exposes the runtime service principal using bundle substitution via ${resources.apps.<app-name>.service_principal_client_id}, allowing you to bind permissions directly to the generated identity during the deploy phase

bundle:
  name: files_db_app

variables:
  catalog:
    default: dev_catalog

targets:
  dev:
    default: true
    workspace:
      host: https://dev
    variables:
      catalog: dev_catalog

  prod:
    workspace:
      host: https://prod
    variables:
      catalog: prod_catalog

resources:
  apps:
    file_app:
      name: 'file-app-${bundle.target}'
  grants:
    app_table_access:
      principal: '${resources.apps.file_app.service_principal_client_id}'
      privileges: [SELECT, MODIFY]
      catalog_name: ${var.catalog}

By parameterizing the catalogs and schemas using bundle variables like ${var.catalog} and ${bundle.target}, permissions are resolved dynamically per stage. Running databricks bundle deploy -t prod in the CI/CD pipeline deploys the updated application code, provisions or updates the PROD-specific service principal and applies all required table privileges in a single, declarative run. More details here

[data_pulse · Accepted solution]

@Sanjeeb2024 
Definitely DABs are good approach here. One thing I would add here is to keep the deployment identity and App run time Identity separate.

For DEV -> PROD, the recommended pattern is to deploy the app with DAB from CI/CD, ideally using a dedicated service principal. The app itself gets its own dedicated runtime service principal in each environment. and it's access to SQL warehouses/ tables/ volumes etc should be declared as app resources / bundle config (where supported) rather than granting manually after deployment. 

For the permissions part specifically: you do not need to manually discover that runtime service principal after deployment. In the bundle it can be referenced  directly using :

${resources.apps.<app-key>.service_principal_client_id}

and use that identity in grants/resources during same deployment.

Typical Flow is:
Git Actions/Dev-ops → CI/CD → bundle validate → bundle deploy -t prod → bundle run <app> -t prod

Note: bundle deploy updates the app resource/source but not necessarily restart the running app, so Databricks recommends bundle run after deployment. Reference for CI/CD from databricks docs here.

For environment specific resources: use bundle targets/variables, so DEV/TEST/PROD can point to respective catalogs/ warehouses / tables etc while keeping the same app definition.

If the App already exists because it was created manually or through another deployment process, can bind existing App to the bundle to avoid App name already exists kind of errors while deploying.

Can also declare App dependencies/resources in the Bundle Yaml. The supported app resource types documented here

[Sanjeeb2024]

Thank you for the details. Yes, I agree we should adopt the DABs for the deployment and define the resources while creating the app ( which can be consider as as solution). However if the permission module is managed separately, lets say via terraform, it is better to separate the app creation and permission module separately for easy management. When we create the app, the sp will created and attach to that app. Subsequent changes may happen to code only.