Deploy asset bundle without recreating Lakeflow SQL gateway and DLT Pipeline

LJacobsen — Fri, 25 Jul 2025 23:11:08 GMT

Hello all,

I have a pre-existing Databricks Asset Bundle that deploys a workflow.

I am starting to investigate Lakeflow Connect using a SQL Server connector. In my existing Databricks Asset Bundle, I added a pipeline YAML file that defines a gateway and ingestion pipeline that ingests data from a single table. The YAML is at the end of this message. I successfully deployed the bundle using the CLI databricks bundle deploy command. So far, this is all working as expected. The gateway and ingestion pipeline are created and run successfully.

However, whenever I redeploy my DAB, it prompts:

This action will result in the deletion or recreation of the following DLT Pipelines along with the
Streaming Tables (STs) and Materialized Views (MVs) managed by them. Recreating the Pipelines will
restore the defined STs and MVs through full refresh. Note that recreation is necessary when pipeline
properties such as the 'catalog' or 'storage' are changed:
recreate pipeline gateway
recreate pipeline pipeline_sqlserver

Would you like to proceed? [y/n]:

This prompt appears regardless of whether any changes have been made to the bundle definition.

We have an automated CI/CD process that regularly deploys our DAB. I don't want to drop and recreate the gateway and ingestion pipeline with every deployment. How can I bypass this prompt?

Pipeline bundle definition

resources:

pipelines:

gateway:

name: ${var.gateway_name}

gateway_definition:

connection_name: ${var.gateway_connection_name}

gateway_storage_catalog: ${var.catalog_name}

gateway_storage_schema: 'gateway'

gateway_storage_name: ${var.gateway_name}

target: 'sql-ingestion'

channel: PREVIEW

permissions:

- level: IS_OWNER

service_principal_name: ${var.service_principal_name}

pipeline_sqlserver:

name: finance_pipeline_bronze_landing

ingestion_definition:

ingestion_gateway_id: ${resources.pipelines.gateway.id}

objects:

- table:

source_catalog: lakehouse_landing

source_schema: dbo

source_table: TableName

destination_catalog: ${var.catalog_name}

destination_schema: bronze_landing

target: bronze_landing

catalog: ${var.catalog_name}

channel: PREVIEW

permissions:

- level: IS_OWNER

service_principal_name: ${var.service_principal_name}

Re: Deploy asset bundle without recreating Lakeflow SQL gateway and DLT Pipeline

thomas-totter — Thu, 11 Sep 2025 19:17:49 GMT

@LJacobsen If i understand you correctly, you have manually created objects (in your case a DLT pipeline) that you now want to manage in your asset bundle. If that's the case "bundle deployment bind" is your friend:
https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/bundle-commands#bind

With this command you can also bind other pre-existing objects like schemas, volumes, jobs, etc.

topic Re: Deploy asset bundle without recreating Lakeflow SQL gateway and DLT Pipeline in Data Engineering

Deploy asset bundle without recreating Lakeflow SQL gateway and DLT Pipeline

Re: Deploy asset bundle without recreating Lakeflow SQL gateway and DLT Pipeline