Hello all,
I have a pre-existing Databricks Asset Bundle that deploys a workflow.
I am starting to investigate Lakeflow Connect using a SQL Server connector. In my existing Databricks Asset Bundle, I added a pipeline YAML file that defines a gateway and ingestion pipeline that ingests data from a single table. The YAML is at the end of this message. I successfully deployed the bundle using the CLI databricks bundle deploy command. So far, this is all working as expected. The gateway and ingestion pipeline are created and run successfully.
However, whenever I redeploy my DAB, it prompts:
This action will result in the deletion or recreation of the following DLT Pipelines along with the
Streaming Tables (STs) and Materialized Views (MVs) managed by them. Recreating the Pipelines will
restore the defined STs and MVs through full refresh. Note that recreation is necessary when pipeline
properties such as the 'catalog' or 'storage' are changed:
recreate pipeline gateway
recreate pipeline pipeline_sqlserver
Would you like to proceed? [y/n]:
This prompt appears regardless of whether any changes have been made to the bundle definition.
We have an automated CI/CD process that regularly deploys our DAB. I don't want to drop and recreate the gateway and ingestion pipeline with every deployment. How can I bypass this prompt?
Pipeline bundle definition
resources:
pipelines:
gateway:
name: ${var.gateway_name}
gateway_definition:
connection_name: ${var.gateway_connection_name}
gateway_storage_catalog: ${var.catalog_name}
gateway_storage_schema: 'gateway'
gateway_storage_name: ${var.gateway_name}
target: 'sql-ingestion'
channel: PREVIEW
permissions:
- level: IS_OWNER
service_principal_name: ${var.service_principal_name}
pipeline_sqlserver:
name: finance_pipeline_bronze_landing
ingestion_definition:
ingestion_gateway_id: ${resources.pipelines.gateway.id}
objects:
- table:
source_catalog: lakehouse_landing
source_schema: dbo
source_table: TableName
destination_catalog: ${var.catalog_name}
destination_schema: bronze_landing
target: bronze_landing
catalog: ${var.catalog_name}
channel: PREVIEW
permissions:
- level: IS_OWNER
service_principal_name: ${var.service_principal_name}
