Hi @turagittech,
Great question and I understand the concern. The good news is that updating the notification email in your Asset Bundle YAML and redeploying should NOT recreate your gateway or ingestion pipeline resources. Let me explain why and walk through the safest approach.
HOW DATABRICKS BUNDLE DEPLOY HANDLES UPDATES
When you run "databricks bundle deploy", the CLI tracks each resource by its internal ID (stored in your bundle's state file under .databricks/bundle). On subsequent deployments, the behavior is:
- If a resource in the bundle config already exists in the workspace, it is UPDATED IN PLACE.
- If a resource is new (not previously deployed), it is created.
- If a resource is removed from the bundle config, it is removed from the workspace.
This means changing the email address in your job's email_notifications section and redeploying will simply update the existing job configuration. It will NOT recreate the gateway pipeline or the ingestion pipeline, and your CDC tracking state (change positions, streaming table state, staging volumes) will remain intact.
Documentation on bundle deploy behavior: https://docs.databricks.com/en/dev-tools/bundles/work-tasks.html
STEP-BY-STEP: UPDATING THE EMAIL SAFELY
1. In your Asset Bundle YAML (likely under resources/sqlserver_job.yml or similar), locate the email_notifications section of your job resource. It will look something like:
resources:
jobs:
sqlserver_dab_job:
name: sqlserver_dab_job
email_notifications:
on_failure:
- old-email@company.com
tasks:
- task_key: refresh_pipeline
pipeline_task:
pipeline_id: ${resources.pipelines.pipeline_sqlserver.id}
2. Change the email address to the new one:
email_notifications:
on_failure:
- new-email@company.com
3. Before deploying, you can preview exactly what will change by running:
databricks bundle validate
This will confirm that the bundle is valid without making any changes.
4. Deploy the update:
databricks bundle deploy
This updates the job in place. Your gateway pipeline and ingestion pipeline are separate resources and will not be affected by a change to the job resource.
5. Verify the change took effect by checking the job configuration in the Databricks UI under Workflows, or run:
databricks jobs get --job-id <your-job-id>
ALTERNATIVE: UPDATE THE EMAIL DIRECTLY IN THE UI
If you want to change the email immediately without going through a bundle deployment cycle, you can also update it directly in the Databricks UI:
1. Go to Workflows in the left sidebar
2. Find your CDC job (the job that triggers the ingestion pipeline)
3. Click on the job name to open it
4. In the Job details panel, scroll to "Job notifications" and click "Edit notifications"
5. Update the email address and click Save
Important note: If you change the email in the UI but NOT in your bundle YAML, the next "databricks bundle deploy" will overwrite the UI change with whatever is in the YAML. So make sure to update both, or just update the YAML and redeploy (which is the cleaner approach for bundle-managed resources).
WHY THE GATEWAY IS NOT AFFECTED
Your Asset Bundle likely defines three separate resources:
1. The gateway pipeline (gateway_definition) - captures changes from SQL Server
2. The ingestion pipeline (ingestion_definition) - applies changes to streaming tables
3. The job - schedules the ingestion pipeline and sends notifications
Changing the email only modifies resource #3 (the job). Resources #1 and #2 are tracked independently by the bundle state, and their configurations are unchanged, so they receive no updates during deployment. Your CDC tracking positions and streaming table state are preserved.
GENERAL TIP FOR BUNDLE CHANGES
If you ever want to be extra cautious before a bundle deploy, you can check the state file to confirm your resource IDs are being tracked:
cat .databricks/bundle/<target>/terraform.tfstate
This will show you the mapping between your bundle resource names and the actual workspace resource IDs. As long as those IDs remain the same, deploy will update rather than recreate.
Hope this clears up the concern. The bundle deploy model is designed specifically for this kind of incremental configuration update without disrupting running resources.
* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.
