Hi,
I'm trying to setup CICD pipeline for Delta Live Table jobs using Databricks Bundles. I have a problem with path to notebook in pipeline. According to this example:
https://docs.databricks.com/en/delta-live-tables/tutorial-bundles.html
YAML file should looks like:
resources:
pipelines:
pipeline_dlt_test:
name: dlt_test_${bundle.target}
clusters:
- label: default
autoscale:
min_workers: 1
max_workers: 5
mode: ENHANCED
libraries:
- notebook:
path: /notebooks/dlt_test
configuration:
bundle.sourcePath: ${workspace.root_path}/databricks/src/notebooks
development: true
catalog: ${bundle.target}_bronze
But it doesn't work:
Assets bundles are ignoring configuration (bundle.sourcePath).
Fix is pretty simple:
resources:
pipelines:
pipeline_dlt_test:
name: dlt_test_${bundle.target}
clusters:
- label: default
autoscale:
min_workers: 1
max_workers: 5
mode: ENHANCED
libraries:
- notebook:
path: ${workspace.file_path}/databricks/src/notebooks/dlt_test
#path: /dlt_test
#configuration:
# bundle.sourcePath: ${workspace.root_path}/databricks/src/notebooks
development: true
catalog: ${bundle.target}_bronze
This sets correct path to the notebook. I'm confused a little it.
I have databricks.yml in root path.
Why bundle.sourcePath variable is ignored?
