Delta Live Tables + Databricks Assets Bundles

[Elderion]

Hi,

I'm trying to setup CICD pipeline for Delta Live Table jobs using Databricks Bundles. I have a problem with path to notebook in pipeline. According to this example:

https://docs.databricks.com/en/delta-live-tables/tutorial-bundles.html

YAML file should looks like:

resources:

  pipelines:

    pipeline_dlt_test:

      name: dlt_test_${bundle.target}

      clusters:

        - label: default

          autoscale:

            min_workers: 1

            max_workers: 5

            mode: ENHANCED

      libraries:

        - notebook:

            path:  /notebooks/dlt_test

      configuration:

        bundle.sourcePath: ${workspace.root_path}/databricks/src/notebooks

      development: true

      catalog: ${bundle.target}_bronze

But it doesn't work:

Assets bundles are ignoring configuration (bundle.sourcePath).

Fix is pretty simple:

resources:

  pipelines:

    pipeline_dlt_test:

      name: dlt_test_${bundle.target}

      clusters:

        - label: default

          autoscale:

            min_workers: 1

            max_workers: 5

            mode: ENHANCED

      libraries:

        - notebook:

            path:  ${workspace.file_path}/databricks/src/notebooks/dlt_test

            #path:  /dlt_test

      #configuration:

      #  bundle.sourcePath: ${workspace.root_path}/databricks/src/notebooks

      development: true

      catalog: ${bundle.target}_bronze

This sets correct path to the notebook. I'm confused a little it.

I have databricks.yml in root path.

Why bundle.sourcePath variable is ignored?

[filipniziol · Accepted solution]

Hi @Elderion ,

A couple of lines from Databricks AI assistant:

The issue you're encountering with the bundle.sourcePath variable being ignored in your CI/CD pipeline setup for Delta Live Tables (DLT) using Databricks Repos and Bundles seems to stem from a misunderstanding of how asset bundle paths are resolved in the DLT configuration.

In the DLT configuration within a YAML file, the bundle.sourcePath is intended to specify the local directory that contains the assets (e.g., notebooks, libraries) to be included in the bundle when you're running databricks repos bundle create. This means it's used during the bundle creation process to determine what local files should be included in the bundle.

However, when specifying paths in the libraries section of your DLT pipeline configuration, you're actually referencing the path within the Databricks workspace or the bundle itself after it has been uploaded. The ${workspace.file_path} variable is used to reference the path of the file within the workspace, which is why your fix works. It correctly specifies the path to the notebook within the bundle or workspace, rather than relying on the bundle.sourcePath which is not used at runtime for resolving asset paths in the pipeline configuration.

The bundle.sourcePath is not used by the DLT pipeline configuration at runtime to resolve paths to assets; it's only used at bundle creation time. To reference assets within your DLT pipeline configuration, you should use workspace or bundle-relative paths, as you've discovered.

In summary, your fix is the correct approach for specifying the path to the notebook in the DLT pipeline configuration. The bundle.sourcePath is for bundle creation, not for runtime asset path resolution in the pipeline configuration.

To sum up "sourcePath" is what you provide to specify where is the bundle defined locally, "workspace" variables are related to your targets after deployment. You are right that the example on the page is quite confusing and probably the configuration generated by default does not help. 

What you can do you can use a relative path to the location where databricks.yml is located, like .../databricks/src/notebooks/dlt_test.

[Elderion]

Perfect, thanks! Sounds like a good explanaition. 🙂 

[ThierryBa]

I had this error once.

you need to specify the extension of your file. If you set the notebook to be python, then it must be .py at the end, likewise .sql if you used SQL

    libraries:

        - notebook:

            path:  ${workspace.file_path}/databricks/src/notebooks/dlt_test.py 

OR

    libraries:

        - notebook:

            path:  ${workspace.file_path}/databricks/src/notebooks/dlt_test.sql