- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2024 12:49 AM
Hello,
We're using a dlt pipeline using an autoloader that reads from a volume inside Unity catalog
The path of the volume is /Volumes/<my-catalog>/...
How can I dynamically access the catalog value of the dlt pipeline to use it in the code ? I don't want to hard code the name of the catalog in the volume path since it's a parameter of the DLT pipeline, I'm assuming there is a way to retrieve it in the code ?
Thank you
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2024 03:41 AM
Hey erigaud,
You can pass information into a DLT pipleine in a similar way to how you can pass information in to a notebook when being run in a job. In the DLT pipeline case you can set values in the configuration.
Configure pipeline settings for Delta Live Tables - Azure Databricks | Microsoft Learn
In this case this would look something like:
Databricks DAB file:
resources:
pipelines:
pipeline_test:
catalog: sandbox
target: test
configuration:
volume_path: "banana"
And then in the notebook that is being run:
path_in_volume = spark.conf.get("volume_path")
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "parquet")
.load(
f"\volumes\{path_in_volume}"
)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2024 03:41 AM
Hey erigaud,
You can pass information into a DLT pipleine in a similar way to how you can pass information in to a notebook when being run in a job. In the DLT pipeline case you can set values in the configuration.
Configure pipeline settings for Delta Live Tables - Azure Databricks | Microsoft Learn
In this case this would look something like:
Databricks DAB file:
resources:
pipelines:
pipeline_test:
catalog: sandbox
target: test
configuration:
volume_path: "banana"
And then in the notebook that is being run:
path_in_volume = spark.conf.get("volume_path")
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "parquet")
.load(
f"\volumes\{path_in_volume}"
)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-27-2024 01:51 AM
Works perfectly, thank you ! It's a shame the documentation does not detail that use case

