topic Re: DLT - Unity catalog and volume - Dynamically access volume path in Data Engineering

DLT - Unity catalog and volume - Dynamically access volume path

erigaud — Thu, 26 Sep 2024 07:49:37 GMT

Hello,

We're using a dlt pipeline using an autoloader that reads from a volume inside Unity catalog

The path of the volume is /Volumes/<my-catalog>/...

How can I dynamically access the catalog value of the dlt pipeline to use it in the code ? I don't want to hard code the name of the catalog in the volume path since it's a parameter of the DLT pipeline, I'm assuming there is a way to retrieve it in the code ?

Thank you

Re: DLT - Unity catalog and volume - Dynamically access volume path

ChrisLawford_n1 — Thu, 26 Sep 2024 10:41:07 GMT

Hey erigaud,

You can pass information into a DLT pipleine in a similar way to how you can pass information in to a notebook when being run in a job. In the DLT pipeline case you can set values in the configuration.
Configure pipeline settings for Delta Live Tables - Azure Databricks | Microsoft Learn

In this case this would look something like:
Databricks DAB file:

resources: pipelines: pipeline_test: catalog: sandbox target: test configuration: volume_path: "banana"

And then in the notebook that is being run:

path_in_volume = spark.conf.get("volume_path") spark.readStream.format("cloudFiles") .option("cloudFiles.format", "parquet") .load( f"\volumes\{path_in_volume}" )

Re: DLT - Unity catalog and volume - Dynamically access volume path

erigaud — Fri, 27 Sep 2024 08:51:21 GMT

Works perfectly, thank you ! It's a shame the documentation does not detail that use case