DLT - Unity catalog and volume - Dynamically access volume path

erigaud
Honored Contributor

‎09-26-2024 12:49 AM

Hello, 

We're using a dlt pipeline using an autoloader that reads from a volume inside Unity catalog

The path of the volume is /Volumes/<my-catalog>/...

How can I dynamically access the catalog value of the dlt pipeline to use it in the code ? I don't want to hard code the name of the catalog in the volume path since it's a parameter of the DLT pipeline, I'm assuming there is a way to retrieve it in the code ?

Thank you

0 Kudos

ChrisLawford_n1
Contributor II

‎09-26-2024 03:41 AM

Hey erigaud,

You can pass information into a DLT pipleine in a similar way to how you can pass information in to a notebook when being run in a job. In the DLT pipeline case you can set values in the configuration. 
Configure pipeline settings for Delta Live Tables - Azure Databricks | Microsoft Learn

In this case this would look something like:
Databricks DAB file:

    resources:
      pipelines:
        pipeline_test:
          catalog: sandbox
          target: test
          configuration:
            volume_path: "banana"

 And then in the notebook that is being run:

path_in_volume = spark.conf.get("volume_path")

spark.readStream.format("cloudFiles")
        .option("cloudFiles.format", "parquet")
        .load(
            f"\volumes\{path_in_volume}"
        )

View solution in original post

erigaud
Honored Contributor

‎09-27-2024 01:51 AM

Works perfectly, thank you ! It's a shame the documentation does not detail that use case