Parametrize DLT pipeline

[Malthe]

If I'm using Databricks Asset Bundles, how would I parametrize a DLT pipeline based on a static configuration file.

In pseudo-code, I would have a .py-file:

import dlt

# Something that pulls a pipeline resource (or artifact) and parses from JSON
table_configs = get_config(...)

for name, config in table_configs.items():
    @Dlt.table(name=name)
    def my_table():
        # do something    

 The context is that I have a description of the data to ingest in a declarative file format and I'd like to use Python to pull those descriptions out of an artifact that I've deployed (and perhaps even built) using Databricks Asset Bundles.

[Emmitt18Lefebvr]

Hello!

To parametrize a Databricks DLT pipeline with a static configuration file using Asset Bundles, include your JSON/YAML config file in the bundle. In your DLT pipeline code, read this file using Python's file I/O (referencing its deployed path). Then, dynamically define your DLT tables using @Dlt.table within a loop, passing relevant configuration parameters to each table function to drive ingestion and transformation logic. Ensure your bundle.yaml includes the config file as an artifact for deployment. This allows for declarative configuration, environment management, and version control of your pipeline setup.