topic Re: Parametrize DLT pipeline in Data Engineering

Parametrize DLT pipeline

Malthe — Fri, 18 Apr 2025 08:11:25 GMT

If I'm using Databricks Asset Bundles, how would I parametrize a DLT pipeline based on a static configuration file.

In pseudo-code, I would have a .py-file:

import dlt # Something that pulls a pipeline resource (or artifact) and parses from JSON table_configs = get_config(...) for name, config in table_configs.items(): @Dlt.table(name=name) def my_table(): # do something

The context is that I have a description of the data to ingest in a declarative file format and I'd like to use Python to pull those descriptions out of an artifact that I've deployed (and perhaps even built) using Databricks Asset Bundles.

Re: Parametrize DLT pipeline

Emmitt18Lefebvr — Fri, 18 Apr 2025 09:23:21 GMT

Hello!

To parametrize a Databricks DLT pipeline with a static configuration file using Asset Bundles, include your JSON/YAML config file in the bundle. In your DLT pipeline code, read this file using Python's file I/O (referencing its deployed path). Then, dynamically define your DLT tables using @Dlt.table within a loop, passing relevant configuration parameters to each table function to drive ingestion and transformation logic. Ensure your bundle.yaml includes the config file as an artifact for deployment. This allows for declarative configuration, environment management, and version control of your pipeline setup.